Metodoloski zvezki, Vol. 12, No. 2, 2015, 85-110 Design and Development of the Income Measures in the European Social Surveys Juergen H.P. Hoffmeyer-Zlotnik1 and Uwe Warner2 Abstract In social surveys "total net household income" is an indicator of the re -spondent's socio-economic status. It describes the economic situation of household members and their positions in an income distribution. It is used as an explanatory variable in mobility studies and as a social-demographic background item in inequality research. This paper shows the impact of questionnaire design on measurements of "total net household income" in social sur -veys. In particular we illustrate how the measurement quality of the income variable depends on the data sources about the national income distributions used to design the answer categories offered to the respondent. Beginning with the fourth round of European Social Survey fielded in 2008 and the following years, the income categories for the question about the "total net household income" amount are built on national income distribution of households resi -dent in the country under study. The response categories of the modified ESS questionnaires have been based on deciles of the actual household income distribution in the country in question. The central organizers of European Social Survey (ESS) instruct the national questionnaire designers to define the income brackets for the answer categories using the deciles of the most reliable national income data source. Analyzing the ESS data from 2008, 2010 and 2012, we found in some countries remarkable divergences from the expected 10% frequencies in each category. In this article we argue that the quality of this new income measure depends on the quality of the reference statistics from which the national household income ranges are derived. The quality of the responses to the survey question about the "total net household income", and finally the quality of the obtained survey measure, depends on the quality of the reference statistics from which the household income categories for the answers is derived. These reference data must cover all types of the household's income from all household members and optimally repre sent the national distribution of household income across the survey universe. That means first that all possible payments accruing to a household and all its members in a given country must be reported in references, and second that all households 1 Institute of Political Science, Justus Liebig University of Giessen, Germany, juergen.hoffmeyer-zlotnik@sowi.uni-giessen.de 2 Perl, Germany, uwe,warner@orange.lu 86 Juergen H.P. Hoffmeyer-Zlotnik and Uwe Warner in the survey's universe must be represented in the statistics used to detect the answer categories. Then the income brackets for the response categories can be calculated using the 10% percentiles from the income distribution in the reference data. Relevance, accuracy, timeliness, comparability, coherence, accessibility and clarity are quality domains of official statistics used as refer -ence data for the survey measurement. We conclude that the central coordinators of the ESS define and communicate minimum threshold values for quality domains of the reference data. The national coordinators should report deviations. This would give the users of ESS data an insight into the quality of the income measurement. 1 Introduction In a previous article (Hoffmeyer-Zlotnik and Warner, 2006) we discussed the measurement of "total net household income" in the first round of the European Social Survey (ESS) fielded in 2002 and we presented a proposal for changing the survey instrument in such a way that cross-national comparative social research with this socio-economic background variable would produce robust results. One advantage of the questionnaire module on "total net household income" we proposed is that it includes a question that captures the impact of household size and the respondent's relationship to the main income earner. Moreover, the composition of the household income, the entire number of income types received by each member of the household in which the respondent lives, and the main source of income are collected using showcards. This helps the respondent to recall the individual components of the income received by all the household members and to add up these amounts to yield the "total net household income". The second advantage of our proposal accounts for the various national income distributions of the participating countries. In countries with different income distributions and different average incomes, the income brackets of the response categories differ across the countries. Our main argument is that the lower income categories are more differentiated in countries with lower average income. In countries with higher average income crude income brackets can be used as answer categories at the lower part of the income distribution, however the categories at the upper part income distribution are differentiated. The ESS 2002 used one common showcard with the same answer categories and income brackets for all participating countries. To a certain extent, our suggestions for improvement were taken into account in the design of the fourth round of ESS implemented in 2008 and in subsequent rounds. The income categories for the question about income level are based on the national income distribution of households resident in the country in question. The response categories of the modified ESS questionnaires are based on deciles of the actual household income distribution in the country in question. In the present article we argue that the quality of this new income measure depends on the quality of the statistics from which the national household income ranges are derived. Design and Development of the Income Measures 87 2 Theoretical background: "Total net household income" as a socio-economic variable in social surveys Demographic and socio-economic measures are so-called background variables. As Braun and Mohler (2003: 101f.) point out, "... they are used as independent varia-bles, as socio-economic covariates of attitudes, behaviour, or test scores, etc. and in all sorts of statistical models, in particular, as endogenous factors in causal analysis. They enable analysts to establish homogeneous subgroups, explain differences of scale scores due to different composition, and to identify spurious correlations and causal relationships. Background variables are also used to assess the quality of a realized sample and to decide on any corrections necessary. The distribution (of a combination) of background variables in a realized sample is compared with the population characteristics from official data. Deviations from the known population distributions can be corrected by appropriate weights." Demographic and socio-economic background variables describe social and cultural concepts of societies and social structures. Besides the three classical variables - sex, age, and education - the number of demographic and socio-economic variables needed to determine relationships between attitudes and social characteristics depends on the research question.3 The three classical background variables, together with occupation, labour force status and employment status, ethnicity and migration background, family- and household structure, and "total net household income", serve to typify the respondents and to describe the social context in which they act. In the social sciences, income data are used as independent socio-economic variables to explain individual differences in social position within a society. Because disposable income is an indicator of the purchasing power that enables people to satisfy their social needs, net household income, and the disposable per capita income of each household member derived from it, determine standards of living and lifestyles in stratified societies (see Lepsius, 1974; Lepsius, 1993: 156ff.). Individuals' different positions in the social network of relationships constitute social inequality. In the words of Hradil (2006: 195f ), "social inequality" refers to the living conditions (working conditions, income, material wealth, education attainment, etc.) that allow some people to achieve ge nerally shared goals of a "good life" (for example health, security, wealth, respect) better than others. A social science concept of income as a background variable must succeed in explaining differences in behaviour, attitudes, orientations, and membership of forms of social organisation observed across groups of respondents. For this reason, an instrument for the measurement of income for social scientific analyses must capture the "more versus less" of net household income and the "top versus bottom" in the 3 Depending on the research interest, it is useful to select further background variables, for example religion, political orientation, marital status, dwelling, health, nutrition, body mass index, etc. 88 Juergen H.P. Hoffmeyer-Zlotnik and Uwe Warner income distribution, and must allow income differences to be examined. Cross-national comparisons have revealed that not only income levels and income differences are of relevance but also the types of income and the different sources from which income is drawn. This is due to the fact that income classes also differ in terms of the composition of the income they receive, and that countries differ in terms of "income packaging" (see Rainwater, Rein and Schwartz, 1987). Hence, in 1968 Gosta Carlsson (p. 189) counted the source and level of the income among the definitive variables of social stratification. It later emerged that income composition also affects the quality of the information provided by respondents. 3 Measurement instrument "total net household income" used in the European Social Surveys 2002, 2004, and 2006 The European Social Survey is an academically driven cross-national social survey that has been conducted every two years across Europe since 2002. By now, the survey collects data about attitudes, beliefs and behaviour of populations in over thirty nations. The aims of the ESS are to study stability and change in social structure, opinions and attitudes of citizens in Europe, and to provide indicators of citizens' perceptions and ju dgments of aspects of their societies and social and political life. To achieve comparability in the operationalisation of the ESS across countries, the Core Scientific Team produces a detailed project specification. These common standards and requirements apply to sample selection, questionnaire translation, and to all methods and processes during data collection and processing, and documentation. The household income measure is part of the question module aimed at creating a socio-demographic profile of each respondent (Section F). This module includes questions on household composition, sex, age, type of residential area, education and occupation of the respondent and of his or her partner and parents, trade union membership, and marital status. The main parts and questions in this section are stable over the consecutive survey rounds. However, the household income question was changed in Round 4 of the ESS, which was fielded in 2008. The corresponding instructions to the national coordinators who supervise the fieldwork in their respective countries were also modified. 3.1 Measurement instrument used in the European Social Survey 2002 The questionnaire used in Round 1 of the ESS 2002 featured two questions designed to measure household income. The first question (F29) asked the respondent to state the main source of income in his or her household; the second question (F30) aimed Design and Development of the Income Measures 89 to identify the income category to which the household's "total net income" belonged. To this end, the respondent was requested to "add up the income from all sources". However, in this pan-European survey, the randomly selected respondents were not given any detailed background information or explanations of the questions. Hence, it was not clear to them which income - and whose income - they should add up. No explanation or definition of "net income" was provided. Respondents were not given any help in recalling the various possible types and sources of income accruing to the household either. Because the interviewees are randomly selected from among all the members of the household aged 16 or over, and only the target person is interviewed, respondents' knowledge of the financial situation of the household as a whole varies depending on the cohort to which they belong and to their position in the household or their relationship to the main income earner/recipient. In 2002, the ESS question about the main source of income in the household read: "F29 CARD 55 Please consider the income of all household members and any income which may be received by the household as a whole. What is the main source of income in your household? Please use this card" (European Social Survey, 2002a: 49). The showcard listed seven types of income: "Wages or salaries; Income from self-employment or farming; Pensions; Unemployment / redundancy benefit; Any other social benefits or grants; Income from investment, savings, insurance or property; Income from other sources" (European Social Survey, 2002b: CARD 55). The respondent was then asked to calculate the total net income of the household and to assign it to one of the income categories presented on the showcard. "F30 CARD 56 Using this card, if you add up the income from all sources, which letter describes your household's total net income? If you don't know the exact figure, please give an estimate. Use the part of the card that you know best: weekly, monthly or annual income" (European Social Survey, 2002a: 47). The ESS Project Instructions for 2002 featured the following interviewer instruction regarding the definition of "net income". However, this information was not intended for the respondent. "At HINCTNT you should obtain the total net income of the household from all sources, that is, after tax. Income includes not only earnings but state benefits, occupational and other pensions, unearned income such as interest from savings, rent, etc. We want figures after deductions of income tax, national insurance, contributory pension payments and so on. The questions refer to cur- 90 Juergen H.P. Hoffmeyer-Zlotnik and Uwe Warner CARD 56 YOUR HOUSEHOLD INCOME Approximate Approximate Approximate WEEKLY MONTHLY ANNUAL J Less than €40 Less than €150 Less than €1800 J R €40 to under €70 €150 to under €300 €1800 to under €3600 R C €70 to under €120 €300 to under €500 €3600 to under €6000 C M €120 to under €230 €500 to under €1000 €6000 to under €12000 M F €230 to under €350 €1000 to under €1500 €12000 to under €18000 F S €350 to under €460 €1500 to under €2000 €18000 to under €24000 S K €460 to under €580 €2000 to under €2500 €24000 to under €30000 K P €580 to under €690 €2500 to under €3000 €30000 to under €36000 P D €690 to under €1150 €3000 to under €5000 €36000 to under €60000 D H €1150 to under €1730 €5000 to under €7500 €60000 to under €90000 H U €1730 to under €2310 €7500 to under €10000 €90000 to under €120000 U N €2310 or more €10000 or more €120000 or more N (European Social Survey, 2002b: CARD 56) rent level of income or earnings or, if that is convenient, to the nearest tax or other period for which the respondent is able to answer. The respondent is given a showcard that enables them to choose between their weekly, monthly or annual income, whichever they find easiest. They will then give you the letter that corresponds to the appropriate amount. This system is designed to reassure the respondent about the confidentiality of the information they are giving." (European Social Survey, 2002c: 21). In the first three rounds of the ESS 2002, 2004, and 2006, the central coordinators of the survey prescribed a common and uniform system of income categories for all participating countries for use in the response to the income question. Table 1 illustrates the survey outcomes of ESS 2002 for selected countries. The countries shown here represent different income distributions. German respondents use mainly the answer categories 4 to 9 (6,000 to 60,000 euros). The majority of respondents in the United Kingdom declare the "total net household income" using the income ranges from category 4 to 10 (6,000 to 90,000 euros) with a high frequency on the ninth income category (36,000 - 60,000 euros). The survey participants in Italy answer with the income categories 3 to 9 (3,600 to 60,000 euros). In Luxembourg the upper income categories 5 to 11 (12,000 to 120,000 euros) are used. Respondents in Portugal answer the income question with the lower categories 2 to 6 (1,800 to 24,000 euros). Interviewees in Finland use the categories 2 to 7, 11 and 12 (1,800 to 30,000, 90,000 - 120,000 and 120,000 and more euros). Design and Development of the Income Measures 91 Table 1: Distribution of "total net household income" in ESS 2002 for select ed countries Germany United Kingdom Italy % valid % % valid % % valid % 1. Up to 1,800 .47 .59 .49 .58 .44 .81 2. 1,800-3,600 .77 .98 1.20 1.42 1.09 2.02 3. 3,600-6,000 1.40 1.77 2.63 3.12 3.31 6.15 4. 6,000-12,000 6.51 8.23 10.60 12.54 10.10 18.73 5. 12,000-18,000 12.58 15.90 9.44 11.17 10.13 18.79 6. 18,000-24,000 13.37 16.90 9.35 11.06 9.24 17.13 7. 24,000-30,000 13.25 16.75 7.75 9.17 7.95 14.75 8. 30,000-36,000 9.83 12.42 8.19 9.69 3.56 6.60 9. 36,000-60,000 14.15 17.89 19.12 22.62 5.77 10.70 10. 60,000-90,000 4.95 6.25 9.42 11.14 1.50 2.78 11. 90,000-120,000 1.12 1.42 3.42 4.05 .43 .81 12. 120,000 and more .72 .91 2.91 3.45 .39 .73 77. Refusal 13.59 5.59 33.89 88. Don't know 7.31 9.88 12.20 99. No answer Total % 100.0 100.0 100.0 100.0 100.0 100.0 valid N 2308.92 1734.50 650.73 missing N 610.13 317.53 556.27 Luxembourg Portugal Finland % valid % % valid % % valid % 1. Up to 1,800 .04 .07 1.48 2.26 4.15 4.63 2. 1,800-3,600 .62 .97 5.61 8.58 12.50 13.96 3. 3,600-6,000 .41 .64 11.13 17.02 14.20 15.86 4. 6,000-12,000 .68 1.07 16.74 25.59 13.60 15.19 5. 12,000-18,000 3.06 4.82 11.79 18.03 14.35 16.02 6. 18,000-24,000 7.66 12.07 7.88 12.04 11.30 12.62 7. 24,000-30,000 10.41 16.40 3.93 6.01 15.35 17.14 8. 30,000-36,000 8.74 13.77 2.60 3.98 3.00 3.35 9. 36,000-60,000 17.44 27.48 2.81 4.30 .75 .84 10. 60,000-90,000 10.13 15.96 .88 1.34 .35 .39 11. 90,000-120,000 3.26 5.14 .32 .49 1.70 13.96 12. 120,000 and more 1.02 1.61 .23 .35 8.70 15.86 77. Refusal 18.38 8.18 .05 88. Don't know 14.72 15.03 12.50 99. No answer 3.42 11.40 14.20 Total % 100.0 100.0 100.0 100.0 100.0 100.0 valid N 984.95 988.22 1791.0 missing N 566.70 522.78 209.0 Source: ESS 2002, own calculations 92 Juergen H.P. Hoffmeyer-Zlotnik and Uwe Warner 3.2 Comparison of the results for "total net household income" from the European Social Survey 2002 and the 2001 European Community Household Panel The European Community Household Panel (ECHP) collects all the types of household income that may occur in the country in question; all household members aged 15 years or over are interviewed. All respondents are asked in detail about their income during the year prior to the interview. Hence, in the course of their involvement in the panel, respondents become experts on their personal monetary situation. The field instrument, which is designed as a personal questionnaire, lists all possible sources of monetary income. Each member of the household is able to recall and state all individually applicable income types from various sources during the interview. The 34 types of income listed by the ECHP take up over 16 pages in the personal questionnaire. In addition to the individual questionnaire for each member of the household aged 15 and older, a household questionnaire is administered to a reference person in the household who is assumed to be able to provide reliable information about income that cannot be assigned to individual members but rather accrues to the household as a whole. The household questionnaire covers 19 types of income, for example, "social assistance payment, non-cash assistance from the welfare office, income from renting property, inheritance of property or capital, a gift or lottery winnings" (European Commission/Eurostat, 2000: 25 -27). Because this survey of the income situation of the household and its members is so comprehensive and detailed, the data from Wave 8 of the ECHP (2001), with the income reference year 2000, can be used as a reference for the measurement of "total net household income" in ESS 2002. We have recoded the ECHP income values into the income categories used in the ESS. The images on the left of Figure 1 are graphical representations of the distribution of responses across income categories in ESS 2002 for the respective countries. The images on the right of Figure 1 show the grouped income distribution in ECHP8 2001. In the case of the United Kingdom, both data sources yield the same income distribution. A slight deviation is apparent in the case of Germany, while marked differences between the two statistics are evident in the case of Luxembourg. The national income distributions from the 8th wave of the ECHP - divided into groups, each of which contains 5% of the population - constitute the second step in the comparison of the "total net household income" data of the two s urveys (see Table 2). They are sorted into the income categories used as response options by the ESS. This step highlights the need to adapt the response categories of the income question to the concrete national income situation. Design and Development of the Income Measures 93 Figure 1: Distribution of "total net household income" using ESS 2002 categories: Comparison of ESS and ECHP for selected countries Source: Hoffmeyer-Zlotnik and Warner, 2014: 147 In Germany, the 15 th to the 19th 5-percent percentile of the ECHP are to be found in the 9th ESS 2002 income category (36,000 - 60,000 euros); the 10th ECHP8-2001 5-percent percentile, whose upper threshold corresponds to the median of the income distribution, is in the 7th ESS 2002 income category (24,000 - 30,000 euros). According to ECHP8-2001, only the wealthiest 5% of Portuguese households have a "total net household income" of over 36,000 euros. In Luxembourg, the ninth ESS1-2002 income category (36,000 - 60,000 euros) covers the ECHP's income distribution from the ninth to the 15th 5-percent percentile. The bottom 5% of the population in the ECHP income distribution for Luxembourg have a net household 94 Juergen H.P. Hoffmeyer-Zlotnik and Uwe Warner income of between 12,000 and 18,000 euros (the 5th ESS1-2002 category), whereas in Portugal the median (the 10th 5-percent percentile) is to be found in the fourth income category (6,000 - 12,000 euros). Table 2: Distribution of the ECHP8 5-percent percentiles across the 12 ESS 2002 income categories for selected countries Germany United Kingdom Italy Luxembourg Portugal Finland ESS Income category No. of the ECHP8 5% Percentile 1. Up to 1,800 --- --- --- --- --- --- 2. 1,800-3,600 --- --- --- --- 1-2 --- 3. 3,600-6,000 --- --- 1 --- 3-5 --- 4. 6,000-12,000 1-2 1-2 2-5 --- 6-11 1-3 5. 12,000-18,000 3-5 3-5 6-10 1 12-15 4-7 6. 18,000-24,000 6-8 6-7 1113 2-3 16-17 8-10 7. 24,000-30,000 9-12 8-10 1416 4-6 18 11-12 8. 30,000-36,000 13-14 11-12 17 7-8 19 13-15 9. 36,000-60,000 15-19 13-17 1819 9-15 --- 16-19 10. 60,000-90,000 --- 18-19 --- 16-18 --- --- 11. 90,000-120,000 --- --- --- 19 --- --- 12. 120,000 and more --- --- --- --- --- --- Source: Hoffmeyer-Z otnik anc Warner, 2014: 148 Overall, the household income of the respondents in Germany and Luxembourg is distributed across six and seven income categories respectively. However, depending on the average national income, the distribution across income categories varies significantly across countries. In Portugal, the top four response categories of the ESS should not have been used in the survey; in Luxembourg the lower four categories should have been left blank. Design and Development of the Income Measures 95 4 The improved income measure: " Total net household income" in the European Social Surveys 2008, 2010, and 2012 The modifications to the income questions in the fourth round of the ESS 2008 affected the framing of the questions, the response categories, and the showcards: "F31 CARD 72 Please consider the income of all household members and any income which may be received by the household as a whole. What is the main source of income in your household? Please use this card." (European Social Survey, 2008b: 59) The modified showcard features separate response options for "income from self -employment (excluding farming)" and "income from farming": "Wages or salaries; Income from self-employment (excluding farming) Income from farming Pensions Unemployment/redundancy benefit Any other social benefits or grants Income from investment, savings, insurance or property Income from other sources" (European Social Survey, 2008b: 59) The text of the "total household income" question gives the respondent an indication of what is meant by "net", as it specifies "after tax and compulsory deductions". "Using this card, please tell me which letter describes your household's total income, after tax and compulsory deductions, from all sources? If you don't know the exact figure, please give an estimate. Use the part of the card that you know best: weekly, monthly or annual income." (European Social Survey, 2008b:60) Since the fourth round of the ESS, each participating country has drafted its own showcard. The response categories are based on the deciles of the actual household income distribution in the country in question. In a note on the drafting of the decile income showcard, the ESS coordinators provide the following instructions to those responsible for running the survey in each country: "An income showcard should be devised with approximate weekly, monthly and annual amounts. You should use ten income range categories, each corresponding broadly to DECILES OF THE ACTUAL HOUSEHOLD INCOME RANGE in your country. These figures should be derived from the best available source for your country. The data source used should match the requirement of the question i.e. deciles of household income for all households (not for example average households or just households with children). Using the median income as 96 Juergen H.P. Hoffmeyer-Zlotnik and Uwe Warner the reference point, 10 deciles should be calculated with the median itself at the top of the fifth decile (Category F). The figures should not appear to be too exact. Minor rounding can be employed to achieve this if necessary" (European Social Survey, 2008b: 60; see also European Social Survey, 2008c: 17). Approximate WEEKLY J Weekly equivalent M F Weekly equiva Weekly equiva Weekly equiva Weekly equiva S Weekly equiva K Weekly equiva P Weekly equiva D Weekly equiva H Weekly equivalent ent ent ent ent ent ent ent ent CARD 73 YOUR HOUSEHOLD INCOME Approximate MONTHLY Monthly equiva Monthly equiva Monthly equiva Monthly equiva Monthly equiva Monthly equiva Monthly equiva Monthly equiva Monthly equiva Monthly equiva ent Approximate ANNUAL Income corresponding to that held by 10% J of households with lowest income (0-10%) ent Income corresponding to that held by next 10% of R households (11-20%) ent Income corresponding to that held by next 10% of C households (21-30%) ent Income corresponding to that held by next 10% of M households (31-40%) ent Income corresponding to that held by next 10% of F households (41-50%) ent Income corresponding to that held by next 10% of S households (51-60%) ent Income corresponding to that held by next 10% of K households (61-70%) ent Income corresponding to that held by next 10% of P households (71-80%) ent Income corresponding to that held by next 10% of D households (81-90%) ent Income corresponding to that held by next 10% of H households (91-100%) Source: European Social Survey, 2008b: CARD 73 R C Twenty-six countries participated in Round 4 of the ESS 2008. Figure 2.1 shows the country-specific distributions of the responses across the 10 income categories. The medians of the income distributions of 15 countries lay in the fifth or sixth income category, thereby fulfilling the ESS requirement quoted above. In six countries, the medians were in a category above the sixth income category, while in five countries the medians of the distribution were in a category below the fifth category. In most of the countries, 50% of the observed household incomes were spread over four or five categories. As the response options are based on the deciles of the national income distribution, we expected that approximately 50% of the surveyed population would use five categories to answer the ESS income question. However, in the Czech Republic half of the respondents used only two income ranges, while in Portugal three categories were used by the surveyed persons. Figure 2.2 shows the country-specific distributions of the responses in Round 5 of the ESS, which was fielded in 2010. As in the previous round, 26 countries participated. The medians of the income distributions in 22 countries lay in the fifth or sixth Design and Development of the Income Measures 97 income category. In one country the median was in a category above the sixth income category, while in three countries the medians of the distribution were in a category below the fifth category. Figure 2.1: Country-specific distributions of responses across the ten income categories in ESS 2008 Cases weighted by Design weight Figure 2.2: Country-specific distributions of responses across the ten income categories in ESS 2010 Country Cases weighted by Design weight 98 Juergen H.P. Hoffmeyer-Zlotnik and Uwe Warner Figure 2.3: Country-specific distributions of responses across the ten income categories in ESS 2012 10- ui Country Cases weighted by Design weight Source for figures 2.1 -2.3: ESS 2008, 2010, 2012, own calculations. The solid horizontal line shows the expected median Figure 2.3 illustrates the distributions in 2012. Data from 24 countries were available for the analysis. The medians of 13 countries lay in the fifth or sixth income category. In four countries, the medians were in a category above the sixth income category, while in seven countries the medians of the distribution were in a category that was lower than expected. In Bulgaria, seven income categories were used by 50% of the surveyed population, whereas in Portugal, Kosovo, and the Czech Republic only three options were used. As the income categories on the showcard for the income question correspond to the deciles of the actual household income range, it is to be expected that in a representative survey with a probabilistic sample each response category will be selected by approximately 10% of the survey population. As can be seen from the countries presented by way of example in Figure 3, our expectation was fulfilled in some cases, but not in others. In ESS 2008, for example, each income category was chosen by almost 10% of respondents in Denmark, Estonia, Finland, France, the United Kingdom, Croatia, Poland, and Slovenia. However, medium-sized deviations from the expected decile distribution were observed in the case of Switzerland, Germany, Spain, Greece, Hungary, the Netherlands, Norway, the Ukraine, and Ireland, where the middle income categories were more strongly represented than expected. In 2008, large deviations from the decile distribution were observed in Belgium, the Czech Republic, Latvia, Portugal, Romania, Russia, Sweden, and Turkey. Design and Development of the Income Measures 99 Figure 3: Distributions across the income categories in ESS 2008, ESS 2010 and ESS 2012 for selected countries ESS 2008 ESS 2010 ESS 2012 1. Belgium 2. United Kingdom 3. Poland 4. Slovenia 100 Juergen H.P. Hoffmeyer-Zlotnik and Uwe Warner ESS 2008 ESS 2010 ESS 2012 5. Germany Source: ESS 2008, 2010, 2012, own calculations. The solid horizontal line shows the expected 10% responses, error bars = 95% confidence interval In the Belgian ESS 2008, the two highest deciles show large deviations from the expected 10% mark (Figure 3, row 1). The highest income category starts at 35,000 euros. However, 33,731 euros is the upper threshold of the 60% decile of EU-SILC4 in Belgium (Table 2). Therefore, considerably more than the expected 10% of the respondents in Belgium chose the ninth and tenth income categories during the ESS interview. The lower income categories were not used to the expected extent by the ESS 2008 respondents. The EU-SILC reports the threshold of the lowest decile at 12,012 euros, which corresponds to the fourth income category on the showcard used by Belgium in ESS 2008 (Tables 3 and 4). Table 3: EU-SILC 2008 "total disposable household income" decile thresholds in Euros for Belgium lowest 10% 20% 30% 40% 50% 60% 70% 80% highest 90% 12,012 15,191 18,741 22,837 27,683 33,731 40,012 47,386 59,951 Source: EU-SILC USER DATABASE Version of 01-08-11, own calculations 4 The European Union Statistics on Income and Living Conditions (EU-SILC) are comparable multidimensional micro-data on income, social inclusion, and living conditions. They cover objective and subjective aspects of these themes in both monetary and non-monetary terms for both households and individuals. They are used to monitor the progress of the Europe 2020 strategy - in particular its headline target on poverty reduction, which provides information on income, poverty, social exclusion, housing, labour, education and health. http://epp.eurostat.ec.europa.eu/portal/page/ portal/income_social_inclusion_living_conditions/introduction# Design and Development of the Income Measures 101 Table 4: Income distribution in Belgium according to the tax register and the income brackets used in the Belgian ESS 2008 DecilesTotal taxable net Average tax (Total taxable net income income from paid in % from register)-(Average register tax paid) Rounded net income as appeared on Showcard 72 1 4,909 0 4,909.000 Less then 5,000 € 2 9,677 1.5 9,531,845 5,000 € to 10,000 € 3 12,001 2.3 11,724,977 10,000 € to 12,000 € 4 14,860 7.9 13,686,060 12,000 € to 14,000 € 5 18,139 12.5 15,871,625 14,000 € to 16,000 € 6 21,816 17.9 17,910,936 16,000 € to 18,000 € 7 26.457 21.2 20,848,116 18,000 € to 21,000 € 8 34,146 24,3 25,848,522 21,000 € to 26,000 € 9 47,834 27.5 34,679,650 26,000 € to 35,000 € 10 >47,834 >27.5 >34,679,650 35,000 € or more Source: European Social Survey 2008d: 3 Table 5: Components of taxable income in Belgium Source: Algemene Directie Statistiek en Economische Informatie, 2009: 36, translation by the authors 102 Juergen H.P. Hoffmeyer-Zlotnik and Uwe Warner The ESS4-2008 Survey Documentation (European Social Survey, 2008d: 38) reported that the income response categories for Belgium were calculated on the basis of "total taxable net income" data from the "Tax statistics for revenues of 2004". The responses in Belgium gave rise to major deviations from the expected 10% mark in all ten response categories. In Belgium, taxable income is made up of wages and salaries, income from self-employment, pensions, unemployment benefit, sickness and disability benefit, income from the rental of property and land, income from investments, and income from property and other sources (Table 5). However, because the ESS measures "total net household income", and many components of household income - for example public and private transfers - are not subject to tax, it is obvious that the lower response categories in Belgium were either not used at all or hardly used. The Polish showcard for the answers to the income question in ESS4-2008 is based on the income distribution of the Polish Household Budget Surveys. In ESS4-2008, the lower two income categories were underrepresented and did not reach the 10% mark; the highest two income deciles were used more often than expected (Figure 3, row 3). We observed no significant changes in the answers to the income questions in the subsequent rounds. The ESS 2008 respondents fulfilled expectations in the United Kingdom, as only the highest decile was overrepresented and less than 10% chose the middle income categories (Figure 3, row 2). Response behaviour was similar in the subsequent ESS rounds. The showcards containing the income categories are based on the data from the Family Resources Surveys. Figure 3, row 4 shows that in ESS 2008 in Slovenia the 10% requirement was fulfilled in the case of the majority of the ten income categories. The lowest category was overrepresented and the highest answer option was not used by the expected 0% of the eligible respondents (Table 6). Table 6: Slovenia: Observed and expected responses across the ten income categories 2008 2010 2012 Observed N Expected N Residual Obs. N Exp. N Resid. Obs. N Exp. N Resid. 1 172 102.0 70.0 119 107.7 11.3 96 96.7 -.7 2 102 102.0 .0 170 107.7 62.3 121 96.7 24.3 3 77 102.0 -25.0 143 107.7 35.3 152 96.7 55.3 4 98 102.0 -4.0 143 107.7 35.3 144 96.7 47.3 5 90 102.0 -12.0 109 107.7 1.3 130 96.7 33.3 6 120 102.0 18.0 96 107.7 -11.7 85 96.7 -11.7 7 109 102.0 7.0 99 107.7 -8.7 85 96.7 -11.7 8 95 102.0 -7.0 70 107.7 -37.7 64 96.7 -32.7 9 100 102.0 -2.0 67 107.7 -40.7 49 96.7 -47.7 10 57 102.0 -45.0 61 107.7 -46.7 41 96.7 -55.7 Total 1020 1077 967 Source: ESS, 2008, 2010, 2012, own calcu ations Design and Development of the Income Measures 103 However, in the case of all other categories, the 10% benchmark was within the 95% confidence interval of equal distribution over the 10 income ranges. This changed in 2010 and 2012. In 2008, Slovenia showed small deviations from the 10% equal distribution over the income categories. The income categories were based on the 2007 Census figures provided by the Statistical Office of the Republic of Slovenia (European Social Survey, 2008d: 253). In 2010 and 2012, the deviations were deemed to be medium, and the reference statistics for the creation of income categories were the EU-SILC (European Social Survey, 2010: 27; 2012: 27). In ESS 2008 in Germany, categories 3, 4, 5, and 7 to 10, and their corresponding 95% confidence intervals, did not respect the 10% benchmarks (see Figure 3, row 5). For the 2010 and 2012 rounds of the ESS, the German ESS coordinators changed the statistical basis for the creation of the income categories. They switched from the 2003 Einkommens- und Verbrauchsstichprobe (Income and Consumer Survey)5 (European Social Survey 2008 e: 30 f.) as reference statistics to the 2008 and 2011 Mikrozensus (a 1% population census)6 respectively (European Social Survey, 2010b: 13 and European Social Survey, 2012b: 12), with the result that the deviations from the equal distribution over ten categories in 2010 and 2012 were deemed to be small (see Table 7). Table 7: Germany: Observed and expected responses across the ten income categories 2008 2010 2012 Observed N Expected N Residual Obs. N Exp. N Resid. Obs. N Exp. N Resid. 1 269 228.6 40.4 170 226.9 -56.9 208 255.3 -47.3 2 289 228.6 60.4 229 226.9 2.1 271 255.3 15.7 3 322 228.6 93.4 259 226.9 32.1 246 255.3 -9.3 4 329 228.6 100.4 224 226.9 -2.9 265 255.3 9.7 5 297 228.6 68.4 273 226.9 46.1 306 255.3 50.7 6 226 228.6 -2.6 241 226.9 14.1 252 255.3 -3.3 7 201 228.6 -27.6 231 226.9 4.1 302 255.3 46.7 8 159 228.6 -69.6 246 226.9 19.1 246 255.3 -9.3 9 108 228.6 -120.6 189 226.9 -37.9 224 255.3 -31.3 10 86 228.6 -142.6 207 226.9 -19.9 233 255.3 -22.3 Total 2286 2269 2553 Source: ESS, 2008, 2010, 2012, own calcu ations In 2008, four countries used the EU-SILC as the basis for calculating the household income deciles; fourteen countries calculated the household income ranges on the basis of other survey data; and eight countries derived the income deciles from population registers or census data (Table 8). 5 The German Income and Consumer Survey is a quota sample realized every five years. 6 The German 1% population census is a random sample of households. Participation in the survey is mandatory for the household members selected. 104 Juergen H.P. Hoffmeyer-Zlotnik and Uwe Warner Table 8: Sources of the income distribution used to create the ESS income categories on the showcards, and deviations from the 10% frequencies per category 2008 2010 2012 country deviation source deviation source deviation source Belgium large register medium SILC small SILC Bulgaria large register large register Croatia small survey medium survey Switzerland medium survey small survey small survey Cyprus large SILC large SILC Czech Rep large SILC medium SILC medium SILC Germany medium survey small census small census Denmark small register medium register small register Estonia small SILC medium SILC small survey Spain medium survey medium survey large survey Finland small survey small survey small survey France small census small survey UK small survey small survey small survey Greece medium SILC medium SILC Hungary medium survey medium survey small survey Israel medium survey large survey small survey Kosovo large survey Latvia large SILC Lithuania large survey Netherlands medium register small register medium register Norway medium register small register small register Poland small survey small survey medium survey Portugal large survey large SILC Romania large survey Russian Fed large survey small survey large survey Sweden large register large survey medium survey Slovakia medium SILC small SILC Slovenia small census medium SILC medium SILC Turkey large survey Ukraine medium survey small survey Ireland medium survey large SILC large SILC Iceland large survey Note: A deviation is deemed to be large if at least one response category deviates by at least 10 percentage points from the expected 10 percent mark. A deviation is considered to be medium if at least one response category deviates by at least 5 percentage points from the ten percent mark. De -viations of 2.5 percentage points from the expected 10 % share are deemed to be small. Source: Data Documentation Reports of ESS, 2008, 2010 2012 available as Survey Documentation Report from http://www.europeansocialsurvey.org/data/round-index.html [accessed 21 March 2014], own calculations Design and Development of the Income Measures 105 5. Conclusion In this article we have focused on the ESS question about the "total net household income". In Hoffmeyer-Zlotnik and Warner, 2014 (228 ff.), we presented a proposed survey instrument of our own for obtaining information about this background variable. As average income levels and income distributions differ in various types of European countries, the response categories must be adapted to the national income situation of the surveyed country. The quality of the responses to the survey question about "total net household income", and ultimately the quality of the survey data obtained, depends on the quality of the reference statistics from which the household income ranges for the answers are derived.7 The example from Belgium shows the increase in the quality of income data collected in the ESS that can be achieved by changing the reference statistics. In 2008, the national coordinators used the tax register with limited income information to design the income categories on the showcard, which resulted in large deviations from the expectation that each category would be chosen by 10% of respondents. In 2010 and 2012, Belgium's national coordinators used the income data provided by EU-SILC, thereby reducing the deviations from the expected responses. The German example shows a different case. In 2008, the national coordinators based the construction of th e "total net household income" categories on the Income and Consumer Survey (Einkommens- und Verbrauchsstichprobe). In the 2010 and 2012 rounds of the ESS they used the Mikrozensus (a 1% population census) as reference statistics for the income ranges offered to the respondents. As a result, a larger number of response categories were closer to the expected 10% benchmark. This leads us to conclude that the quality of income measurements for comparative social surveys depends on the elaborateness with which the response options are created. This, in turn, depends on the quality of the reference statistics about the "total net household income" used to determine the income ranges on the showcard presented to the respondents. Other possible effects are related to the use of scales in surveys: country specific practices in survey design and the culture specific answer behaviour of survey participants. The stability of the national income distribution over time may have an impact on the quality of income measurement in social surveys. 7 Other survey errors also have an impact on the objectivity, reliability, validity, and measurement quality of the total net household income data. They include non- or undercoverage of the sampling frame, systematic unit-non-response, non-random item-non-response, national or cultural attitudes towards highly sensitive survey topics, opinions about the data protection and the privacy of personal information, the interpersonal relation between interviewer and interviewee, translation error, inadequate weighting and wrong extrapolation factors, faulty replacements and weak imputations of missing data, ultimately doubtful and dubious equivalences of the information across cultures and countries, etc. (cf. Groves and Lyberg, 2010 for survey errors in general, for the problem of equivalences in cross cultural and cross national surveys cf. Johnson, 1998, and in particular for errors in comparative survey research cf. Braun, 2003). 106 Juergen H.P. Hoffmeyer-Zlotnik and Uwe Warner To date, a total of seventy-six national datasets have been collected over all rounds of the ESS (see table 9). In nineteen cases, the income categories were based on the national EU-SILC data. Only three (15.7%) of these datasets show small deviations from the national SILC data. Forty datasets used national surveys; 17 (42.5%) of these datasets show small deviations. Thirteen countries used register data and four countries used census data to determine the response options for the ESS interviews. Table 9: Reference statistics used to create the income categories, by deviation Deviation Total Large Medium Small National Source SILC (n) 7 9 3 19 (%) 36.8 47.3 15.7 100.0 Survey (n) 11 12 17 40 (%) 27.5 30.0 42.5 100.0 Register (n) 4 4 5 13 (%) 30.7 30.7 38.4 100.0 Census (n) 0 0 4 4 (%) 0.0 0.0 100.0 100.0 Total (n) 22 25 29 76 (%) 28.9 32.8 38.1 100.0 Note: A deviation is deemec to be large if at least one response category deviates by at least 10 percentage points from the expected 10 percent mark. A deviation is considered to be medium if at least one response category deviates by at least 5 percentage points from the ten percent mark. Deviations of 2.5 percentage points from the expected 10 % share are deemed to be small. Source: ESS Data Documentation Reports and own calculations 6. Recommendations Data from national sources have to fulfil quality criteria if they are to be used as reference statistics to establish the income categories for the survey question on household income (see Ehling and Körner 2007: 9 f.). The first requirement is the relevance of the source data for the household's income information - that is, the degree to which the reference data meet the needs of the survey. This is expressed in the following instruction to the national coordinators of the ESS: "You should use ten income range categories, each corresponding broadly to DECILES OF THE ACTUAL HOUSEHOLD INCOME RANGE in your country. These figures should be derived from the best available source for your country. The data source used should match the requirement of the question ...". (European Social Survey, 2008a: 60; see also European Social Survey, Design and Development of the Income Measures 107 2008b: 17). The formulated question adds "... all household members and any income" and income "from all sources". The second quality component for reference data is accuracy. Accuracy describes the correspondence of the income values in the reference statistics with the (unknown) true income distribution in the entire population. The next quality criterion is timeliness, which refers to the time between the date of the reference data on income and the date of the survey measurement of income in the field. The requirement mentioned by the ESS is "the actual household income" at the time of the interview. However, it is evident that there is a time lag between the date of the official income information and the start of the survey fieldwork. The fourth quality domain is comparability. The concepts of "total net household income" applied in the reference statistics and in the survey instruments must be as similar as possible. Deviations between the reference data and the outcomes of the survey are comparable if they are not due to the tools applied during data collection. A comparison of the income distribution yielded by the survey with the income distribution of the reference statistics is meaningful. The information from the reference statistics must be coherent with the intended survey measurements. The main issue with regard to coherence is that both the reference statistics and the survey use the most similar approaches, classifications, and methodological standards possible when op erationalising "total net household income". Finally, the accessibility and clarity of the reference statistics are important. The researchers who prepare, design, and organise the fieldwork of the surveys must be able to find the income information necessary to create the income categories for the survey question about "total net household income". This presupposes not only ac -cess to the individual data from the reference statistics, clear documentation about the data production, and metadata about the source of the reference statistics. The major elements of clarity are the corresponding quality reports by the providers of the reference statistics. We recommend that, first, the central coordinators of the ESS define and communicate minimum threshold values for each of the aforementioned quality domains. Second, the national coordinators have the best knowledge about the statistical sources available in their country. A close collaboration with experts on the national income situation of households is indispensable to select the reference statistics and to draft the showcard for the question about "total net household income". Together they decide about the best available reference statistics taking the guidelines from the central coordinators into account. Ideal are national surveys collecting income data with similar technology and instruments like ESS, e.g. as a module in the annual Labour Force Surveys. Today, only a small number of countries include "total net household income" in the national Labour Fo rce Surveys. 108 Juergen H.P. Hoffmeyer-Zlotnik and Uwe Warner Third, the national coordinators responsible for the design of the questionnaire and the measurement instrument implemented in the survey should report their decision and justify the selection of the reference statistics. The documented national deviations from the guidelines of the central coordinators and the reference statistics on the income become transparent to the users of ESS. The users of ESS data have now an insight into the quality of the income measurement during the interviews. If these recommendations are followed, we expect more and better cross-country comparisons using the socio-demographic background variable "total net household income". Acknowledgements The authors wish to thank CEPS/INSTEAD, Esch, Alzette for financial support. References [1] Algemene Directie Statistiek en Economische Informatie (2009): Levensstandaard Fiscale statistiek van de inkomens. Aanslagjaar 2007 - Inkomens van 2006. Brüssel economie.fgov.be/nl/binaries/Inkomens%20brochure%20B%20aanslagjaar%2020 07_tcm325-78903.pdf [accessed 21 March 2014] [2] Biemer, P. and Lyberg, L. (2003): Introduction to Survey Quality. Hoboken, NJ: Wiley. [3] Braun, M. (2003): Errors in Comparative Survey Research: An Overview. In: J. A. Harkness, van de Vijver, J.R. F. and Mohler, P. Ph. (Eds.) Cross-cultural survey method: 137-142. Hoboken, NJ: Wiley. [4] Braun, M. and Mohler, P. Ph. (2003): Background Variables. In: J. A. Harkness, van de Vijver, J.R. F. and Mohler, P. Ph. (Eds.): Cross-cultural survey method: 101-116. Hoboken, NJ: Wiley. [5] Carlsson, G. (1968): Ökonomische Ungleichheit und Lebenschancen. In: Glass, David V.; König, R. (Eds) Soziale Schichtung und soziale Mobilität. KZfSS Sonderheft 5. [6] Ehling, M. and Körner, T. (Eds.) (2007): Handbook on Data Quality Assessment Methods and Tools. Wiesbaden: Statistisches Bundesamt. [7] European Social Survey (2002a): Source Questionnaire. Round 1, 2002. ESS Document Date 01-08-02. http://www.europeansocialsurvey.org/docs/round1/fieldwork/source/ESS1_source _main_questionnaire.pdf [accessed 21 March 2014]. [8] European Social Survey (2002b): Source Showcards. Round 1, 2002. http://www.europeansocialsurvey.org/docs/round1/fieldwork/source/ESS1_source _showcards.pdf [accessed 21 March 2014]. Design and Development of the Income Measures 109 [9] European Social Survey (2002c): Project Instructions (PAPI). Round 1, 2002. ESS Document date 15/07/02. http://www.europeansocialsurvey.org/docs/round1/fieldwork/source/ESS1_source _project_instructions.pdf [accessed 21 March 2014]. 10] European Commission/Eurostat (2000): ECHP 2001 Wave 8. Variable List DOC. PAN 159/00. 11] European Social Survey (2008a): The European Social Survey, Source Questionnaire Amendment 03 (Round 4, 2008/9). ESS Document Date 08-08-08. http://ess.nsd.uib.no/ess/round4/fieldwork.html. [accessed 18 October 2012]. 12] European Social Survey (2008b): ESS 2008 Data Protocol Edition 1.2, October 2008. ESS4_data_protocol_eL2.pdf. http://ess.nsd.uib.no/ess/round4/. [accessed 18 October 2012]. 13] European Social Survey (2008c): Data Documentation Report. Ed. 5.1. Appendix A5 http://www.europeansocialsurvey.org/docs/round4/survey/ESS4_appendix_a5_e05 _0.pdf [accessed 18 December 2013] 14] European Social Survey (2008d): ESS4 - 2008 Documentation Report Edition 5.2 http://www.europeansocialsurvey.org/data/download.html?r=4 [accessed 18 October 2014] 15] European Social Survey (2010a): Data Documentation Report. Ed. 3.0. Appendix A2 http://www.europeansocialsurvey.org/docs/round5/survey/ESS5_appendix_a2_e03 _0.pdf [accessed 18 December 2013] 16] European Social Survey (2010b): ESS5 - 2010 Documentation Report Edition 3.1 Appendix A2, Income, ESS5 - 2010 ed. 3.0 http://www.europeansocialsurvey.org/data/download.html?r=5 [accessed 18 October 2014] 17] European Social Survey (2012a): Data Documentation Report. Ed. 1.3. Appendix A2 http://www.europeansocialsurvey.org/docs/round6/survey/ESS6_appendix_a2_e01 _1.pdf [accessed 18 December 2013] 18] European Social Survey (2012b): ESS6 - 2012 Documentation Report Edition 2.0 Appendix A2, Income, ESS5 - 2012 ed. 2.0 http://www.europeansocialsurvey.org/data/download.html?r=6 [accessed 18 October 2014] 19] Groves, R. and Lyberg, L. (2010): Total survey error: Past, present and future. Public Opinion Quarterly, 74 (5), 849-879. 20] Hoffmeyer-Zlotnik, J. H.P. and Warner, U. (2006): Methodological Discussion of the Income Measure in the European Social Survey Round 1; in: Metodoloski zvezki Vol. 3, No. 2: 289-334. 21] Hoffmeyer-Zlotnik, J. H.P. and Warner, U. (2014): Harmonising Demographic and Socio-Economic Variables for Cross-National Comparative Survey Research. Dordrecht, Heidelberg, New York, London: Springer Science + Business Media. 22] Hradil, S. (2006): Die Sozialstruktur Deutschlands im internationalen Vergleich. 2. Auflage. Wiesbaden: VS-Verlag. 110 Juergen H.P. Hoffmeyer-Zlotnik and Uwe Warner [23] Johnson, T. P. (1998): Approches to Equivalence in Cross Cultural and Cross-National Survey Research. In: Harkness, Janet (Ed.): Cross-Cultural Survey Equivalence. ZUMA-Nachrichten Spezial 3. Mannheim: ZUMA [24] Lepsius, R. M. (1993): Demokratie in Deutschland. Soziologisch-historische Konstellationsanalysen. Ausgewählte Aufsätze. Göttingen: Vandenhoeck & Ruprecht. [25] Lepsius, R. M. (1974): Sozialstruktur und soziale Schichtung in der Bundesrepublik Deutschland. In: Löwenthal, R. and Schwarz, H.-P. (eds). Die zweite Republik. 25 Jahre Bundesrepublik Deutschland - eine Bilanz. Stuttgart: Seewald. [26] Lynn, P. (2001): Developing Quality Standards for Cross-National Survey Research: Five Approaches. ISER Working Papers Number 2001-21 www.iser.essex.ac.uk/_er_working_papers/2001-21.pdf [accessed 1 October 2014] [27] ONS (2013): Guidelines for Measuring Statistical Output Quality, Version 4.1. ONS, London. http://www.ons.gov.uons method / guide- /method-quality/quality/guidelines-for k/- -quality/index.html [accessed 1 October 2014] [28] Rainwater, L., Rein, M. and Schwartz, J. (1987): Income Packaging in the Welfare State: A Comparative Study of Family Income. Oxford University Press. [29] Smith, T. W. and Yang-Chih F. (2014): The Globalization of Surveys. GSS Cross-national Report No. 34. publicdata.norc.org/gss/documents/CNRT/CNR34.pdf [accessed 1 October 2014] [30] Statistics Canada (1998): Statistics Canada Quality Guidelines. 3rd edition, Ottawa: Statistics Canada. [31] United Nations Economic Commission for Europe (2011): Canberra Group Handbook on Household Income Statistics. Second Edition. Geneva: United Nations [32] Warner, U., (2012): 'Total Net Household Income' as Demographic Standard Variables for Social Survey; in: Hoffmeyer-Zlotnik, J. H.P. and Warner, U. (Eds.): Demographic Standards for Surveys and Polls in Germany and Poland - National and European Dimension. GESIS-Schriftenreihe Band 10, 187-208. Köln: GESIS - Leibniz-Institut für Sozialwissenschaften.