Metodološki zvezki, Vol. 3, No. 2, 2006, 355-368 The Quality of Survey Questions Published on Slovenian Journalistic Web Sites Simon Ličen1, Katja Lozar Manfreda2, and Valentina Hlebec3 Abstract Some journalistic web sites regularly publish survey questions or polls on current public opinion issues. The goal of these surveys is primarily to entertain the visitors and increase the number of visits. However, these surveys may also shape public awareness and opinion. Therefore, the quality of these survey questions is not to be neglected. We studied the quality of survey questions and response alternatives published on three Slovenian journalistic web sites. The quality of the analyzed survey questions and response alternatives was evaluated using an expert evaluation scheme as checklist. The most striking deficiencies of both survey questions and sets of response alternatives are presented and described. We show that the quality of these questions is often low in comparison to professional survey questions that are used in social science research. 1 Introduction In the last few years journalistic web sites have become increasingly popular. Such web sites, both Slovenian and foreign, are usually related to already existing media4 – newspapers, radio and television stations, etc. Examples of such web sites are www.delo.si in Slovenia and www.nytimes.com in the USA. Some of these media web sites feature short survey questions regarding current public opinion issues. A survey is a “systematic method for gathering information from (a sample of) entities for the purposes of constructing quantitative descriptors of the attributes of the larger population of which the entities are members” (Groves et al., 2004: 2). 1 Faculty of Sport, University of Ljubljana, Gortanova 22, 1000 Ljubljana, Slovenia; Simon.Licen@fsp.uni-lj.si. 2 Faculty of Social Sciences, University of Ljubljana, Kardeljeva pl. 5, 1000 Ljubljana, Slovenia; Katja.Lozar@fdv-uni-lj.si. 3 Faculty of Social Sciences, University of Ljubljana, Kardeljeva pl. 5, 1000 Ljubljana, Slovenia; Valentina.Hlebec@fdv.uni-lj.si. 4 Thus becoming “internet editions” of already existing media. 356 Simon Ličen et al. Short survey questions published on journalistic web sites can thus be a suitable method for researching public opinion. However, this definition identifies several potential problems that impact on specific survey research areas: systemacy, sampling and generalization, among others. Surveys appearing on journalistic web sites are somehow different from other surveys. Couper (2000: 477) studied the various types of web surveys. In his typology, he took into consideration two main criteria: - the probability of the selected sample of respondents and - the type of invitation given to participants in the survey. According to these criteria, Couper classified web surveys on journalistic web sites among the group of “web surveys as entertainment”. The main characteristics of such surveys are as follows: they are nonprobability surveys; they are published on web sites and as such open for everyone to participate; they are thus surveys with general invitation that are related to current public opinion issues. All such surveys feature close-ended questions and thus have fixed response alternatives. We mentioned several potential problems arising from the definition of a survey. These problems are related to as many potential errors in surveys. The main division is between “measurement errors” or “errors of observation” and “errors of nonobservation”. Errors of nonobservation occur because the statistics are measured on a sample instead of a population. Measurement errors represent the difference between the answer to a survey question and the actual measured attribute (Lozar Manfreda, 2001: 66-79 and Groves et al., 2004: 40). Web surveys are nonprobability surveys. They are thus inevitably open to errors of nonobservation. However, this error is acknowledged and accepted. The primary goal of the web surveys is to entertain the visitors of the web site and not to provide a realistic picture of public opinion. Measurement errors can happen as well. Such errors most often occur because of the respondents or because of the question(naire) (Lozar Manfreda, 2001: 66-88). Respondent errors can hardly be prevented, so surveyors can minimize only errors arising from the questionnaire. Survey questions should thus be tested before they are published (Lindström, 2004: 17-18 and Hlebec, 2005). Issues that need special attention when designing a questionnaire have been mentioned in several studies. Among those, Lozar Manfreda, Batagelj and Vehovar emphasized two aspects as “particularly important as regards the design of the Web survey questionnaire: reducing the measurement error and the non-responsive error affected by the design of the Web survey questionnaire” (Lozar Manfreda, Batagelj & Vehovar, 2002:2). The authors claim that little is known about the basic standards for designing Web questionnaires; however, this is not surprising for the vast majority of non-professional Web surveys (which includes surveys on journalistic web sites) (Lozar Manfreda, Batagelj and Vehovar, 2002:3). The Quality of Survey Questions… 357 Two primary goals of these web surveys or polls, as they are often called5, is to entertain visitors to the web sites and to increase the number of visits. They can also be a useful tool to somehow “check the pulse” regarding selected issues among the public as well as being an efficient tool for expressing and exchanging opinions among web site users (Couper, 2000: 278 and Lozar Manfreda, 2001: 29). Public opinion surveys represent the opinion of the public (sometimes we might say “the opinion of the masses”). The public thus shapes and “produces” public opinion. However, we ascertained that the relation between the public and public opinion is a two-way interaction. The influence thus works in the opposite way as well (see e.g. Splichal, 1997: 190). So even though the primary goal of web surveys is to entertain web site visitors, they might fail to take into account the survey’s lack of scientific grounding. The results of such surveys could thus be perceived as the actual opinion of the public regarding the issue at hand. This is true for every type of expression of public opinion, such as in newspapers, on television, or, in our case, on a web site. Knowing this mechanism and manipulating web survey results could lead to the manipulation of public opinion. However, the results of web surveys can also be influenced by potential survey errors that prevent the achievement of realistic results. Both applications of surveying are inammissible and must be avoided. This is why this area of research should not be neglected. In our research we studied and evaluated the quality of survey questions and response alternatives published on three Slovenian journalistic web sites. 2 Methodology We decided to study public opinion surveys published on Slovenian journalistic web sites. The first step was to make a list of all Slovenian media web sites – daily and weekly newspapers, radio and television stations and exclusively web media6. Then we selected the web sites that featured regularly updated public opinion surveys, and finally narrowed our research focus to the web sites that had a publicly accessible archive of the surveys. Three journalistic web sites met the requested conditions: www.24ur.com7, www.rtvslo.si8 and www.dnevnik.si9. 5 The difference between the terms “poll” and “survey” has been analyzed by Schuman (1997). While no clear distinctions between the meanings of the two terms are acknowledged, Schuman establishes that the first term is most often used for private sector opinion studies, while the second is more common in government and scientific domains. Regardless of Schuman’s distinction, we will refer to our research target as (web) survey. 6 Such media publish their contents and are accessible exclusively through the World Wide Web (Trošt, 2003: 43). 7 The web site of a daily news programme on a commercial TV network. 8 The Slovenian public radio and TV service. 358 Simon Ličen et al. 2.1 Characteristics of the web sites After describing several characteristics of these web sites (contents, number of published surveys, survey characteristics, average duration, vote-casting method, etc.), we established a few basic parameter for the published surveys: the number of polls published between July 1st, 2004 and July 1st, 2005 (N), their average duration in days (DUR), the average number of participants (respondents) in each survey (RES), the average number of participants per day (RES/DAY), the number of response alternatives, the format of response alternatives10 and the frequency of a “Don’t know/Don’t care” answer with relation to N (DK-N). Some of these parameters are presented in the following chart: Table 1: Some basic parameters of the web sites analyzed. Parameter www.24ur.com www.rtvslo.si www.dnevnik.si N 253 159 302 DUR 4.04 days 3.32 days 1 day RES 2501 1164 301 RES/DAY 619 350 301 DK-N 57.8% 76.1% 27.5% The short surveys published on the web sites we analyzed displayed a number of formal characteristics in common. All the surveys were single-question polls, usually related to a recent or current event or issue. A variety of topics was addressed, and the media themselves covered different topics with different frequency. The most commonly addressed topic was domestic politics, with 20.5 percent of all the surveys published over the course of the sample year. Domestic politics was followed by surveys regarding topical events in Slovenia (e.g. a dispute regarding Roma children in a Slovenian elementary school), which appeared in 12.4 percent of all surveys. Foreign politics did not receive particular attention, as only 7.6 percent of all the surveys published addressed this topic. None of the three web sites control access to their surveys. Ballot-stuffing is thus not prevented. All three web sites offer the current survey results immediately after the vote casting process. The results are available throughout the period the survey is 9 A national daily newspaper. 10 We divided the response alternatives into three groups: “Yes-No” responses, “Yes-No with argumentation” responses and “Attitude” (or “Pick-the-One-that-Fits-best”) responses. The Quality of Survey Questions… 359 published. After it expires, all three web sites move its results to a survey archive where data from all past surveys can be accessed 11. None of the web sites posted a disclaimer explaining the applied methodology or the validity of the conclusions that might be drawn from such survey results12. Furthermore, none of the analyzed media publish news based exclusively or mainly on the results of these short web surveys. In some cases it is quite the opposite: www.dnevnik.si publishes surveys related to articles from the paper’s latest edition. In some (albeit very rare) occasions, the results from their own online poll were invoked in the television news show 24 ur, but these results were not put in the main focus. Even though the surveys are primarly intended to entertain visitors to web sites, the possible effect of such surveys on the public should not be neglected. As we suggested above, the interaction between public opinion and the public is two-way. Surveys (including short surveys in media) help awake the participant’s feeling of involvement and perhaps even of solving public matters. Furthermore, short surveys are generally not perceived as the actual measures of public opinion, but as a reflection of “the pulse” surrounding the issue at hand. An assessment of such surveys might convince journalists to rely more heavily on advice from survey methodologists when designing the short survey questions. 2.2 Evaluation criteria Among all the surveys archived on the web sites, we took into consideration only those published over the course of one year, from July 1st, 2004 until July 1st, 2005. From these, we then selected a distinct sample of 25 randomly chosen questions for each web site. To assure the unbiased selection of questions for the three samples, we relied on an online application called “Research Randomizer” (Urbaniak and Plous, 2003). This application generates sequences of numbers by using a complex algorithm seeded by the computer’s clock. The generated numbers are thus not completely randomly selected, making the application a “pseudo-random number generator” (Urbaniak and Plous, 2003). Nonetheless, we believe the selection process of this application was sufficiently accurate, and it met the needs of our research. In the table below we present the basic parameters (compare with Table 1) for the analyzed questions from each media (n=25 for all three samples). 11 The existence and accessibility of the poll archive was actually a requirement the web site had to fulfil in order to be included in the study. Web sites that failed to fulfil all the requirements, including this, were excluded from the study. 12 Such a disclaimer is published in only one Slovenian journalistic web site: the web site of the most prominent Slovenian daily newspaper, Delo. 360 Simon Ličen et al. Table 2: Some basic parameters of the samples of analyzed questions (n=25). Parameter www.24ur.com www.rtvslo.si www.dnevnik.si DUR 4.42 days 3.31 days 1 day RES 2711 1206 347 RES/DAY 613 364 347 DK-n 76% 84% 20% The characteristics of the samples obtained closely resemble the overall characteristics of all surveys published on selected sites in the period between July 1st 2004 and July 1st 2005. Therefore, we assume that these surveys are representative of the population of surveys published on selected sites. These three samples were then reviewed by expert evaluation. We decided to employ a checklist that would allow us to highlight errors of observation (or measurement errors – the errors that originate from flaws in the questionnaire itself). Recently, useful checklists and guidelines regarding questionnaire evaluation have been developed by Statistics Sweden (Statistika centralbyran) (see e.g. Lindström, 2004). Following these guidelines, we prepared a checklist to suit the characteristics of web surveying and online polling. The checklist employed was jointly formulated by all three authors of the research. The actual review and coding was then done by the leading author. Table 3: Survey evaluation checklist. Survey questions - The language must be simple. - The questions must be concise. - Ambiguous questions must be defined. - There must be only one issue per question. - Place and time must be determined. - Hypothetical and retrospective questions must be carefully worded. - Questions that impose the answer (e.g. I agree ...) must be avoided. - Emotionally loaded words and misleading questions must be avoided. - Questions regarding opinion must be balanced. - Ensure questions are not frightening or insulting to the participant. Response alternatives Formal rules - Alternatives must feature all possible answers. - Extreme answers in ordinal scales must be sufficiently far apart. Fixed response alternatives - Each alternative must be exhaustive. - Each alternative must be exclusive. - There must not be too many alternatives. - “Yes-No” questions might appear too restrictive. - Do not force participants to answer on a topic about which they know nothing (add a neutral or “Don't know” answer). - Check the order of the alternatives. The Quality of Survey Questions… 361 All questions from the three random samples underwent a check according to these criteria. 3 Research and analysis For each survey analysed, we separately evaluated the quality of the survey question13 and the quality of the response alternatives. If a question or response alternative complied with all the conditions from the checklist above, we evaluated such question/alternative as being of good quality. If a question/alternative failed to comply with one of the conditions but the deficiency was minor, and the question/alternative was still comprehensible, we considered it acceptable. When a question/alternative failed to comply with several conditions or when the deficiency was sufficiently serious as to compromise the clear and unambiguous understanding of the survey itself, the question/alternative was considered as being of poor quality. Analysis was conducted separately for the survey questions and for the response alternatives. The evaluation of one of the two components thus did not influence the evaluation of the other component. Therefore, if a survey question was rated as bad, the corresponding set of response alternatives could still be rated as good. Since we were evaluating survey questions in Slovenian journalistic media, the questions were thus in the Slovene language. When translating the original text, we tried to preserve the original meaning and wording of the survey in such a way as to preserve potential deficiencies as well. To give an idea of our evaluation criteria, we present some examples of the questions we researched and evaluated. First we present a survey we found faultless. Both survey question and response alternatives complied with all the conditions listed in our checklist. Do you support the spread of the European Union? - Yes. - No. - Don't know./Don't care. Now let us consider a survey with an acceptable set of response alternatives. Do you find Sašo Peče suitable for the duty of vice-president of the Slovenian parliament? - Yes. - No. - Don't care. 13 That is, the text that specifies which information should be given (Lindström, 2004: 11). 362 Simon Ličen et al. Here the answer “Don’t care” should have been rephrased to "Don't know, don't care". Somebody might be interested in politics, but might not have an opinion regarding this topic. We thus do not find such a set of response alternatives exhaustive. Furthermore, it requires that the participant answers the question. Not to know is not the same to not care. The proposed answer “Don’ know, don’’t care” might have been stated as a single answer or even as two distinct answers. This survey question points out a new dimension of such surveys, that is their local constraint. Non-Slovenian readers who are not familiar with Slovenian politics most likely will not be familiar with Sašo Peče, either. However, the research studied Slovenian journalistic web sites, and the published survey questions are tailored to the Slovenian socio-cultural environment. The results of our study thus apply to them. In this particular case, recasting the survey answer “Don’t care” to “Don’t know, don’t care” would solve this problem as well as allowing foreign respondents who would like to participate in the survey to simply check this option. Now let us consider a survey question rated as poor. Will our national team stay in group A next year? - Yes. - No. The question obviously fails to define the key thematic reference, which is the sport in question. Furthermore the question also features an emotionally loaded word, since the (presumably Slovenian) national team is defined as “our”. As for the above question’s response alternatives, the survey lacks a “Don’t know/don’t care” answer. We cannot expect everybody to be a sports enthusiast. We continue with a set of response alternatives that we rated as poor. Here the alternatives are neither analogous nor exhaustive. The proposed alternatives were most likely prepared subjectively, so there should be at least the answer “other”. What would make you happiest? - A better education. - A better job. - To have a partner. - A house and a car. - I am happy. - Don’t know. Here is another example of a set of response alternatives that was rated as poor: Do you trust Slovenian journalists? - Completely. - In part. - Seldom. - Not at all. - Don't care. The Quality of Survey Questions… 363 Such a set of response alternatives would fit a different question. A proper question (for this set of response alternatives) would be to what extent do you trust Slovenian journalists? However, since the two survey components were analyzed independently from one another and the question in itself is good, the deficiency lies in the response alternatives. The use of scales of answers is common in scientific survey methodology. To a minor extent we also encountered scales in response alternatives in these three surveys. However, the answers presented above are not correctly distributed. The two ends of the scale – “Completely” and “Not at all” – are not sufficiently distant. If we presume that the answer “in part” is the medium value on the answer scale, then the inclusion of the option “seldom” requires the addition of the answer “mostly” (or “more often than not”). 4 Results Below we separately present the results for the survey question analysis and for the response alternative analysis. The share of good questions is around two-thirds for all three journalistic web sites. There is a small component of acceptable questions and a component of badly formulated questions ranging from 20 percent on www.rtvslo.si to 28 percent on www.dnevnik.si (Figure 1). 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% www.24ur.com www.rtvslo.si www.dnevnik.si Website Good quality Acceptable Poor quality Figure 1: Rating of survey questions. 24 % 20 % 28 % 16 % 8 % 72 % 68 % 64 % 364 Simon Ličen et al. The share of well formulated questions published on the www.24ur.com website was 68 percent. The most common flaw among the questions rated as poor was inadequate explanation of the questions’ key concept. Such a mistake was encountered five times. In several cases, ambiguous expressions and even metaphors were used, which seriously compromise a clean understanding of the question. Sometimes the correct meaning of such ambiguous concepts could be deduced from the contemporary media or social context. Such a procedure is not acceptable in scientific research. In our opinion it is also unacceptable for such short media surveys. It requires that participants associate references in the survey to current events, which does not comply with the need for a unique interpretation of the survey question. The share of well formulated questions for www.rtvslo.si is slightly lower compared to the other media. However, owing to a larger share of “acceptable” questions, the share of poorly formulated questions is also lower in comparison with www.24ur.com. Again several survey questions posted on this web site lacked clarification of ambiguous terms. Metaphors are here avoided; however, we determined another inconsistency, namely, the use of plurals instead of the grammatical singular. All surveys allowed only one answer to be chosen from the set of response alternatives, so the use of the plural is incorrect and misleading. The largest share of flawless questions was recorded on the www.dnevnik.si web site. However, it is interesting that this web site also featured the highest number of badly formulated questions (72 percent and 28 percent, respectively). The most common deficiency among the survey questions on www.dnevnik.si was again ambiguity about the topic reference. Furthermore, several questions were formulated very broadly, generally failing to define the question’s key concept. The surveys on www.dnevnik.si were the only surveys directly related to articles published in the newspaper (as well as on the web site). However, despite the clear relation to the topic at hand, the lack of a key reference in a survey question cannot be accepted. The main flaw in the survey questions evaluated in all three media was thus that the questions were simply not clear enough. This was most manifest in the lack of explanation of key thematic references and the use of ambiguous terms. The results are rather different for the evaluation of the response alternatives. Here the share of good sets of response alternatives is much smaller. Only on one web site (www.rtvslo.si) did we count more than half the sets of response alternatives as being good or faultless. On the other two web sites, the share of good sets of response alternatives did not exceed one-third of all sets. We thus registered shares of poor and unacceptable answers ranging from 36 percent to as much as 68 percent (Figure 2). The Quality of Survey Questions… 365 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 56 % 36 % 12 % 32 % 8 % 56 % 68 % 4 % 28 % www.24ur.com www.rtvslo.si Website www.dnevnik.si Good quality Acceptable Poor quality Figure 2: Rates of clusters of answers. The share of well formulated sets of response alternatives for www.24ur.com was 32 percent. Nine sets of response alternatives (36 percent of all sets) were not exhaustive, lacking one or more alternatives that could reasonably have been expected. Here we also counted the sets lacking an “I don’t know/I don’t care answer”. It seems as if the sets of response alternatives on www.24ur.com tend to include only those responses that are somehow mentioned in the media that report the event. This does not come as a surprise but proves media reliance on media discourse; nonetheless, other valid response alternatives should not be neglected. A possible solution would be the consistent inclusion of the answer “Other”. Another common inaccuracy was the inconsistent inclusion and formulation of the “I don’t know/I don’t care” answer. We acknowledged that the surveys on this web site featured at least six different types of answer in this category. Some of them formulated “I don’t know/I don’t care” as a single answer, while others made a distinction between the two (thus not always including both in a single set of response alternatives). In our opinion such an answer should be included in all sets of response alternatives. Considering the fact that survey questions on journalistic web sites are not scientific surveys, a single answer including both formulations would be adequate. The highest percentage of good sets of response alternatives was recorded on www.rtvslo.si and reached 56 percent of the sets of responses that were analyzed. 36 percent of the sets of responses were judged to be poor. Again, some of the sets of response alternatives lacked an “I don’t know/I don’t care” answer. Another curious fault was also recorded, namely, the 366 Simon Ličen et al. inappropriate use of ordinal scales. Twice in twenty-five sets of answer categories the proposed alternatives are not evenly distributed. Although www.dnevnik.si proved to be the most successful in formulating questions, only 28 percent of the sets of response alternatives published on this web site proved to be good. 68 percent of sets of response alternatives were categorized as bad. The most common fault was the lack of an “I don’t know/I don’t care” answer, which was missing in 15 sets of responses. We find such answer to be mandatory at least in the questions that aim to record one’s opinion. Furthermore, several sets of responses were not exhaustive, and once again an inappropriate ordinal scale was used. A particular type of inaccuracy appeared in this web site’s surveys, namely, the inconsistent application of certain grammatical rules. Most inconsistent were the use of capital letters at the beginning of each alternative and the use of a comma or a period at their end. The most common deficiencies of the sets of response alternatives in all three media analyzed were thus in order: - The lack of a “Don't know./Don't care” answer where necessary. - The response alternatives were not exhaustive – reasonable alternatives were missing. - The answers were not exclusive – they were not always unique and independent from each other. A rather rare, but statistically interesting inaccuracy was the use of inappropriate ordinal scales in two media. 5 Conclusion In our research we studied the quality of survey questions published on journalistic web sites. To this end, we analyzed three random samples of 25 questions from three Slovenian journalistic web sites. We analyzed the survey questions and the response alternatives separately. In the first part of our paper we discussed the function of short web surveys on journalistic web sites. Such surveys both measure and produce public opinion. Research into their quality and validity is thus appropriate. We discovered that only approximately two thirds of the survey questions are of good quality. The situation is even worse and the shares are nearly inverted when it comes to the response alternatives. The surveys have been evaluated by expert evaluation. The checklist employed was to a large extent derived from the guidelines of the Swedish Statistics Bureau (Lindström, 2004). This evaluation method can be subject to criticism. It is a qualitative method of evaluation and thus partially subjective. However, the evaluation checklist is known and published, so the analysis can be repeated. The Quality of Survey Questions… 367 Poor survey questions badly and inaccurately reflect actual public opinion so such mistakes should be minimized. We find our research to be the first step towards improving the quality of survey questions on Slovenian journalistic web sites. The results we obtained confirmed our assumption that such surveys are far from faultless. Among the topics that might be further researched in this field are both production analyses of such short surveys (how media reality is created) and their reception analyses (how survey results are perceived and thus how media create reality). Acknowledgment We would like to thank both anonymous reviewers who provided some extremely useful ideas and guidelines. References [1] Couper, M.P. (2000): Web Surveys: A review of issues and approaches. International Journal of Public Opinion Research, 4, 204-219. [2] Groves, R.M., Fowler, F.J. Jr., Couper, M.P., Lepkowski, J.M., Singer, E., and Tourangeau, R. (2004): Survey Methodology. Hoboken: John Wiley & Sons, Inc. [3] Hlebec, V. (2005): Koraki pri oblikovanju AV [The Steps of Survey Design]. Ljubljana: Faculty of Social sciences. Downloaded from http://av.fdvinfo.net/content.php?id=230 [4] Lindström, H.L. (2004): Design your questions right: How to develop, test, evaluate and improve questionnaires. Stockholm: Statistika centralbyr?n/Statistics Sweden. Downloaded from http://www.scb.se/Grupp/Metod/_Dokument/Design_your_questions_rightB.pdf [5] Lozar Manfreda, K. (2001): Web Survey Errors (Doctoral dissertation). Ljubljana: Faculty of Social sciences. [6] Lozar Manfreda, K., Batagelj, Z., and Vehovar, V. (2002): Design of web survey questionnaires: Three basic experiments. J. Comput.-Mediat. Commun. (Online). Online edition, April 2002, 7. Downloaded from http://www.ascusc. org/jcmc/vol7/issue3. [7] Schuman, H. (1997): Polls, Surveys, and the English Language. The Public Perspective, April/May, 6-7. [8] Splichal, S. (1997): Javno mnenje [Public Opinion]. Ljubljana: Faculty of Social sciences. [9] Trošt, M. (2003): Izzivi spletnega novinarstva: analiza spletnih strani izbranih evropskih časopisnih hiš [The challenges of internet journalism: an 368 Simon Ličen et al. analysis of selected European media web sites]. Bachelor’s thesis. Ljubljana: Faculty of Social sciences. [10] Urbaniak, G.C. and Plous, S. (2003): Research Randomizer. Middletown: Wesleyan University. Downloaded from http://www.randomizer.org/.