Metodološki zvezki, Vol. 2, No. 1, 2005, 15-26 Translation of Measurement Instruments and their Reliability: An Example of Job-Related Affective Well-Being Scale Nino Rode1 Abstract The translation of measurement instruments, like all cross-cultural research, presents many problems for researchers. Because of cultural and linguistic differences, the questions or items can have quite different meaning in the translated instruments, thus threatening the validity and reliability of measurement. These problems are (or should be) addressed by translation back-translation procedure. To illustrate the problems, the case of Job-related Affective Well-Being Scale is presented. The scale was translated from English to Slovene and applied directly in the research without backtranslation or other form of testing the translation. The Cronbach's alpha is used to compare reliability of the results obtained with original (English) version reported by authors of the scale and with the translated (Slovene) version. In the item analysis some items have been singled out as problematic by item diagnostics. Especially the item »excited«, translated as »vznemirjen«, is discussed, since it is possible that backtranslation procedure could fail to detect the problem. At the end some tentative solutions are suggested. 1 Introduction Before using statistical methods on any data, we should make sure that data really represent the concepts they are supposed to measure and that they do it reliably. Assuring validity and reliability isn’t a simple task. Development of valid and reliable measurement instruments requires much work, time and knowledge. It is much easier to adopt instruments already developed by other researchers. A search on the internet reveals a number of sources for measurement scales in any field, or even measurement instruments themselves (most of them commercia), many of which, promise the desired validity and reliability, as well as other desired characteristics. University of Ljubljana, Faculty of Social Work; nino.rode@fsd.si 16 Nino Rode However, adopting of the instruments developed by other researchers frequently involves their translation into other languages. In order to preserve the properties of the instrument such translations mostly follow the Ask-the-Same-Question model, in which the »…translated items must present the same stimulus as the source questionnaire items and do so by referring to the same entities (abstract and concrete) as do the source items.« (Harkens, van der Vijver, and Johnson, 2003: 46). Usually this is achieved through the verbatim translation of the questions. But this type of translation of the instruments involves a number of problems. Many problems stem from the fact that the questions in the questionnaire or wordings of items in the instrument are culturally anchored (Harkens, van der Vijver, and Johnson, 2003: 23). The questions which are appropriate for some contexts are less appropriate in other contexts. This problem is characteristic for any change of the context. However its importance is increased with the use of different languages since many subtle cultural differences are reflected in the differences of languages. One of the most pressing problems is that connotations can be lost in translation. Some words have special historic connotations in some countries and not in others. The context of questions or items can also have impact on their meaning. Questions do not only have a meaning on the basis of the words used, but also in terms of their contexts. The context can alter the meaning of the question even though questions are essentially the same in format and semantics as those in other surveys. In other cases structural differences mean that exactly equivalent objects or entities do not exist, or that terms used to describe one thing in one country describe something else in another. (Smith, 2003: 71) These problems are most evident in cross-cultural research, but that doesn't mean that they are not present in of translation of measurement instruments for other purposes. Most of these problems could and should be solved when the instruments are developed. One of the most useful strategies is cultural decentering of the instrument, which aims to remove the words and concepts that are difficult to translate or are specific to a particular culture (van der Vijver and Leung, 1997: 39, Harkens, van der Vijver, and Johnson, 2003: 32; van der Vijver, 2003: 149). But when the existing instruments are used, this strategy is less useful as it can lead to change of the measurement properties of the instrument. There are many ways to solve the problems of translation of the existing measurement instruments. One of the most well known is translationbacktranslation procedure (cf. van der Vijver and Leung, 1997: 39), where first the instrument is translated into the target language and then translated back to the source language by an independent translator. Comparing original and backtranslated versions of the instrument can reveal some translation problems. But, as van der Vijver and Leung point out, the procedure puts a premium on literal translations, which can lead to unnatural wording of the questions in the Translation of Measurement Instruments and… 17 target language. Usually the procedure pays more attention to the semantics and less to connotations, naturalness, and comprehensibility. (van der Vijver and Leung, 1997: 39). Based on the experiences of cross-cultural research, Harkness (2003: 37) recommends a committee approach to instrument translation. Apart from first translations, most of the work should be done in the group. In this approach different team members assume three roles: translators, translation reviewers, and translation adjudicators. In instrument translation Harkness (2003: 38) recommends the following five basic procedures: translation, review of translation that produces different versions of the instrument, adjudication on which version to use, pretesting of the instrument and documentation of the problems encountered and decisions about solutions. Both of the proposed procedures, the backtranslation, and the committee approach, have one property in common: they are costly in terms of time and other resources. To cut the costs these procedures are frequently avoided in everyday research; instead one-to-one or direct translation is used. This strategy is less costly, and yet often successful enough for the purpose of the research at hand. Also, if some of the aforementioned problems occur due to mistakes in translation, there is no immediate way to see them so researchers can pretend they don't exist. Of course this doesn't necessarily mean that there are no problems at all. What we present here is an example of what went wrong in the translation of the Job-related Affective Well-being Scale from English to Slovene. 2 Methods 2.1 The JAWS scale The Job-related Affective Well-being Scale, the JAWS Scale (van Katwyk, Fox, Spector, and Kelloway, 2000) is a 30 item scale describing emotional reactions of respondents to their job. It is based on a two-dimensional circumplex model in which emotions are represented on a continuous circle. The space of the circumplex is defined by two bipolar dimensions of pleasure and arousal. The pleasure-displeasure dimension represents emotional valence, whereas the arousal dimension, ranging from sleep to high arousal, represents activating potential of emotions. Each affective state can be identified by its position in this space. (cf. van Katwyk, Fox, Spector, and Kelloway, 2000: 219) This basis of the scale presents us with a problem. The scale is clearly based on the two-dimensional model, but it is constructed to be, and it is intuitively clear that it should behave as, a one-dimensional scale in regard to well-being of the respondents, since the high arousal feelings are supposed to have more impact on the well-being than the low arousal ones aligning the effect of the two dimensions. 18 Nino Rode Effectively this leaves us with a scale measuring the well-being of a respondent in one dimension from low quality (negative, bad) to high quality (positive, good). As the results show, this is not necessarily a contradiction. Both the two-dimensional basis and the one-dimensional behavior of the scale are confirmed. In an application of the scale, respondents were asked how often they had experienced each of the emotions at work over the prior 30 days. Responses were made within a five-point scale with anchors Never, Rarely, Sometimes, Quite often, Extremely often or always. The JAWS includes a wide variety of emotional experiences, both negative and positive, which can be placed into four categories (subscales) that fall along two dimensions: pleasurableness and arousal (intensity). The scale can be scored in three ways. 1. An overall score of all 30 items with the negative emotions reverse scored; 2. Separate scores of all 15 negative items and all 15 positive items combined separately without reverse scoring; 3. Four scores matching the four dimension combinations: distress, excitement, depression, and contentment, containing 5 items each. (Spector, 2002) The JAWS Scale was used as a part of a small research study on job satisfaction of social workers in Slovenia. Due to lack of time and other resources, the scale was translated directly from English to Slovene without a backtranslation. The translation was judged by the research group to be adequate for the purpose. The scale was implemented as a part of interview of 96 social workers working in centers of social work. These are government institutions in Slovenia which have the mandate to carry out social welfare and social work tasks. Two of the respondents failed to give an answer to one item each, so total JAWS score was computed for 94 respondents. For the same reason the negative emotions score, the positive emotions score, and the low pleasurable-low arousal score were computed for 95 respondents. Other scores were computed for all 96 respondents. Descriptive statistics for the JAWS Scale and its subscales for Slovenia and the reported U.S. norms (Spector, 2002) are presented in Table 1. As we can see the results are quite similar, so at first glance the behavior of the translated instrument is similar to the original. However, this similarity isn’t necessarily an evidence of good performance of the translated instrument. 2.2 Procedures To assess reliability of the scale a simple item analysis was performed on the gathered data. First, Cronbach a was computed for the JAWS Scale and its subscales. Statistical significance of the equality of reliability in two groups is Translation of Measurement Instruments and… 19 based on the fact that statistic (1-a1) / (1-a2) follows an F distribution with (N1-1) and (N2-1) degrees of freedom. (van der Vijver and Leung, 1997: 60) Table 1: Comparison of descriptive statistics of JAWS Scale and subscales for USA and Slovenia. n Mean Std. Dev. USA 113 405 405 113 113 113 113 SLO USA SLO USA 16,7 11,4 11,3 3,9 3,4 3,2 3,5 SLO 11,9 7,3 6,7 2,6 2,8 2,6 2,3 Total JAWS (all 30 items) 94 105,6 105,8 Negative emotion 95 36,6 32,4 Positive emotion 95 39,5 44,2 High pleasurable-High arousal (HPHA) 96 14,4 15,1 High pleasurable-Low arousal (HPLA) 96 16,5 16,8 Low pleasurable-High arousal (LPHA) 96 9,5 11,4 Low pleasurable-Low arousal (LPLA) 95 11,0 10,1 Results for USA from Spector 2002 In the next steps of the analysis we concentrated only on the total JAWS scores. The second step was to calculate the descriptive statistics for the items and the item analysis statistics. This included the mean and standard deviation of a scale if the item would be deleted, item-total correlations and Cronbach a of the scale if the item would be deleted. Table 2: Comparison of reliability of JAWS Scale for USA and Slovenia. Cronbach a F 2,103 1,380 2,582 3,482 1,679 1,572 1,890 USA 0,95 0,92 0,94 0,90 0,81 0,80 0,80 Slovenia p Total JAWS (all 30 items) 0,89 0,0001 Negative emotion 0,89 0,0185 Positive emotion 0,85 0,0000 High pleasurable-High arousal (HPHA) 0,65 0,0000 High pleasurable-Low arousal (HPLA) 0,68 0,0043 Low pleasurable-High arousal (LPHA) 0,69 0,0108 Low pleasurable-Low arousal (LPLA) 0,62 0,0006 Results for USA from Spector 2002 The third step was to assess the behavior of the scale items with factor analysis. One of the best ways to test a translated measurement instrument would be to assess measurement invariance of the scale when used on Slovene vs. American respondents using confirmatory factor analysis, as explained by Meredith (1993). Since data for the respondents from the USA weren't available, 20 Nino Rode we weren't able to do this. Nevertheless we got some insight through exploratory factor analysis of Slovene data. The factor analysis was performed with unweighted least squares factor extraction and varimax rotation. 3 Results 3.1 Item analysis The comparison of results for Slovenia and the reported Cronbach a for USA (Spector, 2002) is presented in Table 2. The comparison shows that the reliability of the translated JAWS Scale is consistently lower then the original scale. As we can see all the differences are statistically significant at p=0,05, many of them even at p=0,001. Applying rule of thumb according to which Cronbach a > 0,8 represents a good reliability, it can be said that the total JAWS score and scores for negative and positive emotions in the translated instrument are reasonably good, which isn't true for other four subscales. Since Cronbach a of any scale depends on the number of items, the results are not surprising, but the low Cronbach a of the scales with smaller number of items may signal that good reliability of the longer three scales is mostly due to their length. In other words there could be a problem regarding reliability of some items. The results of the item analysis are presented in Table 3. The problematic items are printed in bold typeface. According to the results, there are five problematic items: »at ease«, »bored«, »elated«, »excited«, and »ecstatic«, which have item-total correlation lower than 0,2 and deletion of which would increase Cronbach a of the scale. 3.3 Factor analysis In the next step the factor analysis was performed. Based on the scree plot a two-factor solution was accepted, which explained 38,3% of the total variance. The results are presented in Table 4 and Figure 1. The reliability coefficient W based on communalities (Ferligoj et al. 1995, 46) was 0,918. The factor analysis revealed as problematic the same five items as the item to total correlations. These items have small communality (Table 4) and their factor loadings, (Table 4, Figure 1) differ from other items, which are arranged in a reasonably linear structure. There are some other items which have low communality (»anxious«, »calm«, »furious« and »fatigued«) but their factor loadings are similar to other items and their omission would decrease Cronbach a of the scale, so they aren't much of a threat to its reliability. Translation of Measurement Instruments and… 21 Table 3: Item analysis statistics for JAWS Scale translated to Slovene. mean st. dev. Mean if deleted StDv. if deleted Itm-Totl Correl. a if deleted 1. My job made me feel at ease 3,17 0,969 102,66 11,624 0,190 0,898 2. My job made me feel angry 3,46 0,650 102,37 11,489 0,531 0,890 3. My job made me feel annoyed 3,52 0,758 102,31 11,406 0,561 0,889 4. My job made me feel anxious 2,79 0,878 103,04 11,445 0,428 0,892 5. My job made me feel bored 4,76 0,479 101,07 11,808 0,058 0,897 6. My job made me feel cheerful 3,50 0,786 102,33 11,401 0,545 0,890 7. My job made me feel calm 3,43 0,861 102,40 11,437 0,448 0,892 8. My job made me feel confused 3,67 0,753 102,16 11,547 0,371 0,893 9. My job made me feel content 3,87 0,707 101,96 11,404 0,610 0,889 10. My job made me feel depressed 4,10 0,791 101,73 11,487 0,429 0,892 11. My job made me feel disgusted 4,16 0,871 101,67 11,233 0,687 0,886 12. My job made me feel discouraged 4,38 0,735 101,45 11,450 0,519 0,890 13. My job made me feel elated 2,17 0,785 103,66 11,702 0,152 0,897 14. My job made me feel energetic 3,31 0,762 102,52 11,394 0,574 0,889 15. My job made me feel excited 3,02 0,829 102,81 12,029 -0,256 0,905 16. My job made me feel ecstatic 2,35 0,826 103,48 11,646 0,209 0,896 17. My job made me feel enthusiastic 3,26 0,789 102,57 11,443 0,487 0,891 18. My job made me feel frightened 3,91 0,799 101,91 11,560 0,329 0,894 19. My job made me feel frustrated 3,71 0,838 102,12 11,239 0,710 0,886 20. My job made me feel furious 4,30 0,731 101,53 11,554 0,374 0,893 21. My job made me feel gloomy 3,82 0,879 102,01 11,329 0,565 0,889 22. My job made me feel fatigued 2,83 0,771 103,00 11,540 0,370 0,893 23. My job made me feel happy 3,31 0,804 102,52 11,415 0,514 0,890 24. My job made me feel intimidated 4,04 0,854 101,79 11,490 0,388 0,893 25. My job made me feel inspired 3,14 0,899 102,69 11,374 0,499 0,890 26. My job made me feel miserable 4,29 0,757 101,54 11,360 0,625 0,888 27. My job made me feel pleased 3,73 0,721 102,10 11,353 0,669 0,888 28. My job made me feel proud 3,52 0,839 102,31 11,346 0,575 0,889 29. My job made me feel satisfied 3,66 0,862 102,17 11,370 0,529 0,890 30. My job made me feel relaxed 2,66 0,837 103,17 11,227 0,727 0,886 Factor analysis of the items when the five problematic items were deleted was also a two-factor solution, which explained 42,2% of the total variance. The Cronbach a of the reduced JAWS Scale was 0,913 and reliability coefficient W was 0,932 which is a considerable improvement compared with the complete JAWS Scale. The analysis revealed a reasonably linear structure of factor loadings in the two-dimensional space (Table 4, Figure 2), which corroborates the intuition of unidimensionality of the JAWS Scale regarding the concept of well-being. 22 Nino Rode Table 4: Communalities and factor scores of the items in complete and reduced JAWS scale. Complete JAWS Reduced JAWS Commun. Factor 1 Factor 2 Commun. Factor 1 Factor 2 1. My job made me feel at ease 0,164 0,003 0,405 ---- ---- ---- 2. My job made me feel angry 0,372 0,580 0,189 0,364 0,551 0,244 3. My job made me feel annoyed 0,454 0,652 0,171 0,446 0,617 0,256 4. My job made me feel anxious 0,271 0,513 0,091 0,246 0,475 0,142 5. My job made me feel bored 0,014 -0,028 0,116 ---- ---- ---- 6. My job made me feel cheerful 0,336 0,281 0,507 0,317 0,242 0,508 7. My job made me feel calm 0,235 0,449 0,182 0,237 0,433 0,223 8. My job made me feel confused 0,387 0,610 -0,123 0,387 0,620 -0,043 9. My job made me feel content 0,461 0,416 0,537 0,476 0,324 0,609 10. My job made me feel depressed 0,298 0,543 0,061 0,285 0,517 0,132 11. My job made me feel disgusted 0,520 0,627 0,356 0,527 0,600 0,408 12. My job made me feel discourag. 0,279 0,524 0,065 0,279 0,509 0,140 13. My job made me feel elated 0,218 -0,176 0,432 ---- ---- ---- 14. My job made me feel energetic 0,461 0,214 0,644 0,456 0,135 0,662 15. My job made me feel excited 0,273 -0,509 0,119 ---- ---- ---- 16. My job made me feel ecstatic 0,272 -0,174 0,492 ---- ---- ---- 17. My job made me feel enthusiastic 0,498 0,112 0,697 0,477 0,041 0,689 18. My job made me feel frightened 0,497 0,665 -0,234 0,499 0,691 -0,145 19. My job made me feel frustrated 0,660 0,773 0,250 0,653 0,731 0,345 20. My job made me feel furious 0,198 0,431 0,110 0,205 0,424 0,160 21. My job made me feel gloomy 0,420 0,623 0,180 0,421 0,600 0,247 22. My job made me feel fatigued 0,280 0,528 0,030 0,278 0,517 0,104 23. My job made me feel happy 0,545 0,098 0,732 0,555 -0,015 0,745 24. My job made me feel intimidated 0,454 0,662 -0,127 0,518 0,716 -0,076 25. My job made me feel inspired 0,483 0,074 0,691 0,475 -0,002 0,689 26. My job made me feel miserable 0,477 0,640 0,259 0,484 0,608 0,339 27. My job made me feel pleased 0,519 0,492 0,526 0,509 0,416 0,580 28. My job made me feel proud 0,460 0,263 0,625 0,449 0,188 0,643 29. My job made me feel satisfied 0,393 0,201 0,594 0,416 0,132 0,631 30. My job made me feel relaxed 0,585 0,462 0,609 0,581 0,403 0,647 4 Discussion As we can see, the results of the factor analysis support both the two-dimensional model of affective states and the intuition of the unidimensionality of the concept of well-being. The two-factor model was revealed by exploratory factor analysis, which is in line with the theoretical model of dimensions of emotional valence and arousal. However in this two-dimensional space the factor loadings of the items form a reasonably linear structure, which points to unidimensionality of the measure of well-being. Both item to total correlations and communalities from factor analysis pointed out the same five problematic items: »at ease«, »bored«, »elated«, »excited«, and »ecstatic«. Factor loadings of these items even suggest that there could be not only Translation of Measurement Instruments and… 23 a question of reliability but even of validity of inclusion of these items in the scale. The factor analysis also identified four more suspect items: »anxious«, »calm«, »furious« and »fatigued«. Since these don’t have a visible impact on the reliability of the scale we won’t discuss them now. Factor Plot in Rotated Factor Space 1,0 0,5 0,0 -0,5 -1,0 PJAWb5 P PJAWS23 OÖJAWS14 PJAWS13 • PJAWS15 > PJAWS30 PJAW317 #0 AWS28 „.n.ev,, PJAWS16 OPJAWS6 PJAWS1 __ PJAWS2(9 PJAWS11 PJAWS26 OPJAWS19 AiP-i^ °Q0PJAWS3 PJAWS5PJAWS10 P J a WS4 P J AWS12 PJAWS22 PJAWS8 10 P J AWS24 OPJAWS18 -1,0 -0,5 0,0 Factor 1 0,5 1,0 Figure 1: Factor loadings of the items for the complete JAWS Scale. A question arises, what went wrong with the five problematic items. A closer inspection of item means may give an answer for the item »bored«, which has the highest mean (4,76), very close to maximum item value 5, and low standard deviation (0,479). This item suffers from the so called ceiling effect. Its mean is too close to the maximum value to allow normal dispersion of the answers. The same could be the case for the item »elated«, which has the lowest mean (2,17), though its mean isn't so close to minimum value 1, and its standard deviation (0,785) isn't much lower than the standard deviations of other items. However there are some problems related to the translation of the scale. A of plausible explanation for the items »elated« and »ecstatic« is that these are quite exotic terms, which were translated to similarly exotic expressions in Slovene »vzhičen« (= »elated«) and »vznesen« (= »ecstatic«). Those expressions have a slightly theatrical connotation in Slovene and are seldom used in connection with a 24 Nino Rode job. The authors of the JAWS Scale explicitly required that affective states described by items should be amenable to inclusion with the common stem »my job made me feel …« (van Katwyk, Fox, Spector, and Kelloway, 2000: 221) It seems that this property wasn’t fully replicated in translation to Slovene. Factor Plot in Rotated Factor Space 1,0 0,5 0,0 -0,5 -1,0 PJAWS25Ž PJAWS23 PJAWS14 J PJWS2 PJAWS JAWS27 JAWS6 PJAWS30 P JAWS 11 PJAWS28D PJAWS2 6 OPJAWS19 P J AWS20O P JAWS7 °0QPJAWS12 PJAWS4 w PJAWS8D QP • PJAWS18 -1,0 -0,5 0,0 Factor 1 0,5 1,0 Figure 2: Factor loadings of the items for the reduced JAWS Scale. The explanation for bad performance of the item 1: »at ease« is not straightforward enough. Its mean is more or less at the centre of the means of all items (3,17), but its standard deviation is the largest of all. A tentative explanation for bad performance of this item is perhaps its social (un)desirability. We can safely presume people would say that it's not good for social workers to be at ease on their job. They are supposed to care. It is possible that this consideration made some respondents underreport the frequency of this feeling, which led to larger dispersion of the responses and to lower reliability of the item. Nevertheless, the results are not conclusive, so the problem requires further consideration. Finally, let us have a look at the item that performed worst. That was item 15: »excited«. As we can see its item-total correlation was negative (-0,256), which basically means that the relation between the item and the scale is reversed. Also Cronbach a would improve considerably if this item was deleted. For this item we Translation of Measurement Instruments and… 25 can safely assume that the problem is mostly in translation. A term with a positive connotation in English was translated by a term with a negative connotation in Slovene, as we can see from the comparison of the dictionary definitions. In the Longman dictionary of the English language (1991) for instance word »excite«, from which the term »excited« is derived, is defined as »to rouse strong, esp. pleasurable (cursive NR) feeling«. The term »vznemirjen« is a derivation of the word »vznemiriti« which is defined in the Dictionary of Slovene language (1991) as »…to bring someone to a state of tension because of expectation of something, usually bad, unwanted (cursive NR) «. As the JAWS Scale depends on the difference between the positive and the negative feelings, this disregard for the connotation of the words in the two languages led to a poor performance of the item. There is no straightforward solution for this problem, since many terms for different feelings were used in the scale. For instance »navdušen« might be a good translation of »excited«, but this term was used to translate »enthusiastic« since it denotes a stronger positive feeling than »excited«. If »navdušen« is used to translate »excited« a stronger term in Slovene should be found for »enthusiastic«. Let us conclude with some suggestions on what could be done to eliminate mistakes in translation. If we concentrate on the item which performed worst (»excited«) the simplest way to deal with it could be to add the qualifier »positive« to the item. The term »(pozitivno) vznemirjen«, denoting (positive) excitement, could eliminate the problem of connotation, but it would sound unnatural in Slovene. We could translate »excited« as »navdušen« but doing this would require finding another stronger term to translate »enthusiastic«. Maybe we could find some other Slovene translations of the term »excited« that would have approximately the same meaning, but with positive connotation. All these solutions would be adaptations of the scale As the methods and procedures used of creating it are well documented in the article (van Katwyk, Fox, Spector, and Kelloway, 2000) one could always try to construct the JAWS scale in Slovene from the scratch, but this seems a bit of overkill. In spite of its deficiencies, the JAWS Scale is in our opinion too good to be abandoned. And even in present form it is reliable enough to be used. As for translation of the existing measurement instruments, we suggest that no matter how low the budget for research is, special care should be taken to check the quality of the translated instrument. If nothing else, pilot testing of the instrument should be done. On the pilot results at least item-total correlations should be checked. Inspection of the explorative factor analysis results, especially communalities is also instructive. If data collected by the scale in source language are available, invariance of the scale should be tested with confirmative factor analysis. 26 Nino Rode References [I] DeVellis, R.F. (1991): Scale Development, Theory and Applications. London: Sage. [2] Fajgelj, S. (2003): Psihometrija, Metod i teorija psihološkog merenja. Beograd: Centar za primenjenu psihologiju. [3] Ferligoj, A., Leskošek, K., and Kogovšek, T. (1995): Zanesljivost in veljavnost merjenja, Metodološki zvezki, 11, Ljubljana: FDV. [4] Harkness, J.A. (2003): Questionnaire translation. In: Harkness, J.A., van der Vijver, F.J.R., and Mohler, P.Ph. (Eds): Cross-Cultural Survey Methods. New Jersey: John Wiley & Sons. [5] Harkness, J.A., van der Vijver F.J.R., and Johnson, T.P. (2003): Questionnaire design in comparative research. In Harkness, J.A., van der Vijver, F.J.R., and Mohler, P.Ph., Cross-Cultural Survey Methods. New Jersey: John Wiley & Sons. [6] Meredith, W. (1993): Measurement invariance, factor analysis and factorial invariance. Psychometrica, 58, 525-543. [7] Smith, T.W. (2003): Developing comparable questions in cross-national surveys. In Harkness, J.A., van der Vijver, F.J.R., and Mohler, P.Ph., Cross-Cultural Survey Methods. New Jersey: John Wiley & Sons. [8] Spector, Paul E. (2002): Overview of the Job-related Affective Well-being Scale (JAWS), http://chuma.cas.usf.edu/~spector/scales/jawspage.html 7. 9. 2004. [9] van Katwyk, P.T., Fox, S., Spector, P.E., and Kelloway, E.K. (2000): Using the job-related affective well-being scale (JAWS) to investigate affective responses to work stressors. Journal of Occupational Health Psychology, 5, 219-230. [10] van der Vijver, F., Leung, K. (1997): Methods and Data Analysis for Cross-Cultural Research. London: SAGE Publications. [II] van der Vijver, F.J.R. (2003): Bias and equivalence: Cross-cultural perspectives. In Harkness, J.A., van der Vijver, F.J.R., and Mohler, P.Ph. (Eds.): Cross-Cultural Survey Methods. New Jersey: John Wiley & Sons. [12] (1991): Longmans Dictionary of the English Language. London: Longman. [13] (1991): Slovar slovenskega knjižnega jezika, DZS, Ljubljana.