90     SODOBNA PEDAGOGIKA 5/2006
Dr Janez Sagadin
Methodological Problems Concerning the Criteria for Selecting Gimnazija Candidates
Abstract: The article addresses certain methodological problems and procedures of determining (assessing) the predictive validity of gimnazija candidate-selection criteria, i.e. criteria that are combinations of primary school results (overall achievement and achievement in individual subjects) and the results of other, one-off knowledge examinations (external examinations, entrance examinations). The author also points to reservations/limitations of using the results of knowledge assessment in schools as predictors (independent variables) and dependent variables in correlation and regression procedures. He also indicates the problem of how, in candidate-selection criteria, to appropriately balance the weight of primary school achievements (overall achievement and grades in individual subjects) and the weight of the results of one-off external examinations. He points to some possible models of resolving this problems and the problem of the gimnazija candidate-selection criteria in general. He also touches on the criterion that would only include primary school achievements.
Key words: (gimnazija) candidate-selection criteria, school achievement, achievement/grade in an individual subject, external examination, entrance examination, predictive validity, independent variable, dependent variable, correlation, regression.
UDK: 37.012
Dr Janez Sagadin, professor, Faculty of Arts, University of Ljubljana, retired; e-mail: janez.sagadin@guest.arnes.si
SODOBNA PEDAGOGIKA 5/2006, 90–103
91
1       Introduction
Issues concerning the transition of students from primary school to gimnazija (the Slovenian general upper secondary school intended for students who later wish to continue their education in higher education) and to secondary schools in general have a methodological, school system, didactical and other nature but in a way they are always topical; in recent years issues concerning the selection of gimnazija candidates when enrolment is limited have been particularly in the centre of attention of expert and wider publics, with polemics also appearing concerning the criteria used in the selection of candidates.
This article briefly discusses certain methodological problems (including solutions, reservations/limitations) encountered in connection with the criteria for the selection of gimnazija candidates in Slovenia, including and in particular the problems related to the predictive validity of such criteria (hereinafter: candidate-selection criteria).
2       Problems, solutions, reservations and limitations
2.1   Scruples and limitations concerning knowledge assessment
In connection with the use of primary school grades (in individual subjects and overall achievement) as a candidate-selection criterion and in the determination (assessment) of the predictive validity of this criterion, certain methodological scruples and problems are encountered. The scale of school results has five grades (from ’unsatisfactory’ or 1 to ’excellent’ or 5), but the knowledge matching individual grades is not defined/standardised precisely enough which is, of course, also true of the lowest positive grade such that the scale does not even have a clear lower anchor of the range of positive grades. In such circumstances, it is the nature of the matter itself that necessarily causes differences between teachers concerning
92     SODOBNA PEDAGOGIKA 5/2006
J. Sagadin
the criteria of student assessment and an oscillation in criteria as applied by the same teacher. This is not a favourable basis for using the results of assessment according to the abovementioned scale in the role of independent1 and dependent variables within statistical procedures of determining the predictive validity of variables, with these procedures requiring the variables used to achieve the level of intervality and that their measurement is burdened by as few errors as possible and reliable to the greatest possible extent.
Regarding the requirement for intervality, we face a dilemma of whether to opt for the questionable assumption that school grades are true numerical and equidistant values and that the abovementioned statistical procedures fit them in intervality terms. The dilemma is particularly hard in connection with overall achievement when its categories are expressed numerically (from 1 to 5) and when it is assumed that this is an interval variable. Unfortunately, despite such dilemmas it is not always possible to give up the assumption of the intervality of variables such as grades in individual school subjects and the overall achievement as there are simply no ’by-passes’ on the way to certain empirical findings. This is also the case when determining the predictive validity of primary school grades for the achievement in gimnazija. The use of Pearson’s coefficient of correlation2 for this purpose already involves the above assumption of the intervality of variables and of their measurement with as few errors as possible and with the greatest possible reliability. Here, a ’by-pass’ is Pearson’s coefficient of contingency, but a correlation coefficient is more appropriate for various comparisons of the predictive validity of different independent variables and for the comparison of the results of different researches. Further, in a regression analysis of the predictive validity of independent variables the coefficient of contingency is not really useful. (Let us, by the way, note that in a simple linear regression, i.e. a regression with one independent variable, the correlation coefficient is the same as a standardised regression coefficient).
The abovementioned five-grade scale (in an individual subject consisting of five grades, and in the overall achievement of five categories), which is itself short for its use in regression analysis, is in general additional shrunk to practically only the three highest grades for new gimnazija students and to just the two highest grades in certain gimnazije; even the reduction to (almost) just the highest grade is not entirely excluded. This shrinking of results in the lowering of correlations between independent and dependent variables which is, even more than for the whole population of new students, true of gimnazije, which are in the lead as far as such shrinking is concerned as was shown in a survey of the generation of new gimnazija students in 1966/19673, although
1 Within the procedures of predicting (anticipating, estimating) the value of dependent variables on the basis of the values of independent variables, the term predictor is also generally used for independent variables and the term criterion for dependent variables (as a dependent variable is the criterion for the predictive validity of a predictor).
2 When we later mention a correlation or correlation coefficient, we always refer to this coefficient unless it is discernible that we are speaking of the coefficient of a multiple linear correlation.
3 See Sagadin 1968. All other findings from this research that we mention below also come from this source. The two Italian gimnaziji were not included in the research.
Methodological problems concerning the criteria ...
93
there had not yet been such a concentration of new students with top primary school results as seen in some gimnazije today. We have no control over such influences and their extent as the shrunken scale with new students cannot be extended – the primary school achievements that they bring are what they are, despite everything.
Along with these deficiencies, the school assessment of knowledge must also be recognised for its advantages compared to various one-off tests of the same knowledge. For instance, the final annual grade of a student’s knowledge in an individual subject is the result of the process of assessing their knowledge throughout the school year; in this process, the pool of the student’s knowledge can (if the assessment is appropriately placed in the education process and implemented) be examined in much greater detail than by means of a one-off test at the end of the year. Therefore, such a final grade has an advantage over one-off tests in terms of examining and diagnosing the student’s knowledge. This is true of both primary and secondary schools in a similar way. Consequently, one should not be surprised if research shows that the primary school achievement in correlation with the achievement at a gimnazija or other secondary school surpasses the achievements of one-off other tests of primary school knowledge.4 In fact, one should be more surprised if this were not the case. When that happens, one should invest an even greater effort in investigating the reasons for the situation and should also not forget the abovementioned shrinking of primary school achievement to the highest grades with new students and the potential advantage of the results of other knowledge tests with regard to variability; both may increase the difference between primary school achievement and results of other knowledge tests concerning the correlation with gimnazija achievements.
Regarding the scale of school grades, we have mentioned that the knowledge matching individual grades is not sufficiently defined. Let us add that this problem is also not resolved by any external knowledge test by itself, no matter how elaborate it is it may only be avoided, as with an external test for the selection of candidates the results in points do not have to be changed into school grades.
2.2 Selection/definition of independent and dependent variables, and the related choice of procedures for data analysis and for determining the predictive validity of independent variables
The findings about the predictive validity of certain candidate-selection criteria also depend on the method of choosing or defining the dependent variable. If the criteria are not defined precisely enough, both independent and
4 This does not refer to an individual gimnazija or other secondary school but to the population of gimnazije or other secondary schools, although such findings must also make individual schools ask themselves about the reasons for such a situation. In doing so, one should first look at the shrinking of primary school achievements to the upper end of the assessment scale.
94     SODOBNA PEDAGOGIKA 5/2006
J. Sagadin
dependent variables must be defined in the investigation of their predictive validity. The findings then depend on our decision concerning the variables of both types.
For instance, at the time of the survey mentioned (involving the generation of new gimnazija students in 1966/1967), the two criteria for selecting candidates were achievement in the entrance examinations in the Slovenian language, mathematics and foreign language, and the achievement in primary school. Gimnazije were not harmonised as to the use of these criteria, with each of them having a free hand in this regard. As the procedure for the selection of candidates did not take place according to uniform criteria, which might be the independent variables in the survey, independent variables had to be appropriately defined. It was reasonable to opt for achievements (grades) in mathematics, Slovenian language and foreign language (mostly English and to a lower extent German) achieved in entrance examinations, and for overall achievement5 and grades in mathematics, Slovenian language, English and German attained in the 8th grade of the then eight-year primary school. Each of these independent variables had to be matched with an appropriate dependent variable. Thus, the dependent variable for the overall achievement in the 8th grade of primary school was the overall achievement in the 1st year of gimnazija, and the dependent variable for the achievement in each subject (both for the achievement in the 8th grade and in the entrance examination) was the achievement in the subject concerned in the 1st year of gimnazija. The overall achievement showed the highest predictive validity. The sample assessment of correlation between the overall achievement in the 8th grade and the overall achievement in the 1st year was r=0.653 and the determination coefficient was r2 = 0.43, which means that 43% of the variance in overall achievement in the 1st year was explained by the overall achievement in the 8th grade or its variance. The overall achievement was followed by grades in mathematics, Slovenian language and foreign language attained in the 8th grade, the predictive validity of which surpassed the grades attained in entrance examinations, which was mostly true of mathematics.6 Schools also differed from each other regarding these prognostic values so that it was hard to determine which criterion for the selection of a candidate would be appropriate for all gimnazije. Nevertheless, all the established findings pointed to a possible solution, whose essence was that in the selection procedure gimnazije should prioritise candidates with better overall primary school achievement and, within an individual category of overall achievement, candidates with better grades in
5 In the survey, the overall achievement was used in the following form: for an ’excellent’ achievement, the student must have a higher number of ’excellent’ grades than ’very good’ grades and no grade lower than ’very good’; for a ’very good’ achievement, the student must have a higher total of ’excellent’ and ’very good’ grades than ’good’ grades and no ’satisfactory’ grade; for a ’good’ achievement, the student must have a higher total of the three highest grades than ’satisfactory’ grades; and a ’satisfactory’ achievement results from a higher number of ’satisfactory’ grades than higher grades.
6 Regarding the lead of primary school achievement over other predictors, the results were similar to the results (then accessible to the author) from other surveys in the then Yugoslavia and abroad (see Sagadin 1968, pp. 67-69).
Methodological problems concerning the criteria ...
95
mathematics, Slovenian language and foreign language. Within this model of selection procedure, at gimnazije that preserve entrance examinations, primary school grades in mathematics, Slovenian language and foreign language would be decisive for acceptance to at least the same extent as the grades attained in entrance examinations. (The definition of overall achievement in this model was already given in note 5.) The results of a survey conducted on one generation of new students cannot be mechanically generalised to the following generations. It was necessary to take account of the possibility of results varying from generation to generation and the related need for the entrance criteria to be re-investigated from time to time.
In a survey conducted by Boris Ko`uh on generations of new secondary school students in 1992/1993, 1993/1994 and 1994/1995,7 the highest predictive validity was recorded for the overall achievement and it was similar regarding the lead of primary school grades in individual subjects over the results of other, one-off examinations of primary school knowledge, only that then external tests in mathematics and Slovenian language replaced entrance examinations. The results of this survey clearly show how the predictive validity of different predictors may vary from generation to generation (see Ko`uh 1997).
Both surveys mentioned were conducted on appropriate representative samples. To demonstrate how the prognostic value of a candidate-selection criterion also depends on the definition of the criterion as well as on the definition of the dependent variable used for the determination or estimation of the predictive validity of the criterion, we can also use a smaller, non-representative group of new students. Certainly, one cannot generalise in the sense to transfer the values of correlation coefficients and other amounts obtained for such a group to the entire generation containing the group. One can only generalise the finding that the prognostic value of an acceptance criterion also depends on the definition of this criterion and on the definition of the dependent variable. This will also be the case below. The group will encompass the 2002/2003
generation.8
In our case (with non-representative data), the correlation between the achievement in mathematics attained in the 8th grade of eight-year primary school and the achievement in mathematics in the 1st year of gimnazija was r = 0.472 and the determination coefficient was r2 = 0.223, which means that 22.3%
7 These generations were already subject to the then introduced criterion for the selection of candidates (with a limited enrolment) in the form of a sum of points, namely the following three types of points: a) points achieved in external examinations in mathematics and Slovenian language (a maximum of 40 points was possible for each subject); b) points achieved through adding the overall achievements and grades in mathematics, Slovenian language and foreign language from the 5th to 8th grade of primary school (in total, a maximum of 80 points was possible); and c) Vega and Cankar Awards brought 5 points each.
8 Then the candidate-selection criterion was somewhat changed compared to the criterion at the time of Kožuh’s survey; the 5th grade was not included and the maximum total number of points for overall achievements and grades in mathematics, Slovenian language and foreign language for the 6th to 8th grade of the eight-year primary school was 60, with the same number of points for the external examinations in mathematics and Slovenian language (30 points for each subject).
96     SODOBNA PEDAGOGIKA 5/2006
J. Sagadin
of variance in the achievement in mathematics in the 1st year was explained by the variance in the achievement in mathematics in the 8th grade.9
The correlation between the achievement in the external test in mathematics (a maximum of 30 points was possible) and the achievement in mathematics in the 1st year was r = 0.313 and the determination coefficient was r2 = 0.098, which means that 9.8% of variance in the achievement in mathematics in the 1st year was explained by the variance of the achievement in the external test of knowledge.
And what if the independent variable (predictor) is defined as the sum of the achievement in mathematics in the 8th grade and the achievement in the external test in this subject? The correlation between this sum and the achievement in mathematics in the 1st year was r = 0.364 and the determination coefficient was r2 = 0.132, which means that 13.2% of the variance in achievement in mathematics in the 1st year was explained by the variance of this sum, which was 9.1 percentage points less than when the independent variable was only the achievement in mathematics attained in the 8th grade, and 3.4 percentage points more than when the independent variable was only the achievement in the external test.
In this case, the sum of independent variables brings poorer results concerning the predictive validity than if the achievement in mathematics attained in the 8th grade were only to be used as the independent variable (predictor).
The thing is that in the addition of independent variables, in the correlation between the sum and the dependent variable, each independent variable is weighted in proportion to its standard deviation. It is favourable for this correlation when there is a higher correlation between the dependent variable and the independent variable that has a larger standard deviation than this other independent variable, and when the correlation between the two independent variables is as low as possible.10 In our case, there were no such advantages. After the correlation with the dependent variable, the achievement in mathematics in the 8th grade (0.472) was ahead of the achievement in the external test (0.313), while after the standard deviation the achievement in the 8th grade (0.70506) lagged largely behind the achievement in the external
examination (3.60312).11
It is appropriate to add that (similarly to the sum of two independent variables) also in the sum of several independent variables each independent variable is weighted in proportion to its standard deviation, and that it is
9 The share of explained variance is small, however, in this case this is not of our interest as we do not intend to generalise this result.
10 This conclusion was reached on the basis of a formula for the correlation between the sum of two variables and a third variable. For such a formula and the way to it, see Guilford 1968, p. 475, formula (A.49). Formula (A.50) for the correlation between the sum of several variables and a variable is given on page 476.
11 Here, we used computer results without rounding them off.
Methodological problems concerning the criteria ...
97
favourable for the correlation of the sum with the dependent variable when there is a higher correlation between the dependent variable and independent variables with larger standard deviations, and when the correlations of each independent variable with each other independent variable are as low as
possible.12
Due to greater variability, the achievement in the external examination is in a better position compared to the achievement attained in the 8th grade concerning the possibility of it being established in the correlation with the achievement in the 1st year both when it is on its own as an independent variable and when it is in an ordinary sum together with the achievement in the 8th grade, regardless of whether this sum contributes to the share of explained variance of the dependent variable or not. In terms of the selection procedure, it is essential that the achievement in the external examination, if the acceptance criterion is the ordinary sum of the achievement (grade) attained in the 8th grade and the former achievement, has a relatively excessive weight in the selection of candidates. The problem or the inappropriate approach here is that knowledge is measured in two different units of measurement, and the results of measurement achieved are simply added into an ordinary sum as if the same unit of measurement were used in both measurements.
As opposed to the acceptance criterion in the form of an ordinary sum of independent variables, in the use of multiple linear regression, independent variables obtain appropriate weightings in the form of partial regression coefficients.
Let us see the result in our case if we use multiple linear regression. Here, independent variables will not be added into an ordinary sum, each of them remaining autonomous. In our case, the coefficient of multiple linear correlation between the achievement in mathematics in the 8th grade and the achievement in the external examination as two independent variables, and the achievement in mathematics in the 1st year as a dependent variable is R = 0.492, and the relevant determination coefficient is R2 = 0.242, which means that 24.2% of the variance in the achievement in mathematics in the 1st year is explained by the achievement in mathematics in the 8th grade and by the achievement in the external examination, i.e. by their combined linear influence on this dependent variable.13 The share of explained variance increased by 1.9 percentage points compared to the share of explained variance when only the achievement in the
12 This conclusion was reached on the basis of a formula for the correlation between the sum of several variables and a variable.
13 In sample surveys, it is taken into account that the sample value of the coefficient of a multiple linear correlation is not an unbiased assessment of the population value. There is a similar situation concerning the square of this correlation coefficient, i.e. the determination coefficient. Consequently, we (or the computer) use a formula with which we obtain an appropriate adjusted assessment of the determination coefficient. E.g. see an appropriate formula in Sagadin 2003, p. 291, formula (319). The bias mentioned decreases with the growth in the size of the sample and increases with the growth in the number of independent variables. In large samples and with only two independent variables, the difference between the adjusted and ordinary value R2 is not large.
98     SODOBNA PEDAGOGIKA 5/2006
J. Sagadin
8th grade was used as the predictor, by 11 percentage points compared to the share of explained variance when we used the sum of the achievement in the 8th grade and the achievement in the external examination as the predictor, and by 14.4 percentage points compared to the achievement in the external examination alone as the predictor. If the data of a large sample, representative for a certain generation or population of new gimnazija students, were to result in such findings and if we had to opt for the most appropriate predictor in mathematics we could conclude that it would actually be best to only take the achievement attained in the 8th grade as a predictor, as with an ordinary sum of the 8th grade achievement and the achievement in the external examination the share of explained variance in the achievement in the 1st year is even lower compared to the achievement in the 8th grade as a predictor on its own, and multiple linear regression also does not contribute so much that it would be worth the trouble. Of course, this is not always the case.
Let us add that in cases like the above it is also possible in sample surveys to test the statistical significance of the difference between the share of explained variance when multiple linear regression is used and the share when only the 8th grade achievement is used as the predictor.
By restricting ourselves to mathematics, we wished to demonstrate the data analysis on the way to selecting the appropriate independent variable (predictor). Now we are moving on to combinations of achievement (overall achievement and achievement in individual subjects) attained in primary school, and achievement in external examinations which, from the point of view of the Slovenian practice concerning the candidate-selection criteria, are even
of greater interest.14
Let us take (to make our considerations more transparent) the sum of the overall achievements and grades in Slovenian language, mathematics and foreign language from the 6th to 8th grade as an independent variable, and the sum of the achievements in external examinations in Slovenian language and mathematics as a second independent variable.15 The overall achievement from the 1st year of secondary school will first be taken as the dependent variable. The correlation between the first independent variable and the dependent variable is 0.274, the correlation between the second independent variable and the dependent variable is 0.306, and the correlation between the sum of independent variables and the dependent variable is 0.363 (and the determination coefficient is 0.132). All these correlations are relatively low, but this does not interest us here, with the essential finding being that in this case the sum of independent variables increased the correlation with the dependent variable compared to the case when a single independent variable was used, which means that it is better to take the sum of both independent variables as the gimnazija candidate-selection
14 We have already mentioned in note 8 how the primary school achievement and achievement in external examinations were included in the candidate-selection criteria at the time the (non-representative) data we use come from.
15 This fits the candidate-selection criterion at the time our data come from.
Methodological problems concerning the criteria ...
99
criterion than to only take the first or the second variable. For this increase in the correlation with the dependent variable, the quite low correlation (0.255) between the independent variables was favourable.
With the use of multiple linear regression, in this case the coefficient of multiple linear correlation would be R = 0.367 and the determination coefficient would be R2 = 0.135, which means that the share of explained variance in the dependent variable would only rise by 0.3 of a percentage point compared to the use of the ordinary sum of independent variables. If such a small increase in the share of explained variance in the dependent variable were determined for the large representative sample, that would mean it is better to take as the candidate-selection criterion the sum of independent variables, the use of which is technically simple, and that it is not worth bothering with the procedure involving multiple regression.
However, the outcome will not always be like this. For instance, in Ko`uh’s abovementioned research on the generations of new secondary school students in l992/1993, 1993/1994 and 1994/1995, the sample estimate of correlation between the sum of points covering the overall achievements and grades in the Slovenian language, mathematics and foreign language attained from the 5th to 8th grade of (eight-year) primary school and the points attained in the external examinations in mathematics and Slovenian language as an independent variable, and the overall achievement in the 1st year as a dependent variable, did not, for any of the generations, exceed the correlation between the sum of points from primary school (alone) (the sum of overall achievements and grades in Slovenian language, mathematics and foreign languages from the 5th to 8th grade) as the independent variable and the overall achievement in the 1st year of the secondary school as the dependent variable.16 (Points for the Vega and Cankar Awards were not included).
Now, our data will be used to conduct an analysis as described above, only that this time the dependent variables will be the sum of the overall achievement and the grades in Slovenian language, mathematics and foreign language in the 1st year of secondary school (above, the dependent variables were only limited to the overall achievement in the 1st year). With the dependent variable defined in this way, the outcome of data analysis is the following: the correlation between the first independent variable (the sum of overall achievements and grades in Slovenian language, mathematics and foreign language from the 6th to 8th grade) and the new dependent variable is 0.243; the correlation between the second independent variable (the sum of points attained in external examinations in mathematics and Slovenian language) and the dependent variable is 0.454; the correlation between the sum of independent variables and the dependent variable is 0.426; the coefficient of multiple linear correlation between the independent variables and the dependent variable is 0.476. All these correlations differ from those determined when the dependent variable
16 In this case, the two predictors are also actually sums of several variables, but that does not disable this demonstration.
100   SODOBNA PEDAGOGIKA 5/2006
J. Sagadin
was limited to the overall achievement in the 1st year alone. Thus, also all the predictive validities changed: the predictive validity of individual independent variables, the predictive validity of the sum of independent variables, and the predictive validity of independent variables determined with the use of multiple regression. Thus, we also demonstrated the fact that the predictive validity of the candidate-selection criterion also depends on the definition of the dependent variable (response variable), not only on the definition of the criterion itself.
As far as predictive validity is concerned, the independent variable in the form of the sum of overall achievements and grades in Slovenian language, mathematics and foreign language attained in the 6th to 8th grade did not turn out to be successful when compared to the second independent variable, in connection with the dependent variable in the form of overall achievement in the 1st year, nor in the second case when the dependent variable had the form of the sum of the overall achievement and the grades in Slovenian language, mathematics and foreign language in the 1st year of secondary school. This finding is connected with our non-representative data and must not be generalised. At the same time, it has to be emphasised that the outcome for the primary school achievement is not always so unfavourable.17
One should know that an independent variable in the form of the sum of overall achievements and grades in Slovenian language, mathematics and foreign language attained from the 6th to 8th grade is also the sum of as many independent variables as it contains elements added together, and that the ordinary addition of independent variables does not always contribute to the predictive validity, as we found above. For instance, if the independent variable were only defined as the sum of the overall achievement and grades in Slovenian language, mathematics and foreign language in the 8th grade, with our data the correlation of this variable with the overall achievement in the 1st year would be 0.497, and the correlation with the sum of the overall achievement and grades in Slovenian language, mathematics and foreign language in the 1st year would be 0.495, which means that concerning these correlations the independent variable defined in this way would overtake the sum of points attained in the two external examinations, while the correlation of the independent variable in the form of the sum of overall achievement and grades in Slovenian language, mathematics and foreign language from the 6th to 8th grade with the achievement in the 1st grade is 0.274, and the correlation with the overall achievement and
17 For instance, as we have seen, in Kožuh’s research on representative samples of the generations of new secondary school students involved, a similar independent variable – in the form of the sum of overall achievements and grades in Slovenian, language, mathematics and foreign language attained from the 5th to the 8th grade – proved more successful than the independent variable in the form of the sum that, in addition to these overall achievements and grades from the 5th to 8th grade, also included the points attained in the external examinations in mathematics and Slovenian language. However, also in this case the correlations between the sum of overall achievement and grades from the 5th and 8th grade and the overall achievement in the 1st year were not high (0.40 for the generation 1992/1993, 0.45 for 1993/1994 and 0.45 for the generation 1994/1995). (See Kožuh 1997, p. 389.)
Methodological problems concerning the criteria ...
101
grades in Slovenian language, mathematics and foreign language in the 1st year is 0.243, which is less than the relevant correlations in external examinations. However, if the gimnazija candidate-selection criterion were only to take into account the sum of overall achievement and grades in Slovenian language, mathematics and foreign language in the 8th grade of primary school, the sum of points attained in the external examinations in mathematics and Slovenian language would (could) have an excessive weight in the acceptance criterion compared to primary school achievement, and there could also be problems in the classification of enrolment candidates. It would be different if the candidate selection criterion were to include the sum of overall achievement and all numerical grades in the 8th grade. Then, (with our data) the correlation between the 8th grade achievement included in this way and the overall achievement in the 1st year would be 0.600, while the correlation with the sum of the overall achievement and numerical grade in the 1st year would even be 0.652.
In the addition of overall achievement and the grades in Slovenian language, mathematics and foreign language, we may ask ourselves whether it is appropriate to change the categories of overall achievement into numerical values from 1 to 5 (of course, with enrolment candidates and the accepted candidates there is no value 1) and place it, according to its weight in the sum, at an equal level as a grade in an individual subject. Such a weighting of the categories of overall achievement is less disputable if the overall achievement is not only added to the grades in the three subjects mentioned, but with all numerical grades (as we have already done above) as in this case the weight of the overall achievement in the sum only represents some kind of an ’addition value’ to that expressed by the sum of all numerical grades.
3      Conclusion
The problem of what to use in the current situation as the candidate-selection criterion must be resolved through research, with the results of surveys conducted in the past possibly being of aid although they cannot be mechanically transferred to the present time. The predictive validity of school results (overall achievement and achievement in individual subjects) varies from generation to generation of new students, as well as within the same generation from school to school. Consequently, the predictive validity must be rechecked through research from time to time. Similar is true of the predictive validity of the results of other examinations. The predictive validity of school results as a predictor (independent variable) also depends on what is included in such a predictor and on the definition of the dependent variable. Again, similar is true of other knowledge examinations.
It is not the same how we combine primary school results and achievements in other examinations in the candidate-selection criterion; this decision must be justified with an analysis of the predictive validity of school results, achievements in other examinations and possible combinations of both types
102   SODOBNA PEDAGOGIKA 5/2006
J. Sagadin
of achievement. In principle, it is reasonable to include in the criterion, in addition to the independent variable in the form of appropriately covered primary school achievement (overall achievement and grades in individual subjects), an independent variable in the form of the achievement in some other examinations of primary school knowledge, like well-prepared and performed external examinations or entrance examinations. However, the advantage of such a combination is always realised, as we have said, only when a model of multiple regression is used, with the independent variables obtaining appropriate weights in the form of partial regression coefficients and thus each given an adequate weight in the selection procedure. (We also pointed out reservations concerning the use of this model.) We also emphasised and demonstrated that the ordinary addition of independent variables does not always lead to an increased predictive validity of the candidate-selection criterion. This must be taken into account in the search for solutions in this way.
Regarding the survey conducted on the generation of new gimnazija students in 1966/1967, we also indicated a possibility of a different solution, adjusted to the circumstances concerning the then entrance examinations. Perhaps, in connection with external examinations the solution should also be searched for in similar directions.
The fact that the ordinary addition of independent variables does not always result in an increase in the predictive validity must also be taken into account in the candidate-selection criterion if only primary school achievement is included. From the point of view of predictive validity, it is not the same which elements of primary school achievement are included in the criterion or which elements are added together, for instance, whether the sum only includes the overall achievement and grades attained in the last year or those attained over several years. Mere care that the criterion enables the classification of candidates by achievement and thus their selection does not guarantee that the selection is also made adequately in terms of the predictive validity.
Our contribution has concerned the methodological problems regarding candidate-selection criteria when enrolment is limited from the point of view of entire enrolment generations. However, we also implicitly pointed out that it is difficult to form criteria fitting all gimnazije equally and adequately. In this sense, a special problem is selection situations in those gimnazije for which there are many candidates whose top primary school results are so similar and whose results in the present external examinations are also so similar that these two criteria may become insufficiently sensitive to the differences between the candidates and that it will therefore be difficult to make a selection by using these criteria alone (and even more difficult by only using primary school results). In such potential situations, it will be necessary to seek a way out via an additional criterion or criteria appropriately differentiating the candidates according to their knowledge.
In all of this, one should not forget the possible backwash effect of the candidate-selection criteria on the primary school, including favourable and unfavourable ones.
Methodological problems concerning the criteria ...
103
References
Bucik, V. (1997). Osnove psiholo{kega testiranja. Ljubljana: Filozofska fakulteta.
Ferguson, G. A., Takane, Y. (1989). Statistical Analysis in Psychology and Education (6th ed.). New York etc.: McGraw-Hill.
Guilford, J. P. (1968). Osnovi psiholo{ke i pedago{ke statistike. Beograd: Savremena administracija.
Horst, P. (1971). Messung und Vorhersage: Eine Einführung in die psychologische Testtheorie. Weinheim, Berlin, Basel: Verag Julius Beltz.
Ko{melj, B. et al. (2001). Statisti~ni terminolo{ki slovar. Ljubljana: Statisti~no dru{tvo Slovenije in Statisti~ni urad Republike Slovenije.
Ko`uh, B. (1997). Dosedanje izku{nje z zunanjim testiranjem u~encev osnovne {ole. V zborniku: Kongres pedago{kih delavcev Slovenije: Programska prenova na{e osnovne in srednje {ole, p. 386–390. Portoro`: Zveza dru{tev pedago{kih delavcev Slovenije.
Pedhazur, E. J. (1997). Multiple Regression in Behavioral Research: Explanation and Prediction (3rd ed.). Fort Worth, Philadelphia etc.: Harcourt Brace College Publishers.
Sagadin, J. (1968). Prognosti~na veljavnost sedanjih meril za sprejem novincev v gimnazijo v SR Sloveniji. Ljubljana: Zavod za {olstvo SRS.
Sagadin, J. (1993). Poglavja iz pedago{ke metodologije (2. ed.). Ljubljana: Zavod Republike Slovenije za {olstvo in {port.
Sagadin, J. (2003). Statisti~ne metode za pedagoge. Maribor: Obzorja.
Tacq, J. (1997). Multivariate Analysis Techniques in Social Science Research: From Problem to Analysis. London etc.: Sage Publications.