Organizacija, Volume 40 Research papers Number 6, November-December 2007 Simulation Based Group Learning Mirjana Kljajič Borštnar, Andrej Škraba, Vladislav Rajkovič, Miroljub Kljajič University of Maribor, Faculty of Organizational Sciences Kidriceva 55a, SI - 4000 Kranj, Slovenia, [mirjana.kljajic, andrej.skraba, vladislav.rajkovic, miroljub.kljajic]@fov.uni-mb.si This article describes an experiment investigating simulation based group learning. For this purpose, we have conducted a four-group Solomon experiment under four different conditions: a1) the determination of strategy with the application of the system dynamics (SD) model without group interaction and with a pre-test, a2) the determination of strategy with the application of the SD model and group information feedback and with a pre-test, a3) the determination of strategy with the application of the SD model and without a pre-test, and a4) strategy determination with the application of the SD model and group information feedback and without a pre-test. The observed variables were the criteria function values and frequency of the simulation runs. The hypothesis that simulation model application and group feedback information positively influence the convergence of the decision process and contribute to faster decision-making was confirmed. A model of the learning during the decision-making process was developed. Students’ opinions were analyzed as well and the results show that management students thought that the application of the simulation model did contribute to an increased understanding of the problem, the faster finding of solutions and the increased confidence of participants. All participants agree that the clear presentation of the problem motivates participants to find the solution. Key words: group decision, learning model, system dynamics, feedback, experiment design Učenje, podprto s simulacijskim modelom V prispevku so raziskani principi u~enja, podprtega s simulacijskim modelom. V ta namen smo izvedli Solomonov poskus štirih skupin pod naslednjimi pogoji: a1) definiranje strategije, podprto s simulacijskim modelom brez sodelovanja skupine s pred-testom, a2) definiranje strategije, podprto s simulacijskim modelom s sodelovanjem skupine s predtestom, a3) definiranje strategije, podprto s simulacijskim modelom brez sodelovanja skupine brez pre-testa (v zveznem ~asu) ter a4) definiranje strategije, podprto s simulacijskim modelom s sodelovanjem skupine brez pre-testa (v zveznem ~asu). Pri tem smo opazovali spremenljivki: vrednost kriterijske funkcije (kvaliteta odlo~itve) in pogostost simulacijskih tekov (dinamika iskanja rešitve). Domneva, da simulacijski model in sodelovanje skupine pozitivno vplivata na enostnost skupine in prispevata k hitrejšemu odlo~anju je bila potrjena. Razvili smo model, ki ponazarja u~enje v procesu odlo~anja. Izvedli smo tudi mnenjsko anketo udele`encev poskusa. Rezultati ankete ka`ejo, da se študentje managementa strinjajo s trditvami, da uporaba siulacijskega modela v podporo odlo~anju pripomore k boljšemu razumevanju problema, hitrejšemu odlo~anju ter ve~jemu zaupanju udele`encev. Udele`enci so si enotni, da jasna predstavitev motivira udele`ence k reševanju problema. Ključne besede: skupinsko odlo~anje, model u~enja, sistemska dinamika, povratna zanka, na~rt poskusa 1 Introduction The decision processes in contemporary enterprises are primarily based on the participating subjects. Decisions generated in organizational systems are, therefore, not dependent on an individual decision on a subject but rather on a group of experts working in a specific field. The group better understands the system in question and provides synergistic effects (Hale, 1997). Their interaction in the process of problem solving (decision-making), supported by advanced group support tools and interactive business simulators, could enable more effective individual and group analyses of the problem (Vennix, 1996; Richardson and Andersen, 1995; Kwok and Khalifa, 1998; Langley and Morecroft, 2004; Škraba et al. 2003). Quality decisions can only be made if the decision group has the appropriate information: both anticipative and as feedback. This assumes knowledge of a model of a system, the criteria function and the state of nature. These have been intensively discussed in the literature (Chekland, 1994; Forrester, 1961; Rosen, 1985; Simon, 1997; Sterman, 1994, 2000). The ideal for learning organizations can be approached by the application of SD models (Warren in Langley, 1999). The use of SD models for testing the vision of the evolution of business systems is widely used (Forester, 1961; Simon, 1997; Sterman, 2000). However, the connection of SD models with group support systems (GSS) for the purpose of decision-making support is not 239 Organizacija, Volume 40 Research papers Number 6, November-December 2007 commonly used and researched. An interesting model, intended to explain group learning phenomena, was described in (Lizeo, 2005), where the group learning process was modelled from structural, interpersonal and cognitive factors in the form of a causal loop diagram (CLD) and an SD technique. Experiential learning, as in learning from an enterprise simulation, is researched in the experiment by Gopinath and Sawyer (1999), where the effects of learning during determination of broader business strategy on a business simulator were examined. The application of SD models for strategy determination encourages strategic decision-making and systematic work. In the experiment with the global oil microworld computer of Langley and Morecroft (2004), they explore the effects of various types of feedback on individual learning (outcome feedback and structure feedback). The results suggest that structure feedback positively influences the understanding of the problem and the time for the completion of the task. However, in complex systems, to make a formal experiment in order to prove that efficacy and usefulness of group decision and using a simulation model for decision assessment is a demanding task. There are problems of validity in the design of the research (Chun and Park, 1998). It is difficult to create a laboratory environment in which subjects are motivated to creatively participate in finding the solution as they would in a real world. The dilemma is also in planning the problem (organizational systems), which is inherently complex. There is also the problem of the layout of the interface, as it affects the effectiveness of the subject in the process of problem solving (Howie et al, 2000). Three learning methods (case learning, simulation method and action learning) were researched by Jennings (2002). The participants rated the simulation method as superior to the action learning and case learning methods. In the paper by Škraba et al. (2003), the process of strategy determination was described as well as the impact of group interaction on subject performance by applying the SD model of a simplified business process. The hypothesis that the model application and group information feedback positively influence the convergence of the decision process and contribute to increased criteria function values was confirmed. The experiment was later enhanced with a new group in order to analyze criteria function as well as dynamics of using a simulation model while searching for optimal parameters (Kljaji} Borštnar et al, 2006). The goal of the repeated experiment was to acquire knowledge of the dynamics of the decision process supported by the SD model and the influence of group feedback information. Although the results of the criteria function were similar to the previous experiments, it was surprising that the frequency distribution was different among experimental groups at the beginning of the experiment. The decision-making process was divided into four time intervals; in the first interval, the technical conditions were the same for both the groups using the simulation model. When the first time interval elapsed, subjects had to submit their decisions to the network server. After submitting their decisions, one of the groups continued working individually with the simulator and the other group received information about the decisions made by other group members - group information feedback. The difference in the frequency of the simulation runs suggested that group membership might have affected the group work. This paper describes the four-group Solomon experiment based on the following hypothesis: H1) Individual information feedback introduced into the decision-making process by a simulation model contributes to higher criteria function values (individual learning). H2) Group information feedback introduced into decisionmaking process by a group support system contributes to an increased convergence of the group and increased criteria function values (group leaning). H3) The interaction of the pre-test (group process facilitation) and treatment (group information feedback) contributes to a higher frequency of simulation runs in the search of optimal parameter values. The results of the experiment confirm the hypothesis; the learning model developed in the causal loop diagram technique explains learning under different conditions. 2 Method 2.1 Simulation Model Figure 1 shows the model of the production process as a black box with input parameters r1, r2, r3 and r4 (where r1 is Product Price, r2 Salary, r3 Marketing Costs and r4 the Desired Inventory) and the criteria function J as the output under experimental conditions a1), a2), a3) and a4), described later in the text. The task of the participants is to find the parameter values ri in order to maximize the criteria function. Figure 1: Business model with input parameters under different experimental conditions In Figure 1, ai represents four experimental (decisionmaking) conditions described later in the text. The model developed by the SD method, which was used in the experiment, is shown in Figure 2. The model described in 240 Organizacija, Volume 40 Research papers Number 6, November-December 2007 (Škraba et al, 2003) consists of the production, workforce and marketing segments, which are well known in literature (Forrester 1961; Hines 1996; Sterman 2000). It was stated that the product price (r1) positively influences income. However, as prices increase, demand decreases below the level it would otherwise have been. Therefore, the proper pricing that customers would accept can be determined. If the marketing costs (r3) increase, demand increases above what it would have been as a result of marketing campaigns. The production system must provide the proper inventory level to cover the demand, which is achieved through the proper determination of the desired inventory value (r4). Surplus inventory creates unwanted costs due to warehousing; therefore, these costs must be considered. The number of workers employed is dependent on production volume and workforce productivity, which is stimulated through the salaries (r2). Proper stimulation should provide reasonable productivity. Figure 2: Causal loop diagram of a production model Participants had the task of promoting a product with a one-year life cycle on the market. They had to find the proper values of parameters ri defined by the interval rmin < ri < rmax. The model was prepared in the form of a business simulator (Škraba et al, 2003). The participants changed the parameter values via a user interface, which incorporated sliders and input fields. After setting the parameters in the control panel, the simulation could be processed. The end time of the simulation was set to twelve months. The output was shown on graphs representing the dynamic response of the system and in the form of a table where numerical values could be observed. Each participant had no limitation to the simulation runs, which he/she intended to execute within the time frame of the experiment. The parameter values for each simulation run were set only once, at the start of the simulation. It was assumed that the business plan was made for one year ahead. The criteria function was stated as the sum of several ratios that were easily understood and known to the participants. It was determined that the Capital Return Ratio (CRR) and Overall Effectiveness Ratio (OER) should be maximized with minimal Workforce and Inventory costs determined by a Workforce Effectiveness Ratio (WER) and Inventory / Income Ratio (IIR). The simulator enabled simultaneous observation of the system response for all the variables stated by the criteria function during the experiment. 2.2 The Solomon Four-group Experimental Design Although Hypotheses 1 and 2 have already been confirmed by previous experiments described in (Škraba et al, 2003; Škraba et al, 2007), Hypothesis 3 remained unexplained. Due to the homogeneity of the population and its random allocation into groups, We expected that the results of the criteria function and the frequency of testing in the first 8 minutes would be identical. However, a difference was noted in the temporal course of the variables. This phenomenon cannot be explained by the pre-test -post-test experiment in (Škraba et al, 2003; Škraba et al, 2007). Therefore, we conducted a new experiment according to the Solomon Four-Group Experimental Design. We expect to estimate the effect of group belonging (as a result of the group information feedback that was introduced) and the pre-test effect (as a result of facilitation of the group decision process) on the decision-making results (criteria function value) using this test. Solomon’s design for the suggested experiment is shown in Figure 3. Figure 3: A Solomon four group experiment design; R means random, Oi means observed and X is the treatment groups. Figure 3 shows the random assignment of the population of senior management students into four decision groups. The first two groups in Figure 3 represent the pretest - post-test design (decision groups are facilitated and measured four times during the experiment, after the 8th, 16th, 24th, and at the end after the 30th minute). The last two groups represent the post-test only design. All four groups were supported by a simulation model of a business system. One of each of the two groups (a1 and a2) had additional group information feedback at their disposal. Thus we could asses whether the interaction between the pretest (in our case this also means the facilitation of the group decision process) and the treatment (group infor- 241 Organizacija, Volume 40 Research papers Number 6, November-December 2007 mation feedback) exists. In pre-testing, the subjects were directed by a facilitator. They were told to submit their best chosen parameter values into the network database. After the submission, they continued with the search for the optimal combination of the parameter values. On the other hand, the decision-making process of the two groups working without pre-tests was continuous, without facilitation. All the measurements were automatic and group information feedback was available at all times. For this purpose, we developed a new interface for data acquisition and proceeding. 2.3 Subjects and Procedure 118 senior undergraduate students (52 female and 66 male, between the ages of 20 and 26) from the University of Maribor participated in the experiment in order to meet the requirements of their regular syllabus. The students were randomly assigned to eight groups with 14 to 15 subjects, who were then assigned to work under one of the four experimental conditions: a1, a2 a3 and a4. The subjects who participated in the experiment became accustomed to the business management role facing the stated goal objective, which was in our case presented in the form of a criteria function. A presentation of the decision problem was prepared in the form of uniform 11-minute video presentation, which differed only in the explanation of the experimental condition at the end of each video presentation. The problem, the task and the business model were explained. The structure of the system considered was presented and the main parameters of the model were explained. The evaluation criteria for the business strategies were also considered. The work with the simulator was thoroughly explained in the video. A printed version of the problem description was provided for each subject as well. The participating subjects were familiar with SD simulators; therefore, working with the simulator was not a technical problem. Subjects were awarded for their participation in the experiment with a bonus grade. 2.4 Experimental Conditions: a1) an individual decision-making process supported by a simulation model with testing after the 8th, 16th, 24th and 30th minutes, assumes that each participant submitted the best-achieved set of parameter values {r1, r2, r3, r4} to the network server at the end of each time interval. a2) a decision-making process supported by a simulation model and group information feedback with testing after the 8th, 16th, 24th and 30th minute. Each participant submitted the best-achieved set of parameter values {r1, r2, r3, r4} to the network server at the end of each time interval. Information on the best-achieved parameter values was fed back into the group support system. The participants got feedback on the defined strategies of all the participants in the group Ri = {r1, r2, r3, r4}; i = 1,2,…n as well as the aggregated values in the form of parameter mean values { r 1 , r 2, r 3 , r 4 } . For example, if the parameter considered was Product Price and there were ten participants involved in the decision process, then all ten values for Product Price, recognized as the best by each participant, were mediated via feedback as well as the mean value of the Product Price. The mean value provided the orientation for the parameter search and prevented information overload. In addition to the criteria function as the results of decision making under different conditions, simulation frequency in order to follow the decision makers’ activity was also analyzed. a3) an individual decision-making process supported by a simulation model without a pre-test (testing after the 30 th min) assumed the individual assessment of the decision-maker when determining model parameter values {r 1, r2, r3, r4} by the maximization of the criteria function using the SD model. At the end of the experiment, the subjects submitted the best-achieved parameter values to the network server. a4) a decision-making process supported by a simulation model and continuous group information feedback without the pre-test (testing after 30th min). Each participant submitted the best-achieved set of parameter values {r 1, r2, r3, r4} to the network server at the end of experiment. However, information about the instantaneous optimization of the group is always at the subjects’ disposal. 3 Results and Discussion A total of 118 students (52 female and 66 male), randomly assigned into 8 groups of 14 to 15 subjects, participated in the experiment. 30 students (two groups) participated in condition a1, 29 students (two groups) in condition a2, 30 students (two groups) in condition a3 and 29 (two groups) in the experimental condition a4. For the purpose of analysis of the results, the criteria function was optimized by Powersim SolverTM using two methods: incremental and genetic algorithms. The optimal value of the criteria function was thus set to 1.5. The highest values of the criteria function were selected by the participants of group a2 ( J a2 =1,237 , ?a2 =0,210), followed by the results of group a1 ( Ja1 =1,170 , ?al =0,338) and the results of group a4 ( Ja4 =1,157 , ?a4 =0,290 ). The lowest results were gathered by the group a3, supported by the simulation model ( Ja3 =1,147, ?a3 =0,272 ). The criteria function values selected by the participants working under four different conditions after 30 minutes of experiment time are presented in Figure 4. On the X-axis, the number of participants is shown and on Y-axis the values of criteria function are arranged in ascending order. Figure 4 clearly shows that the selected criteria function values for the four experimental conditions does not differ significantly (this is confirmed by a Kruskal-Wallis test at p=.677). This supports our prior experiment results, where we proved that 30 minu- 242 Organizacija, Volume 40 Research papers Number 6, November-December 2007 tes is sufficient time for solving this particular decisionmaking problem when supported by a simulation model (Škraba et al, 2007). 1,50 1,25 1,00 ~> 0,75 0,50 0,25 0,00 10 15 20 25 No. of Subjects a3 ^^ - a4 Figure 4: Criteria function values achieved by the participants under experimental conditions: a1, a2, a3, and a4. Nevertheless, we continue to present the in-depth analyzes of the dynamics of the decision-making process. 3.1 Learning During the Decision-Making Process Figure 5 shows the Coefficient of Variation of the criteria function values achieved by the participants under experimental conditions: a1 and a2 at the end of each time interval (pre-test and post-test). The results of Friedman’s ANOVA confirmed that criteria function values increase during the experiment time (%a1=30.57, pa1 =.000; %a2=27.30, pa2 =.000), therefore we can conclude that learning takes place during the decision-making process. The results show that the subjects’ decisions did not differ after the first eight minutes, when the same conditions were in place. This was confirmed by a Mann-Whitney test (U=415) at p=.762. After group a2 had received group information feedback, they rapidly approached the optimum criteria function value. The biggest increase in criteria function values is observed after the first time group information feedback was introduced (after the 16th minute), confirmed by a Wilcoxon test (z=-2.995, p=.002). Criteria function values significantly increase after the 24th minute (confirmed by a Wilcoxon test, z=-3.165, p=.001), but hardly changed towards the end of the experiment (in the last eight minutes). This was confirmed by a Wilcoxon test (Z=-.660, p=.510). On the other hand, the group without group information feedback slowly continues to approach the optimal solution and significantly 0,8 0,7 *o 0,6 ro 0,5 *- 0,4 o 0,3 O 0,2 0,1 0,0 16 min 24 min 30 min Time interval Figure 5: the Coefficient of Variation of criteria function values (J) achieved by the participants under experimental conditions: a1, a2 at the end of each time interval improves their results in the final phase of the experiment (after 30th minute). A Wilcoxon test confirmed that the criteria function values significantly improved after each experimental phase (z 1 -2.584, p1=.009; z2=-2.259, z2=.023; z3=-2.869, p3=.004). This means that group a2 took eight minutes less to solve the decision-making problem than group a1. The results prove that learning occurs in the decision-making process supported by the simulation model. On the basis of analysis, we can conclude that the group information feedback introduced into the decisionmaking process contributes to a higher convergence of the decision group and helps achieve faster decision problem solving. 3.2 Analysis of Feedback-Seeking Behaviour in Two Treatment Groups In addition to recording every simulation run executed by a subject, we also recorded every insight into group information feedback. Group information feedback was available to the subjects from the non-pre-test group (a4) at all times from the beginning of the experiment, while the pre-tested group (a2) had group information feedback introduced after each time they had to submit their decisions to the network database. Figure 6 shows the feedback seeking behaviour (an insight into group information feedback) of two groups during the experiment by minute, and Figure 7 shows the number of simulation runs of the two groups per minute during the experiment. We have confirmed with a Mann-Whitney test that the feedback seeking behaviour for group information feedback of the pre-test and non-pre-test treatment groups differs significantly (U=202, p=.001). While group a2 had shown great interest in the group information feedback and an almost constant interest in simulation runs, the interest of group a4 in group information feedback and simulation runs increased almost proportionally. In fact, the frequency of simulation runs for group a2 is almost twice as high as group a4 at the beginning of the experiment and 8 min a1 a2 0 5 30 a1 a2 243 Organizacija, Volume 40 Research papers Number 6, November-December 2007 then decreased after the 24th minute, while the subjects of group a4 had continued to increase the frequency of the simulation runs. We can explain this by 40% of the subjects who stopped performing simulation runs in the last experiment phase (after the 24th minute). These were the subjects that had already approached the optimal solution. L 5 — 4 ^ 3 a2 -+- 8min 16 min 24min 30min Time Phase of the Experiment Figure 6: Feedback seeking behaviour (insight into group information feedback per minute) of groups a2 and a4 "nT 70 ^ 60 c 50 F a4 - F a4 F a2 F a2 L~- ¦ F a2 ~ F a2 ^^ F a4 ^* F a4 Time Phase of the Experiment Figure 7: Frequency of simulation runs per minute during the experiment time of groups a2 and a4 In order to prove that a correlation between the frequency of simulation runs and criteria function value exists, we have performed the Spearman r test. The test confirmed that a reasonably strong correlation exists between the frequency of simulation runs and the criteria function value under experimental conditions a1 (r=.443, p=.014), a3 (r=.432, p=.017) and a4 (r=.500, p=.005), but not under condition a2 (r=.231, p=.227). 3.3 The Interaction of Pre-test and Treatment Figure 8 shows the frequency of simulation runs at the pre-test and post-test (8th and 30th minute) for all four experimental conditions. It is noticeable that the frequency of group a2 (pre-test treatment group) is slightly higher in the first eight minutes than the frequency of the pre-te-sted non-treatment group a1 and that both have higher frequencies than the two non-pre-tested groups (a3 and a4). Towards the end of experiment time, all the groups show an equidistant increase in frequency, except for group a2 (pre-test plus treatment). The groups’ frequency of simulation runs is almost constant. 2,45 2,15 1,85 1,55 1,25 0,95 0,65 ^A a1 a2 a2 a1 ^1 a4 ( y * 4 ¦ a4 S *' ^ ^a3 ¦^a3 "^ pretest posttest test -a2 — Figure 8: Solomon test for the Frequency of simulation runs From Figure 8 we can conclude that the pre-test influenced the number of simulation runs performed. Also, it is evident from Figure 8 that group information feedback impacts the number of simulation runs performed. We have conducted the two way ANOVA test, which confirmed that treatment alone (group information feedback) does not influence the frequency of simulation runs (F=.000, p=.9982), that pre-test (facilitation of the decision process) influences the frequency of simulation runs (F=6.895, p=.01) and that interaction between the pre-test and treatment together influence the frequency of the simulation runs (F=4.076, p=.046). 3.4 Learning Model In order to explain the influence of individual information feedbacks (assured by the simulation model) and group information feedback (brought into the decisionmaking process by GSS) on the efficacy of problem solving, we have developed a CLD model of learning during the decision-making process. The model shown in Figure 9 was modified according to (Lizeo, 2005) and consists of three B and one R loops. Loop B1 represents the decision-making process supported just by a formal CLD model (see Figure 2), paper and pen (described in Škraba et al, 2003; Škraba et al, 2007). The decision maker solves the problem by understanding the problem and the task. The higher the gap between the goal and the performance, the more effort one should put into understanding the problem. Loop B2 a4 a4 a a1 a3 a4 80 40 30 0 8m n 16min 24m n 30mn 244 Organizacija, Volume 40 Research papers Number 6, November-December 2007 and goal (in our case the optimized criteria function). The correlation between the frequency of the simulation runs and the criteria function value was confirmed (pa1=.014; pa3=.017). We named this loop “Individual Learning Supported by Simulator”. Loop B3 represents the direct contribution of the group information feedback, while loop R suggests the reinforcing effects of the group influence on problem solving at group a2 and a4. The decision maker of loop B3 understands the problem and the goal. He or she is supported by the simulator and the group information feedback. While the use of the simulator supports individual learning, the introduced group information feedback enhances the group performance. Consequently, the increased group performance reduces the need to experiment on the simulator. In other words, the decision maker supported by the group information feedback has a broader view of the problem, an insight into new ideas and needs to put less effort into problem solving. On the other hand, the group information feedback stimulates the group members to actively participate in the problem solving, so they perform more simulation runs in the process of searching for the solution. This can be observed from Figures 6, 7 and 8. The frequency of simulation runs in group a2 is higher than that of the other groups’ in the first 16 minutes of the experiment, when the majority of the subjects were still searching for the solution. When the group is satisfied with its performance, the frequency of the simulation runs decreases. Loop R can be further explained by the interaction between group information feedback and facilitation of the decision-making process. As we have observed in Figures 6 and 7 (and is confirmed by a two-way ANOVA), the group information feedback together with facilitation contributes to increased feed- Table 1: Average agreement with the statements in the opinion questionnaire and its standard deviation Experimental Condition Q Short descpription of the question a1 a2 a3 a4 1 the general quality of the experiment 5,733 (0,785) 5,724 (0,996) 5,867 (0,900) 5,483 (1,022) 2 presentation of the decision problem 5,733 (0,980) 5,552 (1,183) 5,833 (0,791) 5,379 (1,208) 3 understanding the decision problem 5,833 (1,392) 5,690 (1,256) 5,733 (0,944) 5,448 (1,378) 4 the simplicity of the use of simulator 6,600 (0,498) 6,586 (0,733) 6,067 (1,143) 6,103 (1,113) 5 the contribution of the simulator to understanding the problem 5,067 (1,484) 5,931 (1,132) 5,833 (1,085) 5,586 (0,867) 6 evaluation of the problem solving time 5,167 (1,683) 5,931 (1,307) 5,100 (1,710) 5,138 (2,031) 7 the motivation for solving the problem 4,733 (1,530) 4,966 (1,149) 5,100 (1,494) 4,345 (1,471) 8 the benefit of participation in the experiment in the course 5,833 (1,020) 6,034 (0,981) 6,133 (1,010) 5,483 (1,089) 9 the organization of the experiment 6,400 (0,894) 6,483 (0,949) 6,333 (0,661) 6,310 (0,712) 10 the contribution of the simulator to the quality of decision 5,900 (1,269) 6,276 *0,797 6,333 (0,884) 5,793 (0,940) 245 Figure 9: A learning model of decision group under various decision-making conditions represents the decision-making supported by a simulation model and corresponds to experimental conditions a1 and a3. The higher the gap between the goal and the performance, the higher the frequency of simulation runs. The search for the optimal parameter values is based on trial and error. The more simulation runs that the decision maker performs, the more he or she learns (on an individual level) and the smaller is the gap between performance Organizacija, Volume 40 Research papers Number 6, November-December 2007 back seeking behaviour and more commitment to problem solving. In this case, facilitation serves as motivation and orientation towards the goal. The subjects of group a2 had to make their decisions three times during the experiment before they submitted their final decisions, while their colleagues in group a4 were left to their own pace and had to make their final decision at the end of the experiment. 3.5 Opinion Questionnaire Analysis The participant’s opinions on their involvement in the experiment were solicited by questionnaires. Participants filled in the questionnaires via a web application. Questions were posed in the form of a statement and agreement with that statement was measured on a 7-point Likert-type scale, where 1 represents very little agreement, 4 a neutral opinion and 7 perfect agreement. The average value of the answers to the statements in the opinion questionnaire and its standard deviation are shown in Table 1. From Table 1, it is evident that the participants expressed high agreement to most of the statements. In fact, only Statement 7 regarding the motivation for participating in the experiment, was evaluated a bit lower. In other words, it was closer to the neutral point, but not negative. We performed an ANOVA test to explore the differences in opinions among the four experimental conditions. The ANOVA test showed a high level of agreement between groups as well. The groups' opinions only differ significantly for two questions: 4) the simplicity of use of the simulator (F=3.067, p=.031), and 5) the contribution of simulator to understanding the problem (F=3.274, p=.024), which can both be explained by the different experimental conditions requiring a slightly different user interface and thus different levels of man-computer interaction. From the opinion questionnaires, we can make some general observations: 1. 99% of the participants agreed on the general quality of the experiment. 2. 83% of all the participants agreed that the decision problem was correctly presented. 3. 68% of all the participants agreed that they understood the decision problem presented. 4. 93% of all the participants agreed that the simulator was easy to use. 5. 84% of all the participants agreed that the use of simulator contributed to their understanding of the problem. 6. 70% of all the participants agreed that there was enough time for decision making. 7. 63% of all the participants agreed that they were motivated to solve the problem. 8. 88% of all the participants agreed that they benefited from participating in the experiment. 9. 97% of all the participants agreed that the experiment was well organized. 246 10. 92% of all the participants agreed that the use of the simulator contributed to better decision-making. These are the across-group averages and represent the overall agreement to the statements. We can say that, in general, the students were satisfied with the experiment as a method of teaching and the use of the simulation in decision support. 4 Conclusion In prior experiments (Škraba et al, 2003; Škraba et al, 2007), we have already proved the positive impact on the decision-making process of individual information feedback assured by a simulation model and group feedback information. However, the results suggested that differences in the frequency of the simulation runs in the first eight minutes of the experiment, where two simulation groups had same conditions, might be caused by the phenomena of group belonging. Hence, a new experiment was introduced - a pseudo Solomon experimental design - and the following experimental conditions were formulated: a1 - an individual decision-making process supported by a simulation model with pre-testing after the 8th, 16th, 24th and 30th min, a2 - a decision-making process supported by a simulation model and group information feedback with pre-testing after 8th, 16th, 24th and 30th minute, a3 - an individual decision-making process supported by a simulation model but without a pre-test (testing only after the 30th min) and a4 - the decision-making process supported by a simulation model and continuous group information feedback but without the pre-test (testing only after the 30th min). The hypothesis that the application of individual information feedback assured by the simulation model positively influences the learning process of an individual decision-maker was confirmed by Friedman’s ANOVA at p=.000. The hypothesis that additional applications of group feedback information contributes to a higher convergence and group unity was confirmed by the Mann-Whitney U-test at p=.006. On the basis of analysis, we can conclude that the group information feedback introduced into the decision-making process contributes to increased convergence of the decision group and helps achieve faster decision problem solving (eight minutes). The results of the analysis have confirmed that there is an interaction of treatment (group information feedback) and testing effects (facilitation) that affects the dynamics of the decision-making process (the frequency of simulation runs at p=.046). Therefore, group feedback and the facilitator are extremely important during complex problem solving. A causal loop diagram model of the learning taking place during the decision-making process by means of simulation model was developed. The results of an opinion analysis show that management students thought that the application of the simulation model does contribute to increased understanding of the problem, faster solution finding and more confidence on the part of the participants. Organizacija, Volume 40 Research papers Number 6, November-December 2007 All the participants agreed that a clear presentation of the problem motivates the participants to find the solution. 5 Acknowledgements This research was funded by the Ministry of Education, Science and Sport of the Republic of Slovenia; Program No. P5-0018. 6 References Checkland PB, Haynes MG. (1994). Varietes of systems thinking: the case of soft systems methodology. System dynamics review, 10(2-3): 189-198. Chun KJ & Park HK. (1998). Examining the conflicting results of GDSS research. Information & Management, 33: 313-325. Forrester JW. (1961). Industrial Dynamics. MIT Press. Cambridge. MA. Gopinath C & Sawyer JE. (1999). Exploring the learning from an enterprise simulation. Journal of Management Development, 18(5): 477-489. Hale R & Whitman P. (1997). Practical problem solving & decision making, Kogan Page. Hines J. (1996). Molecules of Structure, Version 1.1, Building Blocks for System Dynamics Models. LeapTec and Ventana Systems, Inc., USA. Howie E Sy S, Ford L & Vicente KJ. (2000). Human-computer interface design can reduce misperceptions of feedback. System Dynamics Review, 16:151-171. Jennings D. (2002). Strategic management: an evaluation of the use of three learning methods. Journal of Management Development, 21(9): 655-665. Kahneman D & Tversky A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47: 263-291. Kljaji} Borštnar, M. (2006), Relevance of Group Feedback Information to Learning in Decision - Making Process Supported by a Simulation Model, Univerza v Mariboru, Fakulteta za organizacijske vede, Kranj, doctoral thesis, unpublished. Kwok RCW & Khalifa M. (1998). Effect of GSS on knowledge acquisition. Information & Management, 34: 307-315. Langley PA, Morecroft JDW. (2004). Performance and learning in a simulation of oil industry dynamics. European Journal of Operational Research 155:715-732. Lizeo E. (2005). A dynamic model of group learning and effectiveness. Conference proceedings, The 23 International Conference of the System Dynamics Society, Editors: J. D. Sterman, N. P. Repenning, R. S. Langer, J. I. Rowe & J. M. Yanni, July 17-21, 2005 Boston. Richardson GP & Andersen DF (1995). Teamwork in group model building. System Dynamics Review, 11:113-137. Rosen R. (1985). Anticipatory Systems. Pergamon Press. Senge PM. (1990). The Fifth Discipline: The Art and Practice of The Learning Organization. Doubleday/Currency. New York. Simon H. (1997). Models of Bounded Rationality: Empirically grounded Economic Reason. Vol. 3. MIT. Škraba A, Kljaji} M & Leskovar R. 2003. Group exploration of system dynamics models - Is there a place for a feedback loop in the decision process?. System Dynamics Review. 19: 243-263. Škraba, A., Kljaji}, M. & Kljaji} Borštnar, M., (2007).The role of information feedback in the management group decisionmaking process applying system dynamicss models, Group Decision and Negotiation, 16:77-95. Sterman JD. (1994). Learning in and about complex systems. System Dynamics Review, 10: 291-330. Sterman JD. (2000). Business Dynamics: Systems Thinking and Modeling for a Complex World. Irwin/McGraw-Hill: Boston MA. Vennix JAM. (1996). Group model building: facilitating team learning using system dynamics. Wiley: Chichester. Warren K & Langley P. (1999). The effective communication of system dynamics to improve insight and learning in management education. Journal of Operational Research Society, 50: 396–404. Mirjana Kljajič Borštnar graduated in 1998 from the University of Maribor, Faculty of Organizational Sciences. Until 2001 she worked at the Small Business Development Centre. Since 2002 she is employed at the Faculty of Organizational Sciences where she obtained her masters degree and PhD. Her research field covers decision support and knowledge based systems. In 2003 she and her colleagues received the best paper award at the CASYS’03 international conference in Belgium. Andrej Škraba obtained his Ph.D. in the field of Organizational Sciences from the University of Maribor. He works as a teaching assistant and researcher at the Faculty of Organizational Sciences in Cybernetics and in the Decision Support Laboratory. His research interests cover system dynamics, discrete event simulation, agent simulation and decision processes. Vladislav Rajkovič is a professor of information systems and head of the Laboratory for Decision Processes and Knowledge-based Systems (http://lopes1.fov.uni-mb.si) at the Faculty of Organisational Sciences, the University of Maribor, Slovenia. He also works with the Department of Intelligent Systems at the Jo`ef Stefan Institute in Ljubljana. His research interests focus on information systems and artificial intelligence methods for supporting decision and educational processes. For several years he has been involved in the informatisation of Slovenian schools. Miroljub Kljajič graduated from the Faculty of Electronics, the University of Ljubljana in 1970. He received his MS in 1972 and his PhD in 1974 at the same university. In 1970 he was employed at the Jozef Stefan Institute, the Department for Biocybernetics and Robotics. Since 1976 he works at the Faculty of Organizational Sciences, the University of Maribor as Professor of System Theory, Cybernetics and Modelling and Simulation. He has been principal investigator in many national and international projects in the area of modelling and simulation. As author and co-author, he has published over two hundred scientific articles. 247