Allen E., Fenton A., Parry K.: COMPUTERISED GYMNASTICS JUDGING SCORING… Vol. 13 Issue 3: 357 - 370 Science of Gymnastics Journal 357 Science of Gymnastics Journal COMPUTERISED GYMNASTICS JUDGING SCORING SYSTEM IMPLEMENTATION – AN EXPLORATION OF STAKEHOLDERS’ PERCEPTIONS Elizabeth Allen 1 , Alex Fenton 1 , Keith Parry 2 1 University of Salford, United Kingdom 2 University of Bournemouth, United Kingdom Original article DOI:10.52165/sgj.13.3.357-370 Abstract Gymnastics is one of the original Olympic sports, subjectively judged by humans. Judging errors and bias can occur, resulting in medals being incorrectly awarded. The International Gymnastics Federation (FIG) with Fujitsu are introducing a computerised gymnastics judging support system (CGJSS), a technology aimed to enhance fairness and accuracy but there is very little literature evaluating this technology and perceptions. This project aimed to explore stakeholders’ reactions at this critical time. Therefore, interviews were conducted with coaches, judges, media, former and current international gymnasts. The findings concurred with the literature review of judging problems with the current system, including bias and subjectivity. New findings show, among other things, that gymnasts’ scores can differ depending on which round they compete in. The findings also suggest that the CGJSS would be a great innovation for gymnastics to improve credibility by removing bias and helping to make the sport more objective. However, the majority of the participants believed it could not judge the artistry element of the sport. Close monitoring of the effectiveness of the CGJSS is therefore required to identify enhancement and to ensure the investment produces fairer, more reliable and credible results. Successful implementation of the CGJSS could also allow it to be introduced into other subjectively judged sports. Keywords: gymnastics, judging, bias, technology, sport. INTRODUCTION Gymnastics has been a part of the modern Olympic programme since its introduction in 1896 (Leskosek et al., 2012) and has always been scored by qualified judges. The standard composition of a typical judging panel for artistic gymnastics is seven judges: five evaluating the execution of the routine (E- Score) (Mercier & Heiniger, 2017) and two evaluating the difficulty of the routines (D-score) (Leskosek et al., 2012). This is a significant change compared to only one judge when the sport began (Leskosek et al., 2010). Judges are marking in accordance with the Code of Points that, since 1949, have been put in place by the Fédération Internationale de Gymnastique (FIG) (Atikovic et al., 2011). Many processes have been put in place to ensure objectivity of subjectively judged sports, such as gymnastics, but unfortunately injustices still occur. For example, at the Athens 2004 Olympic Games, Paul Hamm was awarded the gold medal after the silver medallist Yang Tae- young was incorrectly deducted marks Allen E., Fenton A., Parry K.: COMPUTERISED GYMNASTICS JUDGING SCORING… Vol. 13 Issue 3: 357 - 370 Science of Gymnastics Journal 358 Science of Gymnastics Journal (Kelso, 2004). Alexi Nemov found himself in the middle of a controversy during the high-bar final at the same Olympic Games. The competition came to a complete standstill for 10 minutes when the audience booed his mark. After much deliberation, the score was increased, promoting Nemov to the third position (Reuters, 2004). In response to the problems in the Athens 2004 Olympic Games, a new scoring system was put in place in 2006 with an open-ended scoring system for the D score (Turner, 2014). Nevertheless, Green and Allen (1984, p. 47) suggest that “where there is a judged sport, there is always controversy”. Figure 1. How the gymnast’s score is calculated. Controversy can arise for many reasons including the order in which the gymnasts perform, corruption and the relationship between the gymnast and the judge, or different judges’ interpretations of the same information (Plessner, 1999). All sports must have integrity and credibility in ensuring results are fair; the errors and controversies that occurred during the Athens 2004 Olympic Games mean that people will question the sport and its integrity (Kelso, 2004). Also, competitions can be exceedingly long. A former president of the FIG commented: “A judge must work for eight hours per day – does that allow the mental capacity to remain coherent? It’s not possible to maintain a coherent mind of criteria. Only the computer does” (Logothetis, 2017, para. 5). Perederji (2013) & Leskosek et al. (2010) both found judge fatigue had an effect on the score produced for the gymnast. Gymnasts and coaches can appeal their score but only the difficulty score and not the execution score. The inquiry process was only introduced in 2004, following the judging problems during the Olympic Games (Zaccardi, 2012). During the men’s team final at the London 2012 Olympic Games, an inquiry was put in for Japan for their last routine which created a delay to the official announcement of the results. The inquiry was upheld: as a result, Ukraine was pushed out of the medals and Japan moved to the second place, having been fourth prior to the inquiry (Zaccardi, 2012). There has been considerable research conducted by various authors, including Ste-Marie, Leskosek et al. and Ansorge, to understand the effectiveness and quality of judging in gymnastics dating back to the 1970s (Bucar et al., 2012). Research, including the studies by Ansorge and Scheer (1988) and Leskosek et al. (2012), indicates that there are problems with the judging system in gymnastics. These problems can include bias that can take many different forms, including a judge favouring a gymnast from their own nation as indicated by Ansorge & Scheer (1988), Ste-Marie (1996) & Popovic (2000). Studies by Ste-Marie & Lee (1991) & Ste-Marie and Valiquette (1996) highlighted bias occurring due to previous exposure of the gymnast and Scheer &Ansorge (1975), Ansorge et al. (1978) & Scheer and Ansorge (1979) found that bias could occur depending on the place a gymnast has within his or her team. The final bias found in research by Leskosek et al. (2010) & Leskosek et al. (2012) is bias that expresses itself in over- or under- scoring gymnasts in competitions. For instance, stress can have an impact on the score, as concluded by Duda et al. (1996). Similarly, a number of studies, including those by Dallas & Kirialanis (2010), Bard et al. (1980), Ste-Marie (1999) & Ste- Marie (2000), indicate that the extent of Allen E., Fenton A., Parry K.: COMPUTERISED GYMNASTICS JUDGING SCORING… Vol. 13 Issue 3: 357 - 370 Science of Gymnastics Journal 359 Science of Gymnastics Journal experience a judge has, can impact the score that a gymnast receives. To provide a solution to the judging problems, the FIG and Fujitsu began a collaboration in 2017 with the aim of setting up a real-time judging support system that is fair and accurate (IT News Online, 2019). The computerised gymnastics judging support system (CGJSS) was initially implemented during the 2019 World Championships in Germany and is planned for the Tokyo Olympic Games (Fujitsu, 2018). The new system utilises artificial intelligence in order to support the judges on the vault during these competitions. Gymnastics skill level is ever increasing, and it can be very difficult for the naked eye to keep up with the number and complexity of manoeuvres that the gymnasts are performing. The new system collects information using 3D sensors and compares the information to the FIG standards to provide a score. “It offers a three-dimensional digital view of the performed elements, with a variety of selections for angle measurements, all of which helps to deliver a fair and accurate judgement” (Fujitsu Sys-Con Media, 2019, p a r a 7 ) . T h e F I G p r e s i d e n t M o r i n a r i Watanabe says, “Once gymnasts see it (CGJSS) they will like the technology and that is a great thing” (Logothetis, 2017). There is very little literature about the CGJSS in the public domain currently due t o t h e f a c t i t i s s o n e w t o t h e m a r k e t . Watanabe is supporting the implementation of the CGJSS and mentions that it is for gymnasts that they are making this change to ensure all their hard work and dedication is rewarded fairly and not misjudged by human error. He suggests it is “a big step towards the future” (Fujitsu, 2019, para. 6). Although the rules as set out by the Code of Points are objectively specified, this does not prevent intentional or unintentional human error (Leskosek et al., 2010). Various studies have been conducted over the years to reinforce this. It is clear that there are numerous errors and biases present in gymnastics judging, whether conscious or unconscious, affecting gymnasts’ results and medals awarded. This can potentially have a negative effect, including on bonuses and rewards. The literature confirms that further intervention is needed to reduce human error and bias in gymnastics and make it more objective to provide more accurate results. This research therefore fills the gap in understanding stakeholders’ perceptions of this new scoring technology. METHODS Our study aims to create further understanding about this new intervention from the perspective of stakeholders by asking the question: does gymnastics need the introduction of the CGJSS? In doing so, the following interview questions were posed in Table 1. Semi-structured interviews, which create structure but also flexibility to gain new insights (Wilson, 2014), with nine key stakeholders were conducted during this project to gain insight into the research questions. The interviews were conducted face-to-face or via Skype. Skype interviews were used when it was impossible to conduct face-to-face interviews due to different time zones and geographical barriers. Participants included judges, coaches, gymnasts and the media. Table 2 indicates the roles of the participants. Roles were anonymised in line with the ethical approval process at the University of Salford. A sample size of nine was chosen following Mears’ (2009) suggestion that depth rather than breadth of information is the goal, that is, to use between 6 and 9 participants. Once the participants start highlighting the same points, the study has reached saturation point. Fusch and Ness (2015, p. 1408) proposed that saturation is met when there is “enough information to replicate the study”. All information Allen E., Fenton A., Parry K.: COMPUTERISED GYMNASTICS JUDGING SCORING… Vol. 13 Issue 3: 357 - 370 Science of Gymnastics Journal 360 Science of Gymnastics Journal collected remained anonymous and collected data kept confidential to ensure that participants’ individual identities were protected and could not be identified publicly (Polit & Beck, 2006). Participants’ responses were transcribed verbatim and the responses for each question were then labeled and ordered into categories and sub-categories to create meanings (See Table 3). A deductive thematic analysis of all codes was then conducted to generate overall themes arising from the interviews (Braun and Clarke, 2006). Two members of the research team independently analysed the data and inductively co-created initial codes that were then discussed and developed into themes in order to ensure consistency. There were no disagreements; however, the process of coding was checked and agreed upon by all authors. Following data reduction, three main themes were identified, described and refined as: (1) bias and subjectivity in judging, (2) benefits to gymnastics, and (3) negative impact on gymnastics. All answers were transcribed. Table 3 presents the grouping of these answers around problems which are then discussed in the results. Table 1 Questions used during the semi-structured interviews. 1 Why is the FIG looking at the CGJSS? 2 What concerns, if any, do you have with the current judging system in gymnastics? 3 How could the CGJSS move gymnastics forwards or backwards? 4 How could the CGJSS change the way gymnasts train or coaches coach? 5 What concerns, if any, are there with the CGJSS? 6 Why do you believe or not that the CGJSS will be good value for money in either short- or long-term? Table 2 Participants in the study and their Participant (P) number. Participant ID Participants’ current and previous roles in gymnastics P1 Team leader of Youth Olympics P2 Former international gymnast, coach and commentator P3 Former international gymnast P4 Press relations manager P5 Coach to international gymnasts and former Olympians P6 International gymnast P7 International gymnast P8 Coach to international gymnasts and FIG judge P9 FIG judge, coach and former gymnast Allen E., Fenton A., Parry K.: COMPUTERISED GYMNASTICS JUDGING SCORING… Vol. 13 Issue 3: 357 - 370 Science of Gymnastics Journal 361 Science of Gymnastics Journal Table 3 A summary of the findings to the sub-questions from the primary data collection. Question Answer from the primary research Why are FIG looking at the CGJSS?  To make the sport more understandable  Removing the subjectivity from the sport  To make the judging more valid and reliable  To remove judging bias  To create a level playing field  The skills within the sport are evolving so quickly, it is becoming increasingly more difficult to see things with the naked eye What problems if any do you see with the current judging system in gymnastics?  Bias judging of performing in later rounds of the competition  Unconscious bias of having seen a gymnast perform previously  International bias of judges from the same nationality  Bias due to federations who are friendly with each other and gymnasts/federations with a well-known name  Bias due to the place within the team  Fatigue due to long days of competition  The judging is subjective  Skill level is rapidly increasing How could the CGJSS move gymnastics forwards or backwards ?  Positive - Make the sport more marketable  Positive - Make the results more reliable and valid  Positive – The sport becomes more objective  Positive – The sport would evolve with time  Negative – It can’t judge all aspects of the sport including artistry  Negative – Potential loss of judges  Negative – Potential loss of traditions of the sport How could the CGJSS system change the way gymnasts train or coaches coach?  Most participants did not feel it would change the way gymnasts train or coaches coach  Positive - The system could be used as an analytic tool  Negative – It would have to be readily available to all nations if it was available outside of competitions to ensure there is no disparity between the rich and the poor countries What concerns if any are there with the CGJSS?  Gymnasts could lose their flair and become robots  The accuracy of the system  Stakeholders adopting the system  The CGJSS is only able to judge certain aspects of the sport  Which variation of the skill the gymnasts will have their skill compared to  The CGJSS crashing during competitions  Potential loss of traditions of the sport  Needs to be readily available to all countries if it is available outside of competitions  Potential loss of judges Do you believe CGJSS will be good value for money in the short term or long term?  Most participants think it is worth the investment both short- term and long-term to ensure a level playing field is created, and that the sport becomes more accurate so that the most deserving gymnasts stand on the podium. Allen E., Fenton A., Parry K.: COMPUTERISED GYMNASTICS JUDGING SCORING… Vol. 13 Issue 3: 357 - 370 Science of Gymnastics Journal 362 Science of Gymnastics Journal RESULTS Bias and subjectivity in judging. Concerns over bias and subjectivity in judging were one of the issues that surfaced in the interviews. P4 suggested: “It’s [bias] something that gymnasts just accept; it’s part of the sport and it shouldn’t really be like that.” Different types of bias were highlighted including gymnasts performing in later rounds; unconscious bias from previous exposure, and international bias of judges judging gymnasts of the same nationality. Biased judging of gymnasts of the same nationality as the judges or federations that are friendly with each other were also pointed out. P1 commented that “bias is a big one as judges tend to know the athletes and coaches of the federations they are judging, so personal relationships develop”. P3 backed this up: “It does depend a lot on what federations are friendly with what other federations”. Bias from previous exposure, whether via competitions, training or online videos, came up as another potential problem. P3 felt this can influence the score: “Unconscious bias from what they may have seen previously; they may make deductions automatically without actually seeing them”. The two judges in our group disagreed with this and felt that seeing a gymnast previously perform was only helping them to prepare for a competition. P8 said: “I have always said it’s my job to warm up in the way that gymnasts do, and that’s what I am doing; and of course someone could say that I am doing something different but I can say, hand on heart, that when I go in, I am prepared not to miss anything”. Gymnasts performing in later rounds were mentioned in a number of interviews. P5 commented: “Depends on what round you are judging. It’s not necessarily intentional, but scores do tend to rise a little bit”. P4 and P6 also felt this and commented, respectively: “I think gymnasts feel that in qualifying there might be a difference between whether you are on in the afternoon or in the morning,” and “People always say it’s better to perform later in the day”. P2 and P3 mentioned that from their own previous experience the place in which a gymnast performs in a team can be a factor for team t a c t i c s . T h e u s e o f t h e C G J S S , “ w o u l d make that hypothesis irrelevant”, said P3 since the computer would not be able to identify who it is judging. Long days and fatigue were mentioned as factors that can have an impact on this higher scoring later in the day. P3 mentions: “Actually judges, when they are doing a World Championships, they are judging from the crack of dawn until late at night with minimal breaks in between and when that sort of tiredness level kicks in, is there going to be human error? Absolutely”. P8 commented: “At world championships, you start at 6 in the morning when the day starts and then you have to give consideration and concentration all the way through to 10 o’clock at night and do that for 2 days in a row; no matter how hard we try…there has got to be an effect on some of your decision making.” P9 reinforced this by saying: “We will have a degree of accuracy when we first arrive … to ask us to be just as accurate 14 hours later… I think that might be asking a bit much of us … you are not going to be 100% accurate from the start of a competition until its end.” The time that the judges are expected to concentrate for over two days is considered to be too long in some cases and the participants felt that it could have a negative effect on the consistency of scoring. Creating a level playing field and the need for fairness were seen as an important consideration. P5 commented on the need for such a scoring system “so that it is totally fair”, and P4 suggested “to take away the mistakes and the bias and creating a sport that is very exact”. Allen E., Fenton A., Parry K.: COMPUTERISED GYMNASTICS JUDGING SCORING… Vol. 13 Issue 3: 357 - 370 Science of Gymnastics Journal 363 Science of Gymnastics Journal Building on this, P3, P4 and P6 all suggested that the CGJSS would create more valid and reliable results. P3 said “that it will make it as objective as possible and it’s a valid and reliable measure…. that will make it a lot easier to justify scores to athletes, to coaches”. P6 suggested the current scoring system “seems to sort of fluctuate quite a lot…it’s hard to compare scores from one competition to another”. P8 commented: “The emphasis is on what can we do to make the sport as objective as possible,” and P9 states: “There is a lot that is asked o f u s … t h e m o r e t h a t w e c a n u s e A I (Artificial Intelligence) to evaluate the objective part, the more accurate scores and deductions the athletes will be awarded.” The participants want to ensure that the scores are fully justified and reliable. P1, P2, P4, P8 and P9 all mentioned that gymnastics is progressing so fast that it is becoming much more difficult to see the gymnasts’ skills with the naked eye, or to accurately observe the mistakes, or even note what skill is being performed. P2 mentioned the 2019 European Championships when it was unclear whether a gymnast had gone out of bounds on the floor. This impacted the decision on who became the European Champion: “We were also at the last Europeans …we thought she might have just bounced her heels out; and we know as gymnasts, performers, you can’t land back in full straight on your toes, so the heels must have gone down, but it happened so quickly.” P2 commented: “I think what’s really cool about this [the CGJSS] is to protect the medal winners to make sure you have the right people on the podium”. P8 mentioned how the skills are advancing and the judges need to see the skill to know exactly what was performed. “10 years ago people weren’t doing quadruple twists on the floor… If I know someone is performing a quadruple twist, I bet every MAG judge is in the hall to make sure they see the quad.” Participants were of the opinion that the CGJSS would really help i d e n t i f y t h e h a r d e r s k i l l s a n d m a k e s u r e that mistakes are not missed, ensuring the results are fair. Nearly all of the participants felt they had seen or experienced an injustice during competition. P4 mentioned “it happens all the time”. A press relations manager who sees the gymnasts immediately after their performance, confirmed that they often say that they were not given a fair score. He admitted that “part of that is emotion because they have just finished competing but there is also frustration”. The results suggest that there are still problems with the current judging system, including bias, subjectivity and fatigue, which can result in medals being awarded to the wrong gymnasts; hence, the FIG are looking at implementing the CGJSS. Additionally, expert judges P8 and P9 observed that the skill levels have improved very much, whereas they are expected to recognise and evaluate skills exactly the same as 10 years ago. As P8 mentioned: “10 years ago people weren’t doing quadruple twists on floor”. P9 stated: “We are asked to make very accurate evaluations in terms of angles of completion within 1 degree … we are asked to look at not only the execution at an angle but also the body itself and the shapes of the body.” They mentioned that the number of elements that the judges are required to judge is becoming too high, including the level of skill and the degree of accuracy of execution. Benefits to gymnastics. Throughout the interviews, a number of reasons why the FIG would be considering the CGJSS became apparent. One of the main reasons appeared to be to make the sport more understandable. P2, P3, P4 and P8 all mentioned this as a reason. As noted by P2: “So that it becomes a bit easier for the Allen E., Fenton A., Parry K.: COMPUTERISED GYMNASTICS JUDGING SCORING… Vol. 13 Issue 3: 357 - 370 Science of Gymnastics Journal 364 Science of Gymnastics Journal general public to understand; I assume the driver behind this is to have more people viewing gymnastics”. P4 added: “I think that anything that can help demystify gymnastics is helpful, because it is an exceptionally complicated sport. We all know it is a complicated sport to explain in terms of the scoring process; the way things are in the modern world, the consumer - the audience - want to know why things happen.” The participants agreed that making the scoring easier to understand would make the audience understand the results better and as a result more people would watch it. P2 and P3 felt that the sport would become more marketable to sponsors and easier to understand for the general public. P3 stated that creating World Records would mean that “sponsors would want to get on board with it and it would also help athletes get personal sponsors”. As a result of being more marketable, P2 and P3 suggested it would create bigger audiences. P8 believes that “it may help educate people and if they use this to improve their knowledge, then this is a good thing.” With a system which would allow competitions and results to be comparable, the sport would be able to have World Records which is currently not possible. P3 and P4 suggested it would create a more valid and reliable sport, the audience would believe the scores and the judges would be able to justify their scores and remove the subjectivity from the sport. P4 thought: “It is to increase the accuracy of the judging and take away mistakes” and P3 asked: “Is it reliable? Is it valid? Is it objective? These can only be positives”. P4 suggests it would make the sport better: “If it makes the judging better, then that’s a benefit; if it gives gymnast and coaches more confidence in the scores, then that’s a benefit, … and if it means that gymnastics is more accessible to more people then it is definitely worth it”. It seemed important to a couple of participants that the stakeholders believed in the new system. P1 stated: “Obviously, the athletes have to adopt the system and believe that it is accurate”. P3 has concerns about hesitancy to try the CGJSS: “For the more traditional older coaches, I can see a few challenges there.” Negative impacts on gymnastics. There was some negativity associated with the CGJSS. The participants were concerned that the sport would lose judges if the system became fully integrated. Interviewees expressed doubts over whether it could judge all aspects of gymnastics. P5 challenged: “How does a computer judge artistry?” and “They [judges] put a lot of work and hours in this and it’s not fair that they don’t get their Olympic Games”. Moreover, over half of the participants were worried that the CGJSS would only be able to judge certain aspects of gymnastics and not the artistry element. P2 stated that he didn’t “think a computer system is good enough yet to interpret performances as well as a judge” and P5 commented that “it’s ok for the elements, but when it comes to artistry, how does a computer judge the artistry?” P6 commented: “I don’t know about the artistry, I don’t know how the system would do that on the floor and the beam”. P8 stated: “The AI side of things is purely theoretical, it goes by the book. There is nothing else but mathematics - but that’s not our sport,” and P9 feels that AI is a great evaluation tool but does not have the capacity to evaluate the emotional side of gymnastics: “I am not sure if any type of AI is going to be able to recognise that”. As gymnastics is a traditional sport, the participants were worried that the tradition would be lost. Over half of the participants think gymnasts would lose the individual flair as they would work towards what the CGJSS wants rather than their own interpretation of the skill. P1 suggests: “It might happen that you train Allen E., Fenton A., Parry K.: COMPUTERISED GYMNASTICS JUDGING SCORING… Vol. 13 Issue 3: 357 - 370 Science of Gymnastics Journal 365 Science of Gymnastics Journal the athlete to execute a move in a certain way, because the computer picks it up better”. P2 said: “As a gymnast and a coach I can say that the way you learn skills and the way you perform them can vary a lot. I think it is like someone’s fingerprint, it is unique to that person and that’s why gymnastics is quite a unique sport”. P4 suggests: “Are you taking out creativity, artistry, individuality, and are you then trying to create robots that are doing everything perfectly?” One participant was concerned that AI may take over the sport with P9 commenting: “Right now, we will use AI to evaluate things, but what happens in the future if technology advances so much …The sport is about trying to evoke a reaction with judges and audience and I don’t want to see that ever get lost for the sake of coming up with something that is technically perfectly accurate.” P 3 a n d P 7 f e e l t h a t g y m n a s t s a n d coaches would be able to use the GGJSS to work out risks versus execution. P3 indicated: “The gymnast can check if the skill is getting credited. It is giving you an insight into what score you can realistically expect to get if you use it and then you can use this to your advantage as it becomes a bit easier to weigh up the risk-to-reward factor.” P7 felt “if you are debating whether to put a new skill in or an extra turn in a spin…it is going to tell you if you are going to get it awarded or not”. P9 commented that “if it will be practical for people to have in their gyms, then it can be used in a really positive way”. Some participants thought the sport will evolve as a result of the use of technology. P8 thinks that “evolving is a good thing, but we need to keep an eye on the bigger picture the whole time.” P9 has seen change throughout his time in the sport and stated that “change always causes different feelings in different people. There will always be questions and sometimes controversy until people get used to the idea.” Although participants did not think that GGJSS would change the way that gymnasts train or coaches coach, there were concerns that if the GGJSS were available outside of competitions it would have to be accessible to everybody to ensure equality. P4 stated: “It doesn’t matter whether you are a rich country or a poor country or whatever, it has to be a really level playing field … otherwise it will just add to the disparity.” P9 is worried: “To be current, it costs lots of money. How are we going to ensure that everyone from the beginner competitor through to the top athlete will have access to all these systems?” Table 4 Concerns highlighted by the participants during the interviews. Concern raised Participant that mentioned the concern Loses the flair of individual gymnasts and their personalities and creates robots P1, P2, P4, P8, P9 The accuracy of the system P1, P2 Stakeholders adopting the new technology P1, P3, P4 Computers are only able to judge certain aspects of the sport P2, P4, P5, P6, P8, P9 Which variation of the skill should it be compared to, potential bias P2 Software crashing at competitions P2, P6, P7, P8, P9 Lose the tradition of the sport P2, P3, P5 If available outside of competitions, it needs to be available to everyone P3, P4 Lose judges P3, P4, P5, P8, P9 Allen E., Fenton A., Parry K.: COMPUTERISED GYMNASTICS JUDGING SCORING… Vol. 13 Issue 3: 357 - 370 Science of Gymnastics Journal 366 Science of Gymnastics Journal Finally, some participants raised concerns about technological aspects and the system crashing, “I know I wouldn’t be happy if it crashed at the beginning of my floor routine,” stated P7. P9 felt the system would have to have “a lot of safeguards, otherwise it would turn out to be a nightmare. If you have got rid of all of your human judges and rely on AI, when it goes down, so does the competition”. In addition to the these opinions, other respondents had similar concerns. Table 4 summarises them. DISCUSSION Our findings support those of scholars, such as Bucar et al. (2012) and Leskosek et al. (2012), as we identified that there are a number of areas where potential bias can be an issue in gymnastics scoring. However, whereas biases due to the position within the team and judges favouring gymnasts of the same nationality have been previously reported, we can add bias due to familiarity as another area in judging. Bias and subjectivity are seen as an ‘accepted’ part of the sport but one that causes consternation for gymnasts. It was primarily for this reason that the introduction of the CGJSS has been welcomed. However, participants were worried that its introduction would result in a reduction of artistic flair that they saw as a key element of gymnastics. It is worth noting that with increased professionalisation and commercialisation of elite sports, success has become more important, and this corresponds with a lower freedom of expression and joy that is often part of sport participation at grassroots and youth levels (Cashmore, 2010). It would not be surprising to see gymnastics evolve in this direction, since the introduction of technology has seen a number of sports change over time (Collins & Evans, 2011; Vera-Rivera et al, 2019). This change may be needed for two reasons. Firstly, the judges among our interviewees noted that as the skill level of gymnasts has increased, it has become harder for them to keep track of complex routines; therefore, technology was needed to ensure accurate scoring. Secondly, there was a view that less subjectivity in scoring would make the sport more appealing for sponsors and easier for the wider public to understand and engage with. These commercial drivers are important considerations if the sport is to continue to grow in an increasingly competitive sporting landscape (Clarkson et al., 2020). Interestingly, fairness was mentioned by our interviewees who were concerned that the technology may only be available to gymnasts from more affluent nations. Indeed, one of the great myths of sport is that it is equitable. Success in Summer Olympic sports has been shown to be dependent on the GDP per capita and the population size (Trivedi & Zimmer, 2014). Therefore, further research would be required to evaluate whether the introduction of this technology further widens the gap between the more and the less affluent nations. In summary, there is a common theme that the participants are positive and excited about the move forward to use the CJGSS technology in gymnastics, but they have some reservations about it. The participants feel that the technology would be helpful if it can help remove subjectivity and bias and result in fairer and more reliable scores to ensure that the most deserving gymnasts stand on the podium. They would not, however, want to see judges completely removed from the sport, especially as they are worried whether the CGJSS is capable of judging all aspects of the sport, with artistry as the main element of concern. Allen E., Fenton A., Parry K.: COMPUTERISED GYMNASTICS JUDGING SCORING… Vol. 13 Issue 3: 357 - 370 Science of Gymnastics Journal 367 Science of Gymnastics Journal CONCLUSIONS The aims of this research were to explore how stakeholders felt about the introduction of a computerised gymnastics judging scoring system (CGJSS) and why gymnastics needed a CGJSS. The use of this technology in gymnastics is currently evolving and this shows how important this research is. There is currently no published data to evaluate the investment needed to implement the CGJSS or the concerns associated with its implementation. Stakeholders’ perceptions and the literature review in this study found that CGJSS needs to be introduced into gymnastics for certain aspects of judging to help remove subjectivity and create a valid, reliable scoring system to ensure the best gymnasts on the day win the medals. The relevant literature confirms that there is a range of problems with the current judging system and the findings from this project concurred with the literature. They include biases in judging due to previous exposure to gymnasts, preferential assessment of gymnasts from their own country, and due to the time of the day gymnasts compete. Fatigue of the judges and advancement in the skills were also mentioned as factors affecting gymnasts’ scores. All these factors meant that nearly all participants felt that they had witnessed an injustice due to judging. Nearly all participants thought that by introducing the CGJSS some of subjectivity could be removed from the sport and most participants felt that the investment was worthwhile to ensure a fairer and more reliable scoring system for gymnastics competitions. An additional, novel finding from this study is that the inherent subjectivity and complexity of gymnastics were seen as a barrier to the sport becoming more popular and marketable. Our participants believe that the popularity of the sport is to some extent based on transparency and integrity. Moreover, there is a sense that for a sport to be popular it needs to be relatively simple and easy to understand – something that gymnastics and its scoring system are not. Nevertheless, over half of the participants were concerned that the CGJSS could not judge all aspects of gymnastics. Fujitsu acknowledging this limitation. However, most of the participants feel that with further development, the CGJSS could move the sport forward in the future. Throughout the literature and in our study, the CGJSS is referred to as a support system. This means that it is only to be used as an aid and not a replacement for judges. Just under half of the participants had concerns about losing judges altogether from the sport, even though this is unlikely. In other sports where technology has been introduced to remove human error in judging/officiating (such as the Video Assistant Referee in association football or the Decision Review System in cricket), the match officials remain an integral part of the game. However, we should also note that the introduction of technology has not removed human error from these sports. Future research should focus on the implementation of AI in judging of other sports. This may also include global and team sports such as football where VAR implementation still suffers from some of the issues raised in this paper. Nevertheless, AI has the potential to address issues such as bias, subjectivity and fatigue, and therefore provides rich opportunities for further research. REFERENCES Ansorge, C. J., Scheer, J. K., Laub, J., & Howard, J. (1978). Bias in Judging Women’s Gymnastics Induced by Expectations of Within-Team Order. Research Quarterly, 49(4), 399-405. doi: 10.1080/10671315.1978.10615552. Ansorge, C. J., & Scheer, J. K. (1988). International Bias Detected in judging Allen E., Fenton A., Parry K.: COMPUTERISED GYMNASTICS JUDGING SCORING… Vol. 13 Issue 3: 357 - 370 Science of Gymnastics Journal 368 Science of Gymnastics Journal Gymnastic Competition at the 1984 Olympic Games. Research Quarterly for Exercise and Sport, 59(2), 103-107. doi: 10.1080/0270367.1988.10605486. Atikovic, A., Kalinski, S. D., Bijelic, S., & Vukadinovic, N. A. (2011). Analysis Results Judging World Championships in Men’s Artistic Gymnastics in London 2009 Year. SportLogia 7(2), 95-102. doi:10.5550/sgia.110702.en.095A. Bard, C., Fleury M., Carriere, L., & Halle, M. (1980). Analysis of gymnastics Judges’ Visual Search. Research Quarterly for Exercise and Sport, 51(2), 267-273. doi: 10.1080/02701367.1980.10605195. Bucar, M., Cuk, I. Pajek, J., Karacsony, I., & Leskosek, B. (2012). Reliability and validity of judging in women’s artistic gymnastics at University Games 2009. European Journal of Sport Science, 12(3), 207-215. doi: 10.1080/17461391.2010.551416. Cashmore, E. (2010). Making sense of sports (5th ed.). New York: Routledge. Clarkson, B. G., Culvin, A., Pope, S., & Parry, K. D. (2020). Covid-19: Reflections on threat and uncertainty for the future of elite women’s football in England. Managing Sport and Leisure, 1- 12. doi:10.1080/23750472.2020.1766377. Collins, H., & Evans, R. (2011). Sport-decision aids and the “CSI-effect”: Why cricket uses Hawk-Eye well and tennis uses it badly. Public Understanding of Science, 21(8), 904-921. doi: 10.1177/0963662511407991. Dallas, G., & Kirialanis, P. (2010). Judges’ evaluation of routines in men’s artistic gymnastics. Science of Gymnastics Journal, 2(2), 49-58. Duda, J. L., Brown Borysowicz, M. A., & St. Germaine, K. (1996). Women’s Artistic Gymnastics Judges’ Sources of Stress. Retrieved 7 February, from https://usagym.org/pages/home/publication s/technique/1996/10/judge.pdf. Fujitsu. (2018). The International Gymnastics Federation to Implement Fujitsu's Judging Support System. Retrieved 22 November, 2018, from http://www.fujitsu.com/global/about/ resources/news/press-releases/2018/1120- 01.html Fujitsu. (2019). “A step towards the future” with the first official use of Fujitsu technology to support judging at the 2019 Artistic Gymnastics World Championships. Retrieved 14 November, 2019, from https://www.fujitsu.com/global/about/reso urces/news/press-releases/2019/1002- 01.html Fusch, P. I., & Ness, L. R. (2015). Are w e t h e r e y e t ? D a t a S a t u r a t i o n i n Qualitative Research. The Qualitative report, 20(9), 1408-1416. Green, L., & Allen, L. (1984). Judgement Day. Women’s sport and fitness, 6, 47-53. IT News Online. (2019). "A step towards the future" with the First Official Use of Fujitsu Technology to Support Judging at the 2019 Artistic Gymnastics World Championships. Retrieved 2 October, 2019, from http://www.itnewsonline.com/jcn/quotA- step-towards-the-futurequot-with-the-First- Official-Use-of-Fujitsu-Technology-to- Support-Judging-at-the-2019-Artistic- Gymnastics-World-Championships/2010. Kelso, P. (2004, 25 August). Uproar in the gym: Russians complain to the IOC. The Guardian. Retrieved from https://www.theguardian.com/sport/2004/a ug/25/athensolympics2004.gymnastics Leskosek, B., Cuk, I., Karacsony, I., Pajek, J., & Bucar, M. (2010). Reliability and validity of judging in Men's artistic gymnastics at the 2009 University Games. Science of gymnastics 2(1), 25-34. Leskosek, B., Cuk, I., Pajek, J., Forbes, W., & Bucar-Pajek, M. (2012). Bias of Judging In the Men’s Artistic Gymnastics at the European Championships 2011. Biology of Sport, 29(2), 107-113. doi: 10.5604/20831862.988884 Logothetis, P. (2017, 4 November). Cracking the vault: Artificial intelligence judging comes to gymnastics. The Allen E., Fenton A., Parry K.: COMPUTERISED GYMNASTICS JUDGING SCORING… Vol. 13 Issue 3: 357 - 370 Science of Gymnastics Journal 369 Science of Gymnastics Journal Guardian. Retrieved from https://www.theguardian.com/sport/blog/2 017/nov/04/ai-judges-gymnastics- olympics. Mears, C. L. (2009). Interviewing for education and Social Science Research: The Gateway approach. New York: Palgrave Macmillan. Mercier, H. , & Heiniger, S. Judging the Judges: Evaluating the Performance of International Gymnastics Judges. MIT SLOAN, Sport Analytics Conference, Hynes Convention Centre, March 3-4, 2017, (2017), 1-18. Perederji, V. V. (2013). The problem of the quality of judging in rhythmic gymnastics. Pedagogics, 17(3), 43-46. doi: 10.6084/m9.figshare.647325 Plessner, H. (1999). Expectation biases in gymnastics judging. Journal of Sport & Exercise Psychology, 21(2), 131- 144. doi: 10.1123/jsep.21.2.131 Polit, D., & Beck, C. T. (2006). Essentials of nursing research: Methods, appraisal, and utilization (6 th ed.). London: Lippincott Williams & Wilkins. Popovic, R. (2000). International Bias detected in judging rhythmic gymnastics competition at Sydney-2000 Olympic Games. Facta Universitatis (Series: Physical Education and Sport), 1(7), 1-13. Reuters. (2004). Nemov uproar overshadows Ponor show. Retrieved 20 February, 2019, from https://www.eurosport.co.uk/artistic- gymnastics/athens/2004/uproar-over- nemov-marks_sto630585/story.shtml Scheer, J. K., & Ansorge, C. J. (1975). Effects of Naturally Induced Judges’ Expectations on the Ratings of Physical Performances. Research Quarterly. American Alliance for Health, Physical Education and Recreation, 46(4), 463-470. doi: 10.1080/10671315.1975.10616704 Scheer, J. K., & Ansorge, C. J. (1979). Influence Due to Expectations of Judges: A Function of Internal-External Locus of Control. Journal of Sports Psychology, 1(1), 53-58. doi: 10.1123/jsp.1.1.53 Ste-Marie, D. M. (1996). International bias in gymnastics judging: Conscious or unconscious influences? Perceptual and Motor Skills, 85(3), 963 -975. Retrieved from https://journals.sagepub.com Ste-Marie, D. M. (1999). Expert- Novice Differences in Gymnastic Judging: An Information-processing Perspective. Applied Cognitive Psychology 13(3), 269- 281. doi: 10.1002/(SICI)1099- 0720(199906)13.3<269::AID- ACP567>3.0.CO;2-Y Ste-Marie, D. M. (2000). Expertise in Women’s Gymnastics Judging: An Observational Approach. Perceptual and Motor Skills, 90(2), 543-546. doi: 10.2466/pms.2000.90.2.543 Ste-Marie, D. M., & Lee, T. D. (1991). Prior Processing Effects on Gymnastics Judging. Journal of Experimental Psychology: Learning, Memory and Cognition, 17(1), 126-136. doi: 10.1037/0278-7393.17.1.126 Ste-Marie, D. M., & Valiquette, S. M. (1996). Enduring Memory-Influenced Biases in Gymnastic Judging. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22(6). 1498-1502. doi: 10.1037/0278-7393.22.6.1498 Sys-Con Media. (2019). "A step towards the future" with the First Official Use of Fujitsu Technology to Support Judging at the 2019 Artistic Gymnastics World Championships. Retrieved 2 October, 2019, from http://www.sys- con.com/node/4410402 Trivedi, P. K., & Zimmer, D. M. (2014). Success at the Summer Olympics: How Much Do Economic Factors Explain? Econometrics, 2(4), 169-202. doi:http://dx.doi.org/10.3390/econometrics 2040169 Turner. A (2014). FIG wants scoring overhaul. Retrieved 20 February, 2019, from http://www.intlgymnast.com/index.php?op tion=com_content&view=article&id=4174 :fig-president-wants-scoring- overhaul&catid=3:interviews&Itemid=56 Allen E., Fenton A., Parry K.: COMPUTERISED GYMNASTICS JUDGING SCORING… Vol. 13 Issue 3: 357 - 370 Science of Gymnastics Journal 370 Science of Gymnastics Journal Vera-Rivera, J. L., Ortega-Parra, A. J., & Ramírez-Ortiz, Y. A. (2019). Impact of technology on the evolution of sports training. Journal of Physics: Conference Series, 1386, 012144. doi:10.1088/1742- 6596/1386/1/012144 Wilson, C. (2014). Interview Techniques for UX Practitioners: A User- Centred Design Method. Waltham, MA: Elsevier. Zaccardi, N. (2012). Explaining the inquiry system that has wreaked havoc on London gym. Retrieved 19 February, 2019, from https://www.si.com/more- sports/2012/08/07/aly-raisman-inquiry- gymnastics Corresponding author: Alex Fenton University of Salford Maxwell Building, University of Salford, UK, M5 4WT E-mail: a.fenton@salford.ac.uk Tel and fax num: 0161 2954127 Article received: 29.3.2021 Article accepted: 23.7.2021