https://doi.org/10.55707/ds-po.v38i3-4.114 Martina Paradiž, PhD Collaborative M-Learning of Collocations Using the ColloQuiz Application Prejeto 28. 3. 2023 / Sprejeto 25. 8. 2023 Received 28. 3. 2023 / Accepted 25. 8. 2023 Znanstveni članek Scientific paper UDK 81’243:378.147:004 UDC 81’243:378.147:004 KLJUČNE BESEDE: mobilna aplikacija, mobilno KEYWORDS: mobile application, mobile-assisted podprto učenje jezika, angleški jezik kot jezik stroke, language learning, English for Specific/Academic Pur- kolokacije, kontrastivna analiza poses, collocations, contrastive analysis POVZETEK – Razširjena uporaba pametnih telefonov ABSTRACT – Students’ widespread use of smart- med študenti odpira nove možnosti digitalnega sodelo- phones has opened up new avenues for digital collabo- vanja pri pouku jezika in hkrati pospešuje raziskave na ration in the language classroom and has accelerated področju mobilnega učenja (m-learning) in MALL (mo- research in the field of mobile-assisted language learn- bile-assisted language learning). Znanje kolokacij tvori ing (MALL) or m-learning. Since mastery of colloca- ključno komponento tako jezikovne kompetence kot tudi tions is a crucial aspect of language proficiency and kakovostnega prevajanja, zato se poučevanje angleške- translation quality in academic discourse, a large part ga jezika v okviru stroke v veliki meri osredotoča na of the focus of LSP teaching is on helping students to pravilno rabo teh leksikalnih enot. Članek predstavlja master the use of these lexical units and become aware zasnovo in funkcije avtorske mobilne aplikacije Collo- of the differences between languages. The paper pre- Quiz, razvite s pomočjo prosto dostopnega orodja, ki sents a mobile application developed by the author ne zahteva programerskega znanja. Z njeno pomočjo for students to compile a bilingual database of aca- so študenti oblikovali dvojezični glosar kolokacij v demic collocations and then use it for reference and akademskem diskurzu in jo nato uporabljali za učenje revision. Based on an online questionnaire, the study in ponavljanje. Članek predstavlja rezultate raziskave reports on students’ perceptions regarding the usability mnenja študentov o uporabnosti in učinkovitosti apli- of the application and its perceived efficacy for lan- kacije pri učenju angleškega jezika v okviru stroke. Na- guage learning. The study also aimed to test for dif- men študije je bil tudi preveriti razlike v odzivih študen- ferences in student feedback between those who used tov, in sicer med tistimi, ki so aplikacijo uporabljali le the ColloQuiz application for reference only and those za ponavljanje kolokacij, in tistimi, ki so sodelovali pri who were involved in the creation of the collocations njihovem zbiranju. Rezultati kažejo na jasno potrebo po database. The results show a clear need for students’ aktivnem sodelovanju študentov pri oblikovanju učnih active involvement in the creation of learning materi- gradiv in spodbujajo učitelje tujih jezikov v okviru stro- als and encourage ESP (English for specific purposes) ke, da s pomočjo prosto dostopnih orodij izdelajo mo- teachers to create similar applications tailored to their bilne aplikacije, prilagojene potrebam svojih študentov. students’ needs. 1 Introduction In education, reliance on traditional textbook methods has been fundamentally challenged by modern technological developments, with the value of adopting new technologies coming to the fore. The rapidly increasing capabilities and versatility of smartphones, combined with their widespread use in students’ daily lives, have encour- aged teachers to harness their potential in language teaching as well. This, in turn, has triggered an equally sharp rise in interest among researchers in mobile-assisted language 118 Didactica Slovenica – Pedagoška obzorja (3–4, 2023) learning (MALL). MALL itself is a subfield of mobile learning (m-learning) and dif- fers significantly from e-learning in that its approaches and applications can be used in both formal and informal learning environments (e.g., Kukulska-Hulme, 2021, p. 30). The prospect of learner-determined and ubiquitous learning (Karakaya and Bozkrut, 2022, pp. 2–3), learner engagement, and development of the skills of lifelong learning and technology use (Rintaningrum, 2023, p. 17) explains the appeal of mobile applica- tions in language teaching and, most importantly, seems to offer significant advantages over traditional approaches in terms of learning achievement (Mihaylova et al., 2022). The benefits of using MALL also extend to the field of English for Specific/Academic Purposes (ESP/EAP) in higher education, where the primary focus is on students’ acqui- sition not only of domain-specific terminology but also of collocations and lexical bun- dles that are specific to formal registers and professional domains (e.g., Charles, 2012). It has been proved that student motivation to learn in ESP courses is highly dependent on a learning environment that promotes student autonomy and self-regulation (Šafranj et al., 2021, p. 58), adequately addresses the interests of the “digital generation”, and promotes students’ information literacy (IL) (Juriševič et al., 2017, p. 110), while giving them the opportunity to participate in the design, development and improvement of in- novative and efficient learning methods (Hmelak et al., 2020, p. 147). In the context of Slovenian higher education, mobile learning is still a largely unexplored field. However, the few studies in which it has been addressed report very positive attitudes of students towards its inclusion, e.g., in learning English vocabulary (Čepon, 2018) and biodiver- sity topics (Dolenc Orbanič et al., 2016, p. 97). While recent technological developments have accelerated production and expand- ed the range of available MALL applications (e.g., Nami, 2020), teachers often struggle to find the most appropriate application that would allow for a combination of formal and informal language learning (inside and outside the classroom), meet the specific educational needs of their students, and match the specific course content they wish to focus on (Fan et al., 2023). In general, not many language teachers have the program- ming skills to create their own mobile applications and are therefore consumers rather than producers of mobile technologies (Patton et al., 2019). Given this situation and the particular needs of university students in ESP courses in terms of collocation fluency, the overall objective of this study was to: □ Develop a mobile application that would allow students to collect, revise and translate collocations (from English into Slovene) that frequently occur in formal discourse; □ Explore its usability and efficacy; □ Obtain student feedback on its perceived advantages and disadvantages so as to improve the application in the future. An additional objective of the study was to: □ Determine the differences in student feedback between those who had only used the application for reference and those who had also contributed to the collocations database. Martina Paradiž, PhD: Collaborative M-Learning of Collocations Using... 119 2 Collocations in language learning and translation Collocations as pairs or strings of frequently co-occurring words (e.g., Hoey, 2005; Sinclair, 1991) have been at the focus of L2 and ESP teaching for decades. The reason for this is that L2 speakers cannot rely on their L1 lexicon to produce appropriate col- locations in L2, as their collocational nature is informed not by grammatical, but by conventional usage (Van der Meer, 1998, pp. 316–317). Unsurprisingly, most errors occur in collocations which differ markedly from their equivalents in L1 (Malenica and Mustapić, 2015, p. 43; Peters, 2015). Research conducted over the past two decades on the relationship between the mastery of English-language collocations and language proficiency shows that authors of L2 texts who use collocations correctly are consid- ered more proficient than those who make more collocational errors (Crossley et al., 2015). Research also shows that the acquisition of collocations is slower than that of single-word lexical units (e.g., Laufer and Waldman, 2011) and that collocational er- rors tend to persist even in individuals with advanced levels of language proficiency (Perez-Llantada, 2014). Researchers therefore suggest that language instruction should focus more on the acquisition and correct use of collocations through targeted teaching approaches (e.g., Reppen, 2009; Wray, 2008). To date, several lists of collocations in academic English have been published, e.g., the Academic Word List (Coxhead, 2000), Academic Formulas List (Simpson-Vlach and Ellis, 2010), Academic Collocation List (Ackermann and Chen, 2013), with some focusing on discipline-specific academic English, e.g., Siengsanoh (2021) for mechani- cal engineering and O’Flynn (2019) for arts and humanities. As useful as such ready- made lists may be, some scholars argue that they do not necessarily meet the actual needs of learners (cf. Durrant, 2009; Hyland and Tse, 2007). Conversely, contrastive analysis and translation of collocations have been shown to yield the most positive ef- fects on learners’ knowledge and correct use of L2 collocations (Alharbi, 2017; Boers et al., 2017). Data-driven learning approaches, e.g., teaching students how to effectively use a target language (TL) corpus to search for frequently used collocations within spe- cific genres, discourses and registers, have been shown to reduce their collocational er- rors in L2 writing (e.g., Fang et al., 2021; Charles, 2012) and translation (Jurko, 2014). While much research has been reported on the use of corpora in language learning and translation, few studies to date have addressed the specific use of mobile applica- tions for learning or translating collocations (for an overview of research on mobile ap- plications in MALL, see Ishaq et al., 2021). In this context, studies have focused on mo- bile applications designed for learning collocations previously collected and entered by the teacher and then used by students in various exercises. One of these studies focused on learners’ perceptions of the use and effectiveness of a mobile application (Amer, 2014), one on learner achievement in collocations acquisition (Okumuş Dağdeler et al., 2020), and another on games used for learning collocations from a database of colloca- tions extracted from Wikipedia (Yu et al., 2016). To date, no studies have focused on the efficacy of a mobile application designed for students to collect collocations and then immediately use them in consolidation activities. 120 Didactica Slovenica – Pedagoška obzorja (3–4, 2023) 3 Materials and methods A mobile application was developed for the purposes of this study that allows stu- dents to compile a list of collocations they have encountered in reading materials during the course and in informal settings (outside the classroom), and to access this collabo- ratively created database to immediately revise the collocations and quiz themselves on their equivalents in the TL. The ColloQuiz application (Paradiž, 2020) was created in the MIT App Inventor environment, which uses a block-based visual programming tool that requires no knowledge of coding and also allows for publishing the completed ap- plication in the Google Play App Store. For a more detailed description of working with the MIT App Inventor platform, see Paradiž (2021). Design and functions of the ColloQuiz mobile application Figures 1–3 show the structure and functions of ColloQuiz. ColloQuiz comprises a total of four screens. The first two screens (Figure 1) consist of the title, brief instructions for entering collocations, a button for additional help, an input form (1–4), and a button to open the second screen (Figure 1, right). Figure 1 Screen 1 (Left) and Screen 2 (Right) of the ColloQuiz Application: Entering and Retrieving Entries Martina Paradiž, PhD: Collaborative M-Learning of Collocations Using... 121 The input form is divided into a field to enter the source language (SL) collocation (1), a field to enter the most appropriate TL collocation (2) and a field to enter a weak or incor- rect collocation (3), followed by the student’s name (not numbered). Once the student has filled in the required fields (1–3), tapping the Submit button (4) registers the entry in the collocations database (in Google Sheets). Tapping the button “View Entries and Revise” (5) opens Screen 2 (Figure 1, right), displaying the list of all entries (7). Tapping on individual entries adds them to a tempo- rary database of collocations, which are then used for revision with flashcards (8) or a quiz (9). The three buttons at the bottom of the screen (6), which appear on all screens of the application, contain hyperlinks to the Free Thesaurus, the Corpus of Contemporary American English, and CJVT Kolokacije (Collocations Dictionary of Modern Slovene). These links give students quick access to currently the most relevant online resources to explore collocations in English and Slovene, and to look up synonyms for individual words that might lead to incorrect or weak collocations in English. Figure 2 shows screenshots of Screen 3, where students can revise collocations using flashcards. A dragging gesture selects a random flashcard from a deck of cards containing previously selected collocations (see Figure 1 (7)). The card shows the col- location in the SL (1), and tapping it flips the card to show the correct equivalent in the TL and an incorrect translation or weak collocation (3). The user can then select a new card from the deck (2). Figure 2 Screen 3 of the ColloQuiz Application: Flashcards for Revising Collocations 122 Didactica Slovenica – Pedagoška obzorja (3–4, 2023) The second option for revising collocations is a built-in quiz. As with flashcards, the user selects the collocations they wish to revise in Screen 2 (see Figure 1 (7)). As shown in Figure 3, the user draws a random collocation in the SL (2) from the deck (3). Two buttons then appear, one showing a correct equivalent in the TL and the other an incor- rect or weak collocation (1). If the correct option is tapped, the message “CORRECT” appears, and if the wrong option is tapped, the message “TRY AGAIN” appears (4). Figure 3 Screen 4 of the ColloQuiz Application: Quiz for Collocations Revision All screens also contain a “Back to Form” button, which takes the user back to Screen 1 (Figure 1). Double-tapping the back button on the phone closes the applica- tion. Each time the user opens ColloQuiz, they must reselect collocations for revision. The task of collaborative collocations database compilation As one of the aims of the study was to explore the differences in student feedback between different types of users (those who used ColloQuiz for collocations entry and revision, and those who used it for revision only), 2 groups of students (n = 15 students in each group) were recruited. The ColloQuiz application was first presented to 3rd-year students of Intercultural Linguistic Mediation at the Faculty of Humanities, University of Primorska, Slove- nia, attending the course Slovene-English Contrastive Analysis (Group 1, n = 15) in Martina Paradiž, PhD: Collaborative M-Learning of Collocations Using... 123 the 2020/21 academic year. Students were instructed to identify collocations appear- ing in various reading texts in L2 during lectures and translate them into L1. For this purpose, monolingual corpora were used to explore collocations in context. On account of its adequate size and disciplinary diversity (Durant, 2009), the COCA corpus was considered the most suitable option for this purpose. When translating collocations into L1, students were encouraged to use the Collocations Dictionary of Modern Slovene (CJVT Kolokacije), which includes links to examples from the Gigafida corpus of writ- ten standard Slovene. Students were then asked to use ColloQuiz to enter collocations into the database. In the next step, they were asked to also find collocations in written and spoken (formal, academic) discourse in L2 outside the classroom and add them to the database. The database was regularly checked by the teacher, who deleted repeated entries, corrected spelling mistakes and arranged the entries alphabetically. Students were encouraged to revise the collocations they had collected using the flashcards and quiz features of the application. Once the database contained 100 collocations, it was introduced to 2nd-year students of the same faculty attending the course English 2 (English for Academic Purposes) in the 2020/21 academic year. For this group of students (Group 2, n = 15), the instruc- tions for using the application were to revise the collocations collected, but not to add new collocations to the database. Student survey Upon completion of both courses, in June 2021, students were emailed a link to an online survey on the 1KA online platform (One Click Survey; www.1ka.si) to assess the usability and efficacy of the ColloQuiz application. Students’ participation in the survey was voluntary and they could withdraw at any time. The questionnaire consisted of nine questions. Six questions were of the closed type, with three of these including the option of adding a comment (“Other. Please spec- ify:”). Three questions were open-ended. In the first part of the questionnaire (Q1 – Q3), students were asked on which operating system they had used ColloQuiz (Android, iOS, web) (Q1); whether they had used it for entering, revising, or both entering and revising collocations (Q2); and which English course they had attended during the aca- demic year (Q3). The second part of the questionnaire consisted of two questions asking students to rate the perceived usability (Q4) and efficacy (Q5) of ColloQuiz on a 5-point Likert scale (1 – Strongly Disagree to 5 – Strongly Agree). Q4 comprised ten state- ments of the System Usability Scale (SUS) proposed by Brooke (1996) (see Figure 6). Q5 comprised eight statements about the perceived efficacy of ColloQuiz for learning new collocations (see Figure 7). To avoid response bias, the statements included both positive and negative wording. The third part of the survey consisted of two open-ended questions (Q6 and Q7) asking respondents what they liked (Q6) and disliked (Q7) about ColloQuiz; a closed-ended question (Q8) asking for an overall rating of ColloQuiz on a scale from 1 (Poor) to 5 (Excellent); and an open-ended question (Q9) asking for their general opinion about ColloQuiz. 124 Didactica Slovenica – Pedagoška obzorja (3–4, 2023) 4 Results Collaboratively built bilingual collocations database At the end of both courses (i.e., at the end of the 2020/2021 academic year), the English-Slovene bilingual collocations database consisted of a total of 121 entries. All collocations were entered into a Google Sheets spreadsheet (see Figure 4 below). The application itself did not offer the possibility to delete or edit the registered entries, and the entries were regularly checked for errors by the teacher. Figure 4 Screenshot of the ColloQuiz Collocations Database in Google Sheets Figure 4 shows a screenshot of the ColloQuiz database in a Google Sheets file into which the collocations were entered via ColloQuiz. Upon submitting each collocation, the following data were registered in the database: time of submission (Column A), col- location in the SL (Column B), (suitable) collocation in the TL (Column C), incorrect or weak collocation in the TL (Column D), student’s name (Column E). In the interest of student privacy, the screenshot does not include information on the latter. Results of the survey The survey was completed by a total of 15 students from both groups (n = 30), representing a response rate of 50 %. This rather low response rate is not particularly surprising as participation in the study was voluntary and the courses in which the study was conducted took place during the COVID-19 pandemic, which also may have had a negative impact on the student participation rate. Out of the 15 students who completed the survey, most (10) students used ColloQuiz on an Android OS-based device. Less than a third of all students (4) used it on an iOS-based device and 1 student used it as a web application. Martina Paradiž, PhD: Collaborative M-Learning of Collocations Using... 125 The following groups of application users were discerned on the basis of their re- sponses to Q2 as to what they used ColloQuiz for, i.e., both entering and revising collo- cations, revising collocations, or entering collocations only. The questionnaire included the latter response on the assumption that such a case might occur due to unforeseen circumstances. However, it was chosen by four students, which was surprising as no such instruction had been given in class. As it had not been anticipated that students would choose this response, no follow-up question on the reasons for this was included in the questionnaire. We can therefore only speculate on the possible reasons why stu- dents decided to only enter collocations but not revise them (e.g., misunderstanding the teachers’ instructions, lack of motivation). Although not anticipated, this outcome provided an additional dimension to the results of the survey. There were 8 students in Group 1 – Contributors and Users, 3 students in Group 2 – Users and 4 students in Group 3 – Contributors. As can be seen in Figure 5, students rated the usability of ColloQuiz very positively, with an overall usability score of 81.5, which ranks this application well above the aver- age score of 68 and rates it as “excellent” according to Sauro’s (2019) categorisation (SUS scores between 80.8 and 84.0). However, due to the low number of respondents, this value needs to be treated with caution (Lewis and Sauro, 2017, p. 44). While stu- dents noted the complexity of the application, they also rated it very highly in terms of its ease of use. Figure 5 Results of the Questionnaire Rating Application Usability (Q4) on a 5-Point Likert Scale (n = 15) I believe most people would learn to use this app very quickly. The app is easy to use. I found the various functions in this app were well integrated. I felt very confident using the app. I think that I would like to use this app frequently. I found the system unnecessarily complex. I needed to learn many things before I could start using this app. I found the app very cumbersome to use. I thought there was too much inconsistency in this app. I needed technical support to be able to use this app. 0 0,5 1 1,5 2 2,5 3 3,5 4 4,5 5 Figure 6 shows the responses in relation to the perceived efficacy of ColloQuiz for learning collocations. The results show students’ overall positive attitude towards the application in terms of its benefits, a generally high level of enjoyment in language learning, and a wish to share it with others. However, some students stated that they did not learn much and expressed less enthusiasm to use ColloQuiz in the future. 126 Didactica Slovenica – Pedagoška obzorja (3–4, 2023) Figure 6 Results of the Questionnaire Assessing the Efficacy of the Application (Q5) on a 5-Point Likert Scale (n = 15) I would like to share the ColloQUIZ app with other people. I found this app beneficial for language learning. I would like to continue using and adding to the ColloQUIZ app. I believe I have learned many new collocations using this app. The app has made language learning fun. I don't believe I have learned much using this app. I would not wish to use this or similar applications again. The app has made learning collocations unnecessarily complicated. 0 0,5 1 1,5 2 2,5 3 3,5 4 4,5 5 The aim of the last part of the online survey was to obtain more insight into the aspects of ColloQuiz that students perceived as positive or negative. In response to Q6 about what they liked about ColloQuiz, almost a third of all students stated that they found it easy to enter collocations. They also found it beneficial that it included links to corpora and the thesaurus, and that all new collocations were stored in one place. They liked the fact that anyone can enter collocations, and create and use quizzes, or that it enabled the entry and revision of collocations at any time and place. Students’ responses to Q7 show that what they particularly disliked about Collo- Quiz was the fact that the list of entries was not arranged alphabetically but in the order in which the collocations had been entered. This made it difficult to check whether a particular collocation had already been entered. One student responded that they would prefer the application to randomly select a few collocations for revision, rather than the user having to manually select individual collocations from the list. According to the responses to Q8, students rated ColloQuiz as very good (4.1; SD 0.9), which correlates to its SUS score of 81.5. Their comments show that they ap- preciate the idea behind this application, that they like it and believe it has great educa- tional value. One student stated that they would like to see similar applications for other languages; one expressed certainty that students would enjoy using this application in the future, but that it needs further development; while another commented that the ap- plication should be made more visually appealing. Differences between the three groups of ColloQuiz users One of the aims of the study was to compare the feedback of groups of users de- pending on the way in which they used the application. Figure 8 shows a comparison of the scores for the closed-ended questions by individual group of users. Martina Paradiž, PhD: Collaborative M-Learning of Collocations Using... 127 Figure 7 Comparison of Results of Responses to Q4, Q5 and Q6 between the Three Groups of Users 5 4,5 4 3,5 3 2,5 2 1,5 1 0,5 0 Systems usability Perceived efficacy Overall rating G 1 - Contributers & Users G2 - Users G3 - Contributors Groups 1 and 2 rated ColloQuiz very high for both usability and efficacy (i.e., 4.7) and also gave a very favourable overall rating. ColloQuiz usability was rated signifi- cantly lower by Group 3 (i.e., 3.8). Of the three groups, Group 3 also gave the lowest scores for its efficacy and overall rating (3 and 3.3, respectively). Interesting observations can also be derived from the scores given by the three groups to individual statements included in Q4 and Q5. Students from Group 1 indi- cated with the highest scores of the three groups that they wished to continue using ColloQuiz and adding to the collocations database, and that they feel they have learned many new collocations. Students from Group 2 found ColloQuiz easiest to use and would most like to share it with others. However, students in Group 3 found it cumber- some and stated that they did not learn much from using it. 5 Discussion In general, the ColloQuiz application was positively received by all 3 groups of students. It obtained high ratings in terms of its usability, simplicity, and well-integrated functions that most people can quickly learn to use. On the other hand, students also commented that it was limited in terms of aesthetic appeal and that it needed to be developed further. Students made very useful comments on how to improve its indi- vidual functions (list of entries, random selection of collocations for revision, etc.). This confirms the claim by Fan et al. (2023, p. 2372) that teacher-designed applications for language learning should be diverse and engaging. 128 Didactica Slovenica – Pedagoška obzorja (3–4, 2023) In terms of the efficacy of ColloQuiz, students reported that they found it useful and learned many new collocations. Among the benefits mentioned were the simplic- ity of the process of adding collocations, the storage of useful collocations in a single database, corpus and thesaurus tools hyperlinked from the application, and the ease of revising collocations. It is also very telling that students not only expressed a desire to continue using this and similar applications for other languages, but also to share the application with others. A relatively high score was also assigned by all groups to the statement that ColloQuiz made language learning fun. This is in line with previous findings about students’ greater enjoyment when learning through mobile applications compared to traditional approaches (e.g., Paradiž, 2021; Polakova and Klimova, 2019), and their increased motivation to learn (e.g., Hao et al., 2019, p. 209; Sung et al., 2015, pp. 3–6). Individual steps in the entire task for Group 1 (i.e., detecting, recording and retrieving collocations) also correspond with the learning stages and associated mo- tivational objectives in developing IL as presented by Juriševič et al. (2017, p. 111), namely creating and maintaining interest; establishing the value and importance of IL; building trust in IL; self-reliance and confidence in one’s achievement; and motivation for further engagement. There were, however, significant differences in students’ perceptions by individual groups. Students in Group 1 rated ColloQuiz most positively and also reported the highest level of its efficacy. In other words, those who were most invested in building and using their own learning materials also benefitted the most from the results thereof, and learned the most out of all three groups. This confirms the conclusion reported in the study by Wang et al. (2020) that “students’ sense of contribution was most statistically significant- ly correlated with their L2 motivation” (p. 26) and that student collaboration, as opposed to individual learning, increases efficiency in learning (Lin et al., 2021). Moreover, it is also consistent with Collis and Moonen’s (2006, p. 53) assertion that “participation is not enough: the participant must also contribute in order to make a difference”. Students in Group 2 (Users) – i.e., those who benefitted from the efforts of other students – rated the ease of application use the highest. This was to be expected, as there were several steps in the process of collecting and entering collocations into the database that they were not required to take. This group also expressed the greatest enthusiasm for sharing the application with others. The application, its usability and efficacy were rated least positively by the newly formed Group 3 of contributors only. They could not see the benefits of ColloQuiz and found it more difficult to use and least beneficial in terms of learning new collocations. Collis and Moonen’s (2006, p. 53) as- sertion above could therefore be rephrased as “Contribution is not enough: contributors must enjoy the fruits of their work in order to feel motivated to learn”. As argued by Garcia Botero et al. (2018), students’ main motivation for engaging in MALL ultimately depends on how efficient they perceive this learning to be in terms of improving their academic performance. In the context of ColloQuiz, students were able to immediately assess their learning achievements using the quiz option. The lowest rat- ing of ColloQuiz and the lowest perceived usability and efficacy by the group that did not use these revision options is therefore unsurprising. A recent study comparing the level of motivation, engagement and knowledge acquisition in gamified activities for colloca- tions learning (Foroutan Far and Taghizadeh, 2022) shows that the addition of game ele- ments (scoreboards, points, badges, etc.) into the learning process significantly improves Martina Paradiž, PhD: Collaborative M-Learning of Collocations Using... 129 learners’ motivation, enjoyment of activities, and academic achievement. Moreover, as argued by Fan et al. (2023, p. 2372), such applications should include a visualisation of students’ learning progress. While this was not specifically stated by respondents, it would therefore be beneficial for ColloQuiz and other similar mobile applications to include more game elements and enable tracking of one’s progress in learning. Finally, we must also allow for the possibility that not all students enjoy MALL and the use of smartphones in language learning. According to Garcia Botero et al. (2018), there are various social and technical circumstances and conditions that can af- fect the way students perceive the use of smartphones in the language learning process. As confirmed by Čepon (2018, pp. 130–131), some students prefer traditional methods of instruction and are less eager to employ new tools in their learning process. 6 Conclusion The ColloQuiz application seems to have been a welcome addition to both English language courses. It provided students with the anywhere-anytime opportunity to iden- tify strong collocations (i.e., in their course materials during class as well as in daily life); to translate and explore them using the online corpus and dictionary/thesaurus tools; and to enter them into a shared database for immediate revision and easy retrieval in the future. The focus thus shifted from the teacher and teacher-created or teacher- selected materials to the students, who were empowered to search for new knowledge independently and to use it in the spirit of lifelong learning. Considering the fact that these two English courses were limited to one semester, this was a welcome advantage for the students. While the application itself was rated very positively, students also suggested certain aspects in which it can be improved. Such engagement in the develop- ment of teaching tools is in itself a move away from hands-on classroom management towards a more active and participatory approach to learning. This successful use of a teacher-created language-learning application can also be seen as evidence that it can be done by teachers with limited or no programming skills. MIT App Inventor is just one of the options for creating smartphone and web-based applications available to such users. There are also several forums and tutorials online with user-created instructions that make it much easier to carry out such projects. There are limitations to this study that need to be taken into account. The sample of respondents was very limited, as was the response rate. This pilot study should therefore be repeated on a larger sample of students and other languages for specific purposes before its results can be generalised in any way to MALL in Slovenian higher education or to MALL in general. Nevertheless, as a pilot study in the field of teacher-developed applications in MALL, it provides valuable recommendations and suggestions for up- grading ColloQuiz and developing future mobile applications. More importantly, it of- fers important insights into the different perceptions of the use and efficacy of MALL depending on the degree of student involvement in the design and use of learning ma- terials. As it turns out, students seem to be most motivated when they are involved in the collaborative design of the learning materials and can benefit from the fruits of their efforts invested in learning. 130 Didactica Slovenica – Pedagoška obzorja (3–4, 2023) Dr. Martina Paradiž Sodelovalno mobilno podprto učenje kolokacij z aplikacijo ColloQuiz Splošna razširjenost pametnih telefonov ter njihova vse večja zmogljivost in pove- zljivost odpirajo številne možnosti za izkoriščanje njihovega potenciala tudi pri učenju in poučevanju tujih jezikov. Uporaba pametnih telefonov pri učenju jezika namreč po- nuja možnost učenja v formalnem in neformalnem okolju (Kukulska-Hulme, 2021) in možnost uporabe sodelovalnih pristopov (Paradiž, 2021), pri čemer raziskave kažejo na večjo učinkovitost tovrstnega učenja v primerjavi s tradicionalnimi pristopi (Mihaylo- va idr., 2022). Potreba po raziskavah na področju mobilno podprtega učenja jezikov (ang. mobile-assisted language learning) (MALL) je zato vse bolj prisotna, predvsem v sklopu poučevanja angleškega jezika v okviru stroke, ki je del večine dodiplomskih in podiplomskih študijskih programov v Sloveniji. Čeprav je izbor mobilnih aplikacij za učenje jezika vse širši, pa je po drugi strani težko najti aplikacijo, ki bi v vseh pogledih odgovarjala potrebam študentov in vsebini predmeta (Fan idr., 2023). Obenem tudi učitelji jezika zaradi omejene ali nezadostne podkovanosti v rabi programskih orodij navadno ostajajo uporabniki in se ne preizkušajo kot razvijalci mobilnih aplikacij (Pat- ton idr., 2019). Pri učenju in poučevanju angleškega jezika v okviru stroke ter prevajanju je pre- cejšen poudarek na pravilni rabi kolokacij kot tipičnih besednih zvez v akademskem diskurzu in posameznih znanstvenih vedah (Hoey, 2005). Ti elementi oziroma vzorci jezikovne rabe namreč ne sledijo slovničnim pravilom, ampak je zanje značilna ustalje- nost v rabi in je zato za njihovo uspešno usvajanje potreben vpogled v njihovo dejansko rabo v ciljnem jeziku/diskurzu (Van der Meer, 1998). Raba mobilnih aplikacij pri po- učevanju kolokacij v jeziku stroke je še nezadostno raziskana. V tovrstnih študijah gre predvsem za uporabo predhodno (s strani učitelja) izdelanih seznamov kolokacij, ki jih študenti nato uporabljajo pri vajah na pametnih telefonih. Raziskave, ki bi vključevala sodelovanje študentov pri zbiranju kolokacij s pomočjo mobilne aplikacije, še ni bilo. Članek predstavlja raziskavo, v sklopu katere je avtorica, ki nima znanja iz pro- gramskih jezikov, s pomočjo orodja App Inventor razvila mobilno aplikacijo ColloQuiz, ki je študentom omogočila zbiranje kolokacij v slovenskem in angleškem akademskem diskurzu. S pomočjo te baze kolokacij so študenti nato aplikacijo ColloQuiz uporabljali za učenje in ponavljanje kolokacij v izvirnem in ciljnem jeziku. V prispevku so podrob- neje predstavljene njene posamezne funkcije in možnosti uporabe. Drugi cilj raziskave je bil preveriti mnenje študentov o uporabnosti in učinkovitosti aplikacije ColloQuiz. V ta namen je bil zasnovan spletni vprašalnik z devetimi vprašanji odprtega in zaprtega tipa, s pomočjo katerega so študenti posredovali mnenja o predno- stih in pomanjkljivostih aplikacije, predloge za njeno izboljšavo in mnenje o uporabno- sti tovrstnih aplikacij pri učenju angleškega jezika v okviru stroke oziroma pri učenju in prevajanju kolokacij. V delu vprašalnika, ki se je nanašal na uporabnost aplikacije, je bila uporabljena metoda SUS (Systems Usability Score), ki jo je razvil Brooke (1996). Tretji cilj raziskave je bil preveriti, v kolikšni meri se razlikujejo mnenja med štu- denti, ki so aplikacijo uporabljali samo za ponavljanje in učenje kolokacij, ter tistimi, ki so jo uporabljali tako za vnos kolokacij, njihov prevod v ciljni jezik kot tudi za učenje Martina Paradiž, PhD: Collaborative M-Learning of Collocations Using... 131 kolokacij. Zato je bila aplikacija ponujena v uporabo dvema enako velikima skupinama študentov, pri čemer je prva skupina vnašala kolokacije in jih uporabljala v dodatnih aktivnostih v aplikaciji (skupina 1 – Vnašalci in uporabniki), druga pa jih je zgolj pre- gledovala in uporabljala za ponavljanje (skupina 2 – Uporabniki) (s pomočjo kartic in kviza), ni pa sodelovala pri njihovem zbiranju in vnašanju v skupno bazo. V fazi analize rezultatov spletne ankete se je oblikovala še ena sicer nepredvidena skupina študentov. To je bila skupina, ki je aplikacijo uporabljala zgolj za zbiranje in vnos kolokacij, ne pa tudi za njihovo pregledovanje in ponavljanje (skupina 3 – Vnašalci). Ta skupina se je oblikovala povsem po naključju, saj je bila v anketi pri vprašanju o načinu upora- be aplikacije dodana možnost “samo za vnašanje kolokacij” za primer nepredvidenih okoliščin in poteka dogodkov, ki bi morda kakšnemu študentu onemogočili uporabo aplikacije. Čeprav izbira odgovora ni bila predvidena, je formiranje dodatne skupine odprlo nadaljnje vpoglede v zaznane koristi in učinkovitost tovrstnega učenja glede na način in dinamiko same uporabe mobilne aplikacije. Baza kolokacij, ki je bila zgrajena v obliki preglednice Google Sheets skozi sode- lovalno delo študentov z aplikacijo ColloQuiz, je po zaključku eksperimenta vsebovala 121 kolokacij v slovenščini in njihove prevode v angleščino. Rezultati ankete kažejo, da je bila aplikacija zelo pozitivno ocenjena z vidika uporabnosti oziroma zasnove posa- meznih funkcij (SUS rezultat 81,5 oz. ocena odlično). Študenti so se pozitivno oprede- lili predvsem do enostavnosti njene uporabe, možnosti vnosa in ponavljanja kolokacij kjerkoli in kadarkoli ter povezav do korpusa in tezavra. Tudi splošno mnenje študentov o učinkovitosti aplikacije za učenje kolokacij je bilo pozitivno. Iz večine odgovorov je razvidno, da jim je uporaba aplikacije koristila in jim popestrila ter olajšala učenje jezika. Poleg omenjenih prednosti aplikacije so študenti podali tudi koristne predloge za izboljšanje njenih pomanjkljivosti. Pogrešali so možnost takojšnje abecedne razvrsti- tve vnosov, prav tako si želijo naključnega izbora kolokacij za ponavljanje s karticami ali kvizom namesto njihovega ročnega izbora. Njihovi komentarji se nanašajo tudi na potrebo po nadaljnjem razvoju in izboljšanju vizualnega izgleda aplikacije, kar je po- polnoma razumljivo glede na dejstvo, da je bila aplikacija razvita s pomočjo prosto do- stopnega programskega orodja brez strokovne pomoči razvijalca programske opreme. Kljub temu študenti aplikaciji priznavajo pedagoško vrednost in želijo tovrstne aplika- cije uporabljati tudi pri učenju drugih jezikov. Rezultati ankete so razkrili precejšnje razlike v mnenju študentov o uporabnosti in učinkovitosti aplikacije ColloQuiz glede na način njene uporabe. Študenti v skupi- ni 1 – Vnašalci in uporabniki, torej tisti študenti, ki so bili vključeni v celoten proces oblikovanja in uporabe mobilnega učnega materiala s pomočjo aplikacije ColloQuiz, namreč zbiranja, prevajanja in vnosa kolokacij ter njihovega učenja/ponavljanja s po- močjo kartic in kviza, so aplikacijo ocenili najbolj pozitivno tako glede uporabnosti, učinkovitosti kot tudi splošnega vtisa. Študenti v skupini 2 – Uporabniki, torej tisti, ki so aplikacijo uporabljali le za učenje/ponavljanje kolokacij, niso pa vnašali koloka- cij v bazo, so aplikacijo ocenili enako visoko glede uporabnosti in najbolj izpostavili enostavnost njene uporabe. Med vsemi študenti so študenti v tej skupini tudi v največji meri izrazili željo po deljenju aplikacije z drugimi uporabniki. To ne preseneča, saj so jo le uporabljali in jim je bilo prihranjeno delo v sklopu zbiranja in vnašanja kolokacij. Najbolj zanimivi pa so bili odzivi na novo formirane skupine 3 – Vnašalci, saj odgovori študentov te skupine kažejo, da jim aplikacija ni zelo olajšala ali popestrila učenja ko- 132 Didactica Slovenica – Pedagoška obzorja (3–4, 2023) lokacij ter da se z njeno uporabo niso veliko naučili. To sicer glede na način uporabe, ki se od običajnega zapisa kolokacije s pisalom na papir razlikuje zgolj po uporabljenem mediju, ni presenetljivo. V splošnem pozitivni odzivi študentov na rabo in učinkovitost mobilne aplikacije pri učenju angleškega jezika v okviru stroke potrjujejo trditev Wanga idr. (2020, str. 26), da je motivacija pri učenju tujega jezika povezana z občutkom doprinosa, prav tako pa tudi ugotovitve Lina idr. (2021), da sodelovanje pri učenju povečuje učinkovitost učenja. Nižje zadovoljstvo skupine 2 (Uporabniki) po drugi strani potrjuje stališče av- torjev Collis in Moonen (2006, p. 53), da samo sodelovanje ni dovolj, pač pa morajo sodelujoči za dosego najboljših rezultatov tudi prispevati k doseganju skupnega cilja. V luči rezultatov raziskave, v sklopu katere se je oblikovala tudi skupina, ki ni uporabljala gradiva, pač pa zgolj prispevala k njegovemu nastanku, bi to trditev veljalo nekoliko spremeniti, namreč da samo sodelovanje pri doseganju skupnega cilja ni dovolj, sode- lujoči morajo za učinkovito učenje tudi uporabljati sadove svojega in skupnega dela. Na splošno že pozitiven sprejem mobilne aplikacije za učenje kolokacij sam po sebi priča o učinkovitosti njene uporabe pri učenju kolokacij, saj študente pri mobilno podprtem učenju jezika dokazano najbolj motivirajo učni dosežki oziroma učinkovitost učenja (Garcia Botero idr., 2018). V navezavi na ugotovitve avtorjev Foroutan Far in Taghizadeh (2022), da vnašanje elementov in mehanizmov igre oziroma “igrifikacija” učnih materialov (npr. sprotno točkovanje nalog, raba nivojev) pozitivno vpliva na mo- tivacijo in učinkovitost učenja, bi veljalo aplikacijo ColloQuiz in prihodnje tovrstne aplikacije nadgraditi tudi z elementi računalniških iger. V pričujoči raziskavi je bila odzivnost študentov zelo nizka (morda tudi zaradi po- sebnih okoliščin med pandemijo covida-19), zaradi česar je rezultate raziskave potreb- no interpretirati s precejšnjo mero previdnosti. Vseeno pa rezultati te pilotne študije odpirajo zanimive vpoglede v percepcije učnega pristopa s strani študentov, saj le-ta preusmerja fokus z učitelja na študenta in v ospredje postavlja sodelovanje med študen- ti, njihovo lastno generiranje učnega materiala in odpira možnosti za učenje jezika tako v formalnem kot tudi neformalnem okolju. V prihodnje bi mobilno aplikacijo ColloQuiz veljalo nadgraditi v skladu z odzivi študentov in izsledki drugih raziskav s tega področja oziroma razviti druge tovrstne aplikacije, ki bi jih uporabljalo večje število uporabni- kov. Pilotna študija namreč dokazuje, da lahko tudi učitelj brez znanja programiranja ustvari mobilno aplikacijo, ki je uporabna in učinkovita pri učenju ciljnih vsebin. REFERENCES 1. Ackermann, K. and Chen, Y. H. (2013). Developing the Academic Collocation List (ACL) – A Corpus-Driven and Expert-Judged Approach. Journal of English for Academic Purposes, 12(4), 235–247. https://doi.org/10.1016/j.jeap.2013.08.002. 2. Alharbi, R. M. (2017). Acquisition of Lexical Collocations: A Corpus-Assisted Contrastive Analysis and Translation Approach. [PhD Thesis]. Newcastle University. Available at: http:// theses.ncl.ac.uk/jspui/handle/10443/3660. 3. Amer, M. (2014). A Study of Learners’ Usage of a Mobile Learning Application for Lear- ning Idioms and Collocations. CALICO Journal, 31(3), 285–302. https://doi.org/10.11139/ cj.31.3.285–302. Martina Paradiž, PhD: Collaborative M-Learning of Collocations Using... 133 4. Boers, F., Demecheleer, M., He, L. et al. (2017). Typographic Enhancement of Multiword Units in Second Language Text. International Journal of Applied Linguistics, 27(2), 448–469. https:// doi.org/10.1111/ijal.12141. 5. Charles, M. (2012). “Proper Vocabulary and Juicy Collocations”: EAP Students Evaluate Do-it-Yo- urself Corpus-Building. English for Specific Purposes, 31(2), 93–102. https://doi.org/10.1016/j. esp.2011.12.003. 6. Collis, B. and Moonen, J. (2006). The Contributing Student: Learners as Co-Developers of Le- arning Resources of Learning. In: Engaged Learning with Emerging Technologies (pp. 49–67). Springer. https://doi.org/10.1007/1-4020-3669-8_3. 7. Coxhead, A. (2000). A New Academic Word List. TESOL Quarterly, 34(2), 213. https://doi. org/10.2307/3587951. 8. Crossley, S. A., Salsbury, T. and Mcnamara, D. S. (2015). Assessing Lexical Proficiency Using Analytic Ratings: A Case for Collocation Accuracy. Applied Linguistics, 36(5), 570–590. https://doi.org/10.1093/applin/amt056. 9. Čepon, Slavica (2018). Using Snapchat. Didactica Slovenica – Pedagoška obzorja, 33(3–4), 123–135. 10. Dolenc Orbanić, N., Cotič, N. and Furlan, P. (2016). Mobilno učenje na primeru spoznavanja biodiverzitete. Didactica Slovenica – Pedagoška obzorja, 31(1), 86–99. 11. Durrant, P. (2009). Investigating the Viability of a Collocation List for Students of English for Academic Purposes. English for Specific Purposes, 28(3), 157–169. https://doi.org/10.1016/j. esp.2009.02.002. 12. Fan, X., Liu, K., Wang, X. et al. (2023). Exploring Mobile Apps in English Learning. Journal of Education, Humanities and Social Sciences, 8, 2367–2374. https://doi.org/10.54097/ehss. v8i.4996. 13. Fang, L., Ma, Q. and Yan, J. (2021). The Effectiveness of Corpus-Based Training on Collocati- on Use in L2 Writing for Chinese Senior Secondary School Students. Journal of China Compu- ter-Assisted Language Learning, 1(1), 80–109. https://doi.org/10.1515/jccall-2021-2004. 14. Foroutan Far, F. and Taghizadeh, M. (2022). Comparing the Effects of Digital and Non-Digital Gamification on EFL Learners’ Collocation Knowledge, Perceptions, and Sense of Flow. Com- puter Assisted Language Learning, 1–33. https://doi.org/10.1080/09588221.2022.2146724. 15. García Botero, G., Questier, F., Cincinnato, S. et al. (2018). Acceptance and Usage of Mobile Assisted Language Learning by Higher Education Students. Journal of Computing in Higher Education, 30(3), 426–451. https://doi.org/10.1007/s12528-018-9177-1. 16. Hao, Y., Lee, K. S., Chen, S. T. et al. (2019). An Evaluative Study of a Mobile Application for Middle School Students Struggling with English Vocabulary Learning. Computers in Human Behavior, 95, 208–216. https://doi.org/10.1016/j.chb.2018.10.013. 17. Hmelak, M., Rudaš, A. and Lepičnik Vodopivec, J. (2020). Vključevanje študentov v razvoj ino- vativnih izobraževalnih modelov. Didactica Slovenica – Pedagoška obzorja, 35(3–4), 147–163. 18. Hoey, M. (2005). Lexical Priming: A New Theory of Words and Language. Routledge. 19. Hyland, K. and Tse, P. (2007). Is there an “Academic Vocabulary”? TESOL Quarterly, 41(2), 235–253. https://doi.org/10.1002/j.1545-7249.2007.tb00058.x. 20. Ishaq, K., Mat Zin, N. A., Rosdi, F. et al. (2021). Mobile-Assisted and Gamification-Based Lan- guage Learning: a Systematic Literature Review. PeerJ Computer Science, 7, e496. https://doi. org/10.7717/peerj-cs.496. 21. Juriševič, M., Šorgo, A. and Boh Podgornik, B. (2017). Motivacija, učenje in informacijska pismenost dodiplomskih študentov. Didactica Slovenica – Pedagoška obzorja, 32(2), 101–116. 22. Jurko, P. (2014). Target Language Corpus as an Encoding Tool: Collocations in Slovene-Engli- sh Translator Training. ELOPE: English Language Overseas Perspectives and Enquiries, 11(1), 153–164. https://doi.org/10.4312/elope.11.1.153-164. 23. Karakaya, K. and Bozkurt, A. (2022). Mobile-Assisted Language Learning (MALL) Research Trends and Patterns through Bibliometric Analysis: Empowering Language Learners throu- gh Ubiquitous Educational Technologies. System, 110, 102925. https://doi.org/10.1016/j.sy- stem.2022.102925. 134 Didactica Slovenica – Pedagoška obzorja (3–4, 2023) 24. Kukulska-Hulme, A. (2021). Reflections on Research Questions in Mobile Assisted Language Learning. Journal of China Computer-Assisted Language Learning, 1(1), 28–46. https://doi. org/10.1515/jccall-2021-2002. 25. Laufer, B. and Waldman, T. (2011). Verb-Noun Collocations in Second Language Writing: a Cor- pus Analysis of Learners’ English. Language Learning, 61(2), 647–672. https://doi.org/10.1111/ j.1467-9922.2010.00621.x. 26. Lewis J. R. and Sauro, J. (2017). Can I leave this One Out? The Effect of Dropping an Item from the SUS. Journal of Usability Studies, 13(1), 38–46. 27. Lin, C., Barrett, N. E. and Liu, G. (2021). English Outside the Academic Sphere: A Mobile‐Ba- sed Context‐Aware Comparison Study on Collaborative and Individual Learning. Journal of Computer Assisted Learning, 37(3), 657–671. https://doi.org/10.1111/jcal.12514. 28. Malenica, F. and Mustapic, E. (2015). Acquisition of English Collocations Through Productive Skills: Analysis of Translation Errors. International Journal of English Linguistics, 5(4). http:// dx.doi.org/10.5539/ijel.v5n4p42. 29. Mihaylova, M., Gorin, S., Reber, T. P. et al. (2022). A Meta-Analysis on Mobile-Assisted Language Learning Applications: Benefits and Risks. Psychologica Belgica, 62(1), 252–271. https://doi.org/10.5334/pb.1146. 30. Nami, F. (2020). Educational Smartphone Apps for Language Learning in Higher Education: Stu- dents’ Choices and Perceptions. Australasian Journal of Educational Technology, 36(4) 82–95. https://doi.org/10.14742/ajet.5350. 31. O’Flynn, J. (2019). Developing an Academic Collocation List for Arts and Humanities. [Master’s Dissertation]. University of Warwick. 32. Okumuş Dağdeler, K., Konca, M. Y. and Demiröz, H. (2020). The Effect of Mobile-As- sisted Language Learning (MALL) on EFL Learners’ Collocation Learning. Dil Ve Dilbilimi Çalışmaları Dergisi, 16(1), 489–509. https://doi.org/10.17263/jlls.712891. 33. Paradiž, M. (2020). ColloQuiz. [Mobile application software]. Koper: Univerza na Primorskem. 34. Paradiž, M. (2021). CoLecTer KIN: Mobile Application for Collaborative Bilingual Glossa- ry Compilation in the ESP Classroom. ELOPE: English Language Overseas Perspectives and Enquiries, 18(1), 91–106. https://doi.org/10.4312/elope.18.1.91-106. 35. Patton, E. W., Tissenbaum, M. and Harunani, F. (2019). MIT App Inventor: Objectives, De- sign, and Development. In: Kong, SC., Abelson, H. (Eds.) Computational Thinking Education (pp. 31–49). Singapore: Springer. https://doi.org/10.1007/978-981-13-6528-7_3. 36. Pérez-Llantada, C. (2014). Formulaic Language in L1 and L2 Expert Academic Writing: Con- vergent and Divergent Usage. Journal of English for Academic Purposes, 14, 84–94. https://doi. org/10.1016/j.jeap.2014.01.002. 37. Peters, E. (2015). The Learning Burden of Collocations: The Role of Interlexical and Intralexical Fac- tors. Language Teaching Research, 20(1), 113–138. https://doi.org/10.1177/1362168814568131. 38. Poláková, P. and Klímová, B. (2019). Mobile Technology and Generation Z in the English Lan- guage Classroom – A Preliminary Study. Education Sciences, 9(3), 203. https://doi.org/10.3390/ educsci9030203. 39. Reppen, R. (2009). Exploring L1 and L2 Writing Development Through Collocations: A Corpus-Based Look. In: Barfield, A. and Gyllstad, H. (Eds.). Researching Collocations in Another Language: Multiple Interpretations (pp. 49–59). Palgrave Macmillan. https://doi. org/10.1057/9780230245327_4. 40. Rintaningrum, R. (2023). Technology Integration in English Language Teaching and Learning: Benefits and Challenges. Cogent Education, 10(1), 2164690. https://doi.org/10.1080/233118 6X.2022.2164690. 41. Sauro, J. (2019). Five Scales to Measure Customer Satisfaction. MeasuringU. https://measurin- gu.com/csat-scales/. 42. Siengsanoh, B. (2021). Lexical Collocational Use by Thai EFL Learners in Writing. LEARN Journal: Language Education and Acquisition Research Network, 14(2), 171–193. 43. Simpson-Vlach, R. and Ellis, N. C. (2010). An Academic Formulas List: New Methods in Phra- seology Research. Applied Linguistics, 31(4), 487–512. https://doi.org/10.1093/applin/amp058. 44. Sinclair, J. (Ed.) (1991). Corpus, Concordance, Collocation. Oxford, UK: Oxford University Press. Martina Paradiž, PhD: Collaborative M-Learning of Collocations Using... 135 45. Sung, Y. T., Chang, K. E. and Yang, J. M. (2015). How Effective are Mobile Devices for Language Learning? A Meta-Analysis. Educational Research Review, 16, 68–84. https://doi.org/10.1016/j. edurev.2015.09.001. 46. Šafranj, J., Gojkov Rajić, A. and Bulatović, V. (2021). Motivation for Foreign Language Com- munication. Didactica Slovenica – Pedagoška obzorja, 36(2), 49–63. 47. Van der Meer, G. (1998). Collocations as one Particular Type of Conventional Word Combina- tions. Their Definition and Character. EURALEX ’98 Proceedings I–II, Papers submitted to the Eighth EU-RALEX International Congress on Lexicography in Liège, Belgium, 1, 313–322. 48. Wang, Z., Hwang, G.-J., Yin, Z. et al. (2020). A Contribution-Oriented Self-Directed Mobile Learning Approach to Improving EFL Students’ Vocabulary Retention and Second Language Motivation. Educational Technology & Society, 23(1), 16–29. 49. Wray, A. (2008). Formulaic Language: Pushing the Boundaries. Oxford Applied Linguistics. Oxford: Oxford University Press. 50. Yu, A., Wu, S., Witten, I. H. et al. (2016). Learning Collocations with FLAX Apps. In: Dyson, L. E., Ng, W. and Fergusson, J. (Eds.). Proc. 15th World Conference on Mobile and Contextual Learning: Mobile Learning Futures – Sustaining Quality Research and Practice in Mobile Lear- ning, October 24–26, 2016 (pp. 291–294). Sydney: The University of Technology. Martina Paradiž, PhD (1976), Assistant Proffesor at University of Primorska, Faculty of Humanities Naslov / Address: Trg 15. aprila 1b, 6215 Divača, Slovenija Telefon / Telephone: (+386) 030 358 677 E-mail: martina.paradiz@fhs.upr.si