Developing an Automated Writing Evaluation Tool for Slovene: Insights from a National Teacher Survey

Abstract

We present the most important findings of an extensive study involving 1,024 teachers from primary and secondary schools in Slovenia. The study was conducted as part of the project Empirical foundations for digitally-supported development of writing skills (PROP). Its main purpose was to develop an automated writing evaluation (AWE) tool that would enable teachers in primary and secondary schools to support students in correcting their written texts in Slovene. AWE tools for English texts already exist and are well established, but no such comprehensive tool is yet available for Slovene. Their most important advantage is that they enable faster, more appropriate, and more accurate feedback to students on their written texts. On the one hand, the AWE tool includes machine identification and categorization of grammatical errors, which relieves teachers of routine corrections and gives them more time to focus on higher taxonomic goals. On the other hand, it provides a digitally designed model for giving feedback. In order to develop such a tool for reviewing students' written texts in Slovene, it was essential to get an overview of existing practices in correcting students' texts and insight into the needs of teachers in this regard. Therefore, we conducted an extensive survey among teachers of various subjects in primary and secondary schools throughout Slovenia. The results showed that teachers spend even more than 10 hours per month correcting students' written texts. It was found that Slovene language teachers and then class teachers correct students' texts and provide them with feedback more often than teachers of other subjects. Less than half of teachers monitor their students' language progress, more than 20% of high school students do not have to correct their written texts, and approximately 20% of students who do correct their texts are not checked again by their teachers. The survey also looked at teachers' experiences with correcting the written texts of students with special needs. We found that about 20% of teachers do not get any recommendations or advice from special education teachers on how to grade the writing of students with special needs. Almost a third of teachers believe that students with special needs do not write more correct texts, even if they are given extra time to write. Approximately 45% of teachers believe that the difference in the accuracy of written texts produced by students with special needs, depending on whether they write by hand or type, depends on the individual. Less than half of teachers (39.7%) believe that AWE tools are welcome, while others have reservations about their use or are undecided. A similar proportion of teachers would also use such a tool (38.6%), while other teachers would use it if it were easy to use or if they knew how to use it, or only in certain cases or in higher grades. Only 7.2% of respondents were explicitly against the use of such a tool. The survey results confirm that the development of a digital tool for correcting students' written texts is useful and necessary. It would primarily complement teachers in their correction work, rather than replace their feedback, and serve as an aid to students directly while they are writing, helping them to produce more accurate texts and develop their independence. The tool could make writing easier, especially for students with special needs. The results also show that teachers of other subjects (not just Slovenian) should be encouraged to correct students' written texts to a greater extent and made them aware of the importance of feedback in this activity. The PROP project and the survey of Slovenian teachers conducted as part of it offer the possibility of comparative research for other languages for which, like Slovene, no AWE tool currently exists.