In many countries high-stakes oral tests of modern languages are assessed by external assessors who have been particularly trained to assess that particular test. This is not the case in Sweden where every teacher assesses his or her own students which puts a great deal of faith in the ability of the teacher to assess correctly. In this paper six teachers of French as a modern language in Sweden, of which three are experienced teachers and three are novices without a teaching degree, assess the oral production of six students in grade 9. The students have been audio-recorded in pairs when they have taken part in an assignment which highly resembles the oral exam of the so called subject-test in modern languages. This test is not compulsory to do in Sweden.

The purpose of the study is to gain an understanding of what the respondents include in their comments for an assignment which is set to be assessed as an overall assessment of the student’s oral ability of French. Another purpose is to look at the teachers’ grading and written assessments from a validity and reliability-point of view.

The respondents have filled in two digital surveys; one about their background in teaching and how familiar they are with the subject-test and its oral exam as well as the guidelines that follow; another called the assessment survey where they listen to the audio-recordings produced for this study and note what grade they want to give each student’s oral performance. In the assessment survey the teachers are urged to write down the basis of their assessment.    

The results show that the summative assessments of the teachers are nor equivalently graded, nor equivalently commented. Even though this is the case it can be said that all the teachers have commented in a valid way, with the exception of a teacher who didn’t comment anything on two students. They have used the criteria from the curricula and in the instances that they have used other parameters not found in the assessment overview, such as comparing students with each other and noticing a student’s effort, similar comments can be found in material produced by the Swedish National Agency for Education, which comments on the knowledge requirements related to oral ability of modern languages for grade 9.

What makes the results to be inequivalent is that the intra-rater-reliability and inter-rater-reliability is low. Based on a grading scale of six grades the three students who received the lowest average grades have a one grade difference between the highest and lowest grade given by the respondents. The three students with the highest average grades have a two grade difference between the highest and lowest grade given by the respondents.

In their comments the respondents have commented on different things, and on several occasions respondents differ in their views relating to how well a student uses vocabulary, the level of fluidity in the language, even to the point as regarding the same language aspect in a negative or positive light. It can also be seen that respondents give similar comments but different grades to the same student.