Assessment: the messiness of marking and troublesome knowledge

ITLSIG Co Secretary-General Helen Puntha

At the November 2021 ITLSIG meeting, ITLSIG committee member Dr Fiona Meddings provided an excellent presentation on ‘The messiness of marking and troublesome knowledge’ based on research she undertook with twenty-six academics across four Higher Education institutions. Here I will share my learning and reflections from some key topics which came up within the presentation itself and the ensuing group discussion.

Experiences of novice markers

Fiona noted that novice markers are likely to find marking very time-consuming and challenging depending on the initial mentorship they have received; she gave the example that they may not realise that it is ok to return earlier scripts and re-grade (prior to finalising and releasing grades to students) whilst their own assessment literacy and discernment grows throughout the marking process; this can be to the detriment of both marker and student.

It is potentially difficult for a novice marker to challenge a more experienced markerwho is often their direct line manager. Novice markers tend to compare their marking to the marks of the more experienced peer with whom they are paired, and this carries the risk of creating a ‘glass ceiling’ effect for students. Fiona gave an example of an interesting phenomenon that can affect both novice and experienced markers alike: markers may grade assessments which are easier to read higher even if they do not address the criteria as well, as one research participant noted, “you’ve got to get past the veneer of goodness”.  Fiona highlighted that a novice marker is a valuable asset to a marking team; after ten years an experienced tutor might not read the assessment brief fully to check if something new has been added and they may skim-read the actual work whereas novice markers have a closer relationship with the assessment criteria, they tend to read more slowly and carefully and as a result are likely to have useful insights in particular for work which is on the boundary between grade bands.

Different marking systems

We discussed that there are differences in marking systems across countries, regions and institutions. This can create challenges for colleagues marking across programmes of studies with different marking systems. It also creates a unique challenge for individuals who are beginning to work in a new country with a different marking system; colleagues may be a very experienced marker in their home nation but will need to consciously familiarise themselves with assessment criteria in the place they are working/marking.

Fiona noted that even within a marking team on the same programme of study, individuals may have different marking approaches, e.g. one person may start at 0 and add marks for things students have done; another starts at 100 and take marks off; those two individual markers would be very unlikely to arrive at the same overall grade using those different approaches and that could create a big differentiation in mark awarded by first and second markers which then takes time to resolve. Fiona noted that accuracy, exactness and precision are very difficult to achieve regardless of the marking system used.

We discussed different perspectives on students failing an assessment; in some systems a student failing an assessment would flag a need to review the validity of the assessment itself. In other systems markers would only review an assessment or components of an assessment if a number of students failed, e.g. if the majority of students fail one particular exam question then the question may be removed; tutors would remove said question prior to grade calculation and report this to the exam board.

Challenges and opportunities of feedback practices

We discussed that feedback can be extremely time-consuming with rising student numbers and that institutional policies which stipulate that feedback should be provided within a certain timeframe, e.g. 21 days, 15 days and this creates a challenge to provision of quality feedback. Fiona noted that a way round this, depending on your institutional policies can be to provide cohort feedback because there tends to be a pattern in what students are demonstrating at different grade bands. Cohort feedback involves giving all students their individual grades then providing a feedback handout / holding a feedback session where the tutor gives cohort feedback along the lines of “most students who achieved this grade were doing this aspect well and need to improve in this aspect and this aspect … students who got this other grade band were …”. Another way of approaching this is to use grading matrices where you highlight how students have met (or not met) different aspects of the criteria using pre-written feedback which demonstrates how you would achieve specific grades on specific aspects of the assessment criteria. Matrices provide more personalised feedback without the need for lots of written text. See example from Nottingham Trent University, UK.

It was highlighted that students do not always read their feedback. Fiona noted that if students come and see her specifically about assessment she will ask them to bring any feedback they were previously given with them or they access it digitally within the meeting so that she can help them decipher it. Even if students do read the feedback they may not understand it so it is a responsibility of the tutor to help students understand their feedback. We discussed that it is easier to write feedback for work which is poor, harder when the work is good but essential that we give all students the correct amount and type of feedback as to be helpful, e.g. a student who gained 75% will be wondering how they can achieve the extra 25% next time.

Model answers

The suggestion of model answers was raised and this prompted a lively discussion. There were different perspectives on this; one being that model answers can introduce a reference point, quickly help students understand the assessment criteria and therefore raise standards. A second cautionary perspective was that students sometimes copy the model answer rather than use it as a guide. A suggestion was made that a useful exercise is to provide students with several model answers with a range of grades and ask the students to mark these themselves in groups and discuss with each other and the class why each submission should achieve a certain grade; this can sharpen their assessment literacy.

References

Fiona noted that some tutors do not check undergraduate references and that this is a travesty which is doing a disservice to our undergraduate students. Some academics have commented that they only check postgraduate references because the quality of those references could impact them personally as their name may be on any published work by that student. Fiona made the powerful argument in opposition to this mindset that

it makes no sense to build a house with poor foundations (at undergraduate) then wonder why the roof is toppling off (at postgraduate level).

This discussion point raised the broader issue of the varied importance given to referencing across different academic cultures and within different nations.

Assessing groupwork

We spoke about challenges of groupwork; the main challenge for markers being how to distribute marks fairly amongst students according to contribution. Fiona and colleagues came up with a solution where tutors provide the overall grade of 100% to a group and ask students what percentage each group member should get based on their contribution. Tutors can then calculate the overall grade of individual group members taking into account this report of their contribution. The idea of students taking responsibility for group marking was viewed as positive, and it was noted that it could be done anonymously.

A lot more ground was covered within the session such as the role of subjectivity, intuition, normative vs criterion referencing and other topics. Thank you Fiona for a thought-provoking session.

Key takeaway ideas for practice:

• It is useful for a marking team to have a thorough discussion about the approach they will take and their understanding of the assessment brief prior to beginning marking, this can support familiarisation with the assessment criteria and reduce the likelihood of large variations between markers;
• Electronic marking and feedback is advantageous in terms of sharing with external examiners also novice markers so they can understand the criteria and so that students and tutors can easily access students’ previous feedback;
• Assessment feedback which contains the following elements will be most useful to students: what was good, what was not so good, what could the student do differently in future assessments (feedforward);
• It is useful if groupwork assessment contains some element of student responsibility for the grades awarded to different group members;
• Students should be prompted at regular intervals to refer back to any previous feedback they have received and to check the meaning of it with tutors if in doubt;
• Students achieving higher grades are entitled to receive the same quantity and quality of feedback as students achieving lower grades, e.g. details of what they should keep doing (they may not be conscious of certain things they are getting right) and what they could start doing to add extra marks.;
• Detailed, targeted cohort feedback can be a valuable way of providing meaningful, timely feedback to students whilst reducing workload for tutors.

Fiona’s presentation was followed by an exciting foray into Iceland led by Tiana who took us on a virtual tour of this fascinating island and introduced us to Icelandic language, history and culture. Thank you, Tiana, we felt transported by your wonderful tour.