Standard Setting

Owned by Neill Magill
Oct 25, 2022
2 min read

Note: All standard setting techniques should take into account any questions which have been excluded post-exam.

Modified Angoff

Stage: Pre-Exam

Modified angoff is a standard setting technique which is applied before an exam is run. Calculating the cut score is quite simple, take the average (mean) of all the ratings of the questions. Some question types may have multiple rating dropdown menus depending on how they are marked. For example, a multiple response question with marking per option will have multiple rating dropdowns.

Note: The rating dropdown menu increment is 5%.

Group Modified Angoff

Stage: Pre-Exam

How the group Modified Angoff is designed to work is to allow multiple academics to perform their own review in their own time. The group review option can then pull in these individual reviews and present an interface to allow a review team to see how similar the scores are and arrive at a group consensus.

Ebel

Stage: Pre-Exam

Ebel is a standard setting technique which is applied before an exam is run. Calculating the cut score is a two step process. The first step requires an academic or a group to categorise each question (or part of a question depending on the marking) along two orthogonal dimensions: 1) difficulty, and 2) importance. Difficulty categories are: easy, medium and hard. Importance categories are: essential, important and nice to know. All questions must firstly be categorised into one of the 9 types in this 3x3 matrix. The second step is then to specify the percentage of borderline candidates you expect to get each of the 9 categories correct.

The cut score is calculated by:

  1. For each of the 9 categories totalling the number of marks assigned to that category.
  2. Multiply the number of marks in each category by the percentage of borderline candidates in that category.
  3. Sum the above results for all 9 categories ($cut_marks).
  4. Pass mark = ($cut_marks / ($total_marks * 100)) * 100

The same principles can be applied to an optional second grid for calculating distinction pass mark.

Hofstee

Stage: Post-Exam

The Hofstee method works by plotting a cumulative 'survival curve' of student percentage scores on a graph. It then uses HTML 5 to allow an academic to define four points: 1) minimum failure rate, 2) maximum failure rate, 3) minimum passing point, and 4) maximum passing point.

Because this form of standard setting requires post exam marks there is a check that the exam has finished (i.e. past the end time/date). Also, it is using data from the cache tables so there is a call to the ResultsCache class.

Borderline Method

Stage: Post-Exam

The only paper type in Rogo to support the Borderline Method is currently OSCE stations. The Borderline Method works by calculating its cut score based on the scores of borderline students. For this to work the OSCE station must have its 'Overall Classification' to one of the options that includes a 'borderline' method. Staff, in the process of marking, will then manually rate students as passing, failing or a borderline candidate. If the 'Pass Mark' dropdown in 'Edit Properties' dialog is set to 'Borderline Method' then Rogo will automatically calculate the cut score:

Cut score = median of borderline candidate marks.