Standard Settings

Introduction

Standard setting is important for all examinations, but particularly so for those where a professional accreditation results, such as in medicine.  We set standards that provide a cut-off point that separate the competent from the non-competent.  These standards should be established and regularly reviewed by panels of examiners and practitioners.  They should also be applied on an exam-by-exam basis. 

There are effectively two ways to set standards.  In test-centred models, the panel reviews test items and provides judgements as to 'just adequate' levels of performance.  Examinee-centered models require that judges identify and sort an actual group of examinees into fail, pass and distinction groups. Both models can complement each other and one can be used to refine the output of the other.

Test-centered models

Both the test-centred models below should be applied before the exam, not after.

Modified Angoff Model

The Angoff model is an absolute method of standard setting.  It does not compare candidates to each other.  The panel develops a definition of the a hypothetically  'minimally competent practitioner' (MCP).  They are then shown an exam item and asked to consider what percentage of MCPs would be able to answer the question.  This procedure is repeated throughout the exam paper.  Ratings are then averaged across all items and all judges to determine the passing point.

Setting up Modified Angoff

Click the Create New Modified Angoff link in the sidebar. Rogo displays the exam script with orange dropdowns.  You set the percentage of minimally competent candidates that can be expected to answer the question correctly:

You can then select to Save and Exit or Save and Continue.  Clicking the checkbox Save ratings into question bank allows the percentages to be persisted outside the current paper for future reference.

Rogo displays the various ratings set for the paper:  incomplete standards are flagged in red.  As you can see, I have only set standards for 60 out of 193 possible marks on the paper.

Marking

The Modified Angoff model calculates the passmark by multiplying the total marks for each response by the mean percentage set by all examiners.  The individual question marks are then summed to give the passmark. 

There is no scaling of scores unlike the Ebel Model.

Ebel Model

Ebel's method is far more granular than the Angoff method.  It uses a 3x3 grid to classify examination questions.  One axis describes difficulty: easy, medium and hard, while the other  describes the importance of the subject being tested:  nice to know, important and essential to know.  A preponderance of hard, nice-to-know exam questions will bring the pass mark down; more easy, essential to know questions push the pass mark up.

Setting up the Grid.

Rogo uses Ebel Grid Templates to set up the grid and then the template is applied to the examination.  Each cell must specify a percentage of its questions that a borderline candidate is expected to get right. 

A typical grid set up in this fashion is shown below:

Setting up the questions

With the paper open, click the Standard Setting link in the sidebar. Then click Create New Ebel Method.  Rogo will present you with the examination script augmented with orange dropdowns.  Each dropdown indicates the difficulty and importance of the question:

After you have set the importance for each question, you will come to the end of the script. You will see the Pass and Distinction grids.  Here you can set up the percentages of questions in each category that a student is expected to get in order to pass:

Use of the Distinction Grid is optional.  If you decide not to use this grid, you can choose to use the top 20% of candidates or simply not apply it.

You can either Save and Exit or Save and Continue,  the latter allows you to carry setting standards for the script.  Save and Exit displays your standard setting exercise in the list for that examination:

Marking

Rogo calculates a passmark for the paper.  It sums up the total number of marks allocated to each grid cell, and multiplies it by the percentage for that cell.  It then repeats this process across each cell. The passmark or 'cut score' is total of the weighted marks.  So, if a percentage goes up in a grid cell then the passmark increases.

Adjusting scores for Pass Marks and Distinctions

Rogo allows you to set pass and distinction marks in the Paper Properties dialogue.  These generally are 40 or 50% and 70%.  However, since the Ebel Grid generates its own pass and distinction marks, this complicates the picture.  Rogo scales the overall percentage mark according to the interval in which it lies.  If a standard-set passmark of 55% is awarded, then Rogo will scale this to 40 or 50%.  If a standard-set distinction of 80% is awarded, then Rogo will scale this to 70%.  Intermediate marks are scaled accordingly.  The diagrams below show how this is done:

If the Distinction Grid option is Do Not Apply then two (not three) intervals are used for scaling scores to 100%.  If however Use Top 20% is selected, then Rogo awards distinctions to the top 20% of the standard-set cohort regardless of the paper's current distinction mark.

Examinee-Centered Models

There are two such models applied in Rogo:  borderline and Hofstee.  These are applied after the exam, not beforehand.

Hofstee Model

The Hofstee model is a post-hoc method of standard setting. 

1. Rogo plots the cumulative number of exam results at a given percentage.  This gives a 'survival curve':
   
2. The Max Pass, Min Pass are the maximum and minimum acceptable cut scores that represent a pass on the paper. Max Fail and Min Fail are the max/min percentages of students that either pass or fail the examination.  You can set these by dragging the coloured lines in from the graph margins, or typing them in directly.
   
3. Rogo uses these percentages to set a bounding box on the curve.  It then draws a diagonal between the northwest and southeast corners of this box.  The intersection of the diagonal with the survival curve yields the cut score:

In a pathological situation one could draw a bounding box that does not overlap the curve.  For this reason, Hofstee standard setting should not be used pre-exam.

Distinction standard setting works identically to pass standard setting, although one should ensure that the distinction cut score exceeds the pass cut score.

Save options

• No change to paper:  none of the changes made in this screen affect any of the marks awarded
• Set paper pass/distinction marks to standard setting: the two cut scores are used in the Properties dialog for the paper for pass and distinction marks
• Set paper to standard setting and scale marks:  uses the same the scaling method as Ebel Grid scaling with an active distinction grid

Borderline Model

Rogo uses this model in OSCE Stations.  At the end of the OSCE Station, the examiner will record whether the candidate has passed, failed or is a borderline candidate.  However the OSCE Station has also recorded the candidate's total marks.  Borderline standard setting takes the median of the marks of all the borderline candidates for that station.  Borderline candidates who exceed the median are subsequently passed, and those who don't are failed.