Item-based Marking on FLOWmulti

WISEflow offers an assessment method for FLOWmulti, called item-based marking. This feature allows the manager, via the flow, to automatically assign marks according to a point mapping system. The item-based marking also allows the manager to assign assessors to score individual items. 

Item-based marking can be set up on FLOWmulti flows. Click on Manage Settings on the Marking and Review pane. See the Managing Assessors and Reviewers article for more details.

The manager can edit the matrix and distribute markers to the various items on the assignment when item-based marking is enabled. The Manage matrix button will only be available if there is an assignment on the flow. It will not work as intended without an assignment on the flow as there will be no items on the flow. Item-based marking will be removed automatically when the assignment is removed from the flow and needs to be set up again when a new assignment has been added.

Setup

1. Reset the matrix, which will delete the existing matrix and all marks given to items. Or reset the matrix and marker distribution, which will delete the existing matrix, including all marks that might have been given and make it possible for another marking distribution method to be selected.
2. Once you have finalised the allocation, you need to save the setup of the matrix.
3. The matrix allows you to:
   • Deactivate items if items need to be removed from the assessment/exam and should not be marked. 
   • Set the question to be automatically scored by the system.
   • Set the minimum and maximum marks that markers can score in the marking tool.
4. Assign items that cannot or should not be auto-scored to markers or marker groups.

At the bottom of the page, you can see the grade scaler on the flow and adjust if necessary. 

1. Define the conversion intervals by dragging the interval border or entering the appropriate value directly.
2. The system automatically calculates the maximum number of marks that participants can receive on all items.
3. Grade range as per chosen assessment type.

Assigning Assessors to Items

If a FLOWmulti assignment consists only of auto-scorable questions, it is possible to assess the flow without adding an assessor or reviewer. The system will automatically distinguish between questions that can be auto-scored and those which need an assessor to be evaluated (i.e. essay questions, audio recordings, etc.). If an assignment contains items that need to be assessed manually, these will be highlighted in the matrix. You will have to appoint an assessor to those sections or deactivate the item from the assessment before the setup can be saved. 

If items need to be assessed by an assessor, you have to add assessors to your flow and assign them to specific items. 

When assessor groups are allocated to items, all assessors within the assessor group have to agree on the same points per participant and item even if reviewer approval is set as the procedure for submitting final grades.

Once you have added assessor(s) or assessor groups to your flow, you will need to assign them to specific items via the item-based marking matrix. A shortcut to this page will be at the top of the work distribution on the marking page. You cannot appoint assessors to individual participants through the work distribution. 

You can either assign an entire item to one assessor or assessor group (100%) or split the items between multiple assessors/assessor groups, split up equally by the system. After entering the setup for the item-based marking, you will have to assign the relevant items.

Marking Period

Once the marking period has started, you will be able to access the Marking overview from the participant list at the bottom of a flow page. 

Marking Overview

1. Access the grade scaler to make adjustments if necessary. Drag the lines or manually change the distribution and click Save grade scaler to change the distribution.
2. List of all participants enrolled on the flow.
3. The score column shows how many marks the participant currently has. This score changes as markers complete their assessments and score items.
4. The current calculated grade for a participant is based on the scores calculated in the individual items.
5. The finalised grade and the registered mark for the participant.
6. Overview of the items, whether an item is auto-scored or scored by an assessor  /assessor group and how many marks the participant has received in each item. Hovering over each cell shows the origin of the score (by an assessor or automatic scoring).
   The different colours in the cells depict the assessment process of the section.

   The assessor has not opened the item.	The assessor has opened the item but has not marked it yet.	The assessor has marked the item as Done but has not submitted any scores.	A score has been registered.	Only for assessor groups  Assessors have not entered the same grades.  An agreement has to be reached.

7. The Unlock submitted scores and Transfer items are emergency measures in case wrong scores are registered, or markers are unable to assess items. See the tab Emergency Features below for more information.

Finalising Grades

The calculated mark will appear in the Scaled Grade column when all assessors have finished their marking and registered all grades. You can adjust the mark distribution key by clicking the Adjust grade scaler to the right. Changes to the grade scaler can change the marks in the Scaled grade column. When everything looks correct, you can register the final grade. Click Set final grades.

The system will then ask you to approve this action.

When marks have been finalised, they cannot be changed any further.

Emergency Features

Item-based marking in FLOWmulti has incorporated simple emergency measures to handle problems or mistakes with the marking. All features and examples below are only available during and after the marking period of the flow.

Unlocking Scores for Assessors

When an assessor has registered a final grade, and there is no conflict with a co-assessor, the assessment is locked and changes can no longer be made. If, after this point, changes do need to be made e.g. an error has been made, you can either unlock all submitted scores for a single item or unlock all submitted scores for a marker.

Unlock Submitted Scores for a Single Section

By clicking on the pencil icon at the bottom of a specific item, you can unlock all marks that have been submitted by a marker for this particular item and will mark them as Done instead. The assessor can now change the scores for this item. The assessor has to register all scores for this item again.

Unlock All Submitted Scores for an Assessor

This feature unlocks all scores made by one or more assessors so that they are marked as Done instead. This means that assessors have access to all items and can change the score. Assessors must register all scores again.

Transfer Items to Different Assessors

When an assessor is unavailable or cannot finish the marking, the item-based marking allows you to substitute markers. It is also possible to add a new marker to the flow instead if one is available.

Transfer items does not work for marker groups.
All changes made by the providing marker (annotations, etc.) will be transferred as well.  

1. In the marking overview, click Transfer items.
2. Mark the unavailable assessor to the left and the receiving assessor to the right.
3. Click Transfer to substitute the assessor.

Deactivate or Change Marking Method for Items

Item-based marking allows the manager to deactivate an item if it should no longer be included in the assessment process. It is also possible to switch an item from automatic scoring to assessment by an assessor if manual scoring is required. 

On the Marking Overview, click the at the bottom of a section column in the score table. Click Deactivate item to remove the item and its marks from the assessment.

It is always possible to activate items again via the matrix (1). From here, you can also set an item to manual assessment instead of automatic scoring by assigning it to one or multiple assessors (2). It is only possible to switch an item back to automated scoring if it contains only auto-scorable questions.  

Reset Finalisation of Grades

When grades are finalised, you can see the following message at the bottom of the marking overview:

For now, the item-based marking system is locked to avoid undesirable changes to grades. If complications arise (e.g. an item has been found to contain content that is not part of the curriculum and needs to be removed from the assessment), it is possible to unlock the system again.

On the flow page, find the drop-down menu in the top right corner: . Click Reset grades, tick Execute the action, and click Proceed.

This feature resets all grades and opens up the item-based marking page again (only to the point that you have to finalise grades again).

Redistribute Items or Remove Item-Based Marking Method

If you find, after the marking period has started, you need to change the distribution of items for the assessors or use the normal assessment mode (where assessors are assigned to participants and not items), the manager can change or reset the whole marking mode.

From the Marking Overview, click Open assessor distribution and confirm that you would like to go to the setup.

This opens the matrix for the flow (where you distributed the items to assessors).

Any changes made and saved on this page resets all assessments done by all assessors. We advise that you only commit to making changes here in situations where it is necessary.

From here, you can:

1. Redistribute the items by changing the matrix table. Click Save setup to save the changes made.
2. Reset the matrix table to have no distribution.
   • Click on  to open the drop-down menu.
   • Click Reset matrix and confirm.
3. Remove item-based marking mode from the flow, allowing you to choose the usual Assign to participants marking method.
   • Click on  to open the drop-down menu.
   • Click Reset matrix and assessor distribution method and confirm.
   • This leads you back to the flow page. From here, you can remove item-based marking under Marking and Review.