Evaluating and scoring submitted data can help your team triage large amounts of submissions. Adding a series of weighted questions will result in an average score for every submission your team reviews. Sorting on this score will help segment your submissions prior to deeper evaluation. Qualitative and quantitative review questions will help your team with their final review.
NOTE: The submission Reviewer Fields feature is only available for the 'Advanced' feature subscription and above.
COMING SOON!
- When Should I Use a Reviewer Field?
- Visibility on Reviewer Fields
- Form Builder and Types of Reviewer Fields
- Reviewer Field Scoring and Weighting
- Adding Reviewer Fields to your Advanced View
- Best Practices
When Should I Use a Reviewer Field?
If your submission evaluation process involves collecting internal feedback from one or more team members (or other interested parties), then reviewer fields are a good choice for your workflow. This is similar to the 'Internal Field (Admin-Only)' that is hidden from submitters, but a single 'Reviewer Field' can be filled out by multiple team members.
You can add a Reviewer Field to any form at any time. Just like internal Fields, when you add a Reviewer Field, it will be available on all submissions to that form, even if the submission happened before the Reviewer Field was added to the form.
In addition to allowing multiple team members to fill out a single reviewer field, you can apply weighting to your review questions to create an aggregate numeric score for each submission your team reviews. Use advanced views and the weighted score of each submission to help sort and make bulk evaluation workflow decisions. More examples of how to design your evaluation workflow can be found in the Best Practices area.
Visibility of Reviewer Fields
Review fields are similar to form questions where you set the 'Internal Field (Admin-Only)' property to 'On'. Just like Internal Fields, the submitter can never see any values entered by your team into the Reviewer Fields.
The Internal Field setting is best used for things where you only need to record one piece of information that is universal for the submission. Reviewer Fields are designed to allow each team member to provide their own feedback answer independently. All of the Reviewer Field feedback and scoring can be used to aid in the final decision.
When you add a reviewer field to a form, you get to choose if each evaluator can see the other answers or if the answers from other evaluators should be hidden so as not to bias their answer. You set the visibility option in the properties of the review question inside the form builder.
More important than the property you set in the form builder is the permissions of your reviewer. Administrators or limited administrators with Full Access will always be able to see everyone's review answers. Limited administrators with 'Own Evaluations and Scores Only' will follow the property to keep the responses private from each other or not.
NOTE: As we continue to develop our evaluation tools, we will create formal reviews where the permissions will not matter when inside a review session. A reviewer will only see what the 'Review Question' property dictates.
Form Builder and Types of Reviewer Fields
The reviewer fields exist on their own tab in the Design Tools pallet in Form Builder. All Review Fields have the property to allow you to keep reviewer responses private and also have the option for you to provide Reviewer Instructions under the Reviewer Question label. Every question also has a weight that will contribute to the overall score for the submission.
NOTE: Reviewer Fields can be added to a form at any time. A Reviewer Field will be available on submissions even if the submission happened before the field is added.
Rating Picker
This is a valuable field for quantitative evaluation with a visual component. You can set a maximum value that an evaluator can select and also choose which icon is used to represent the rating picker (stars or hearts currently). When the evaluator chooses a rating, it resolves to a numeric field that can be summed or averaged to help with the decision making.
Feedback
The feedback element is used for collecting text and qualitative feedback from your reviewers. You are allowed to set a maximum number of characters for your input field.
Numeric Input
The numeric input is similar to the rating picker except that the evaluator types in a number rather than clicking on the rating control icons to assign a value. Maximum values can be assigned along with weights to contribute to an overall score. You are allowed to set a maximum value for your numeric input.
Yes/No Checkbox
If you need a simple checkbox to indicate if a criteria is met or not, this is the review element for you to use. It is a simple checked or unchecked value that your evaluator can choose for any question.
Single Select Dropdown
You can create a list of options for your evaluator to choose from. The list can be converted to a list of radio buttons or checkboxes if that works better for your team.
Reviewer Field Scoring and Weighting
The Review questions have the ability to contribute to an overall score for each submission. If you want a particular reviewer questions to contribute to a score you must check the box in the properties to 'Include in tally for scoring'.
Once you have decided to include a field in the scoring tally, you will then have the option to add a weight to the value.
- For numeric and rating picker elements, the score/number entered by the reviewer is multiplied by the weight to come up with a score for that question. For example, if the Rating Picker is out of 10 and has a weight of 10, if the reviewer chooses a value of '7', then the overall score for that reviewer question in '70'. Value 7 multiplied by weight 10 equals 70.
- For all other questions, as long as an answer is provided, the weight for that question is added to the overall score. For example, if a Checkbox Review field has a weight of '25' and the evaluator does not check the box, the score for the question will be '0'. If the evaluator checks the box, the score for that question will be '25.
Scores for each review question are averaged across all of the evaluators and an overall score is computed for each submission that takes into account the scores and weights for the answers from each reviewer. This results in a single average score for each submission which can be used as part of your evaluation workflow.
NOTE: In the future we will allow single select dropdown review questions to have different scores for each different item in the dropdown.
Adding Reviewer Fields to your Advanced View
If you want to add your Reviewer Fields to an Advanced Submission View, you will first have to limit that view to only show submissions for the form you are evaluating. You can add a column for each of the the review questions on your form. If you want the evaluator to perform the review right from the advanced submission view, the fields should be made editable. This Reviewer Field column will show your own answer to the question, and if you have permission, it will show you a button to view all of the other answers to the question.
In addition to the answers for each review question there is a 'Score' column that can be added to the view for each Review Field. This is the your score multiplied by the weight property of the review question. If you have permission, you will be able to see everyone else's score for that question as well.
There is also a Universal column you can add for 'Submission Score' that shows the aggregate score for all answers to review questions. This column is special because it is available at all times, not just when limiting a view to a single form.
Comments
0 comments
Article is closed for comments.