Evaluating and scoring submitted data can help your team triage large amounts of submissions. Adding a series of weighted questions will result in an average score for every submission your team reviews. Sorting on this score will help segment your submissions prior to deeper evaluation. Qualitative and quantitative review questions will help your team with their final review.
NOTE: The submission Reviewer Fields feature is only available for those with a 'Plus Advanced' or 'Premium Advanced' subscription. If you would like to upgrade your account, email us at info@dosomegood.ca.
**VIDEO** Check out this quick video that walks you through the different 'internal use field' options and how to easily add them to your form:
- Two Types of Internal Use Fields
- When Should I Use an Internal Use Field?
- Visibility on Reviewer Fields
- Form Builder and Types of Reviewer Fields
- Reviewer Field Scoring and Weighting
- Adding Reviewer Fields to your Advanced View
- Filling Out Reviewer Fields in your Advanced View
- Filling Out Reviewer Fields in the Submissions Details Page
- Best Practices
Two Types of Internal Use Fields
There are two powerful types of internal use fields you can use independently or together.
- Shared Admin Fields (also known as 'Internal Field - Admin Only') - Used when administrators want to capture information intended for internal use only, such as GL codes.
- Reviewer Fields - Used for structured scoring and feedback
When Should I Use a Reviewer Field?
If your submission evaluation process involves collecting internal feedback from one or more team members (or other interested parties), then reviewer fields are a good choice for your workflow. This is similar to the 'Internal Field (Admin-Only)' that is hidden from submitters, but a single 'Reviewer Field' can be filled out by multiple team members.
You can add a Reviewer Field to any form at any time. Just like internal fields, when you add a Reviewer Field, it will be available on all submissions to that form, even if the submission occurs before the Reviewer Field was added to the form.
In addition to allowing multiple team members to fill out a single reviewer field, you can apply weighting to your review questions to create an aggregate numeric score for each submission your team reviews. Use advanced views and the weighted score of each submission to help sort and make bulk evaluation workflow decisions. More examples of how to design your evaluation workflow can be found in the Best Practices area.
Visibility of Reviewer Fields
Reviewer Fields are designed to allow each team member to provide their own feedback independently.
When you add a reviewer field to a form, you can choose if each evaluator can see the other answers or if the answers from other evaluators should be hidden so as not to bias their answer. You set the visibility option in the 'Properties' window of the review question inside the Form Builder tool.
More important than the property you set in Form Builder is the permissions of your reviewer. Administrators or Limited Administrators with 'Full Access' will always be able to see everyone's review answers. Limited Administrators with 'Own Evaluations and Scores Only' will follow the setting you've chosen in the 'Properties' window for the fields to keep the responses private from each other or not.
Form Builder and Types of Reviewer Fields
Reviewer fields exist on their own tab in the 'Design Tools' area in the Form Builder tool. All reviewer fields have the property to allow you to keep reviewer responses private and also have the option for you to provide Reviewer Instructions under the Reviewer Question label. Every question also has a weight that will contribute to the overall score for the submission.
NOTE: Reviewer Fields can be added to a form at any time. A Reviewer Field will be available on submissions even if the submission happened before the field is added.
Let's look at the different types of reviewer fields you can choose from:
Rating Picker
This is a valuable field for quantitative evaluation with a visual component. You can set a maximum value that an evaluator can select and also choose which icon is used to represent the rating picker (stars or hearts currently). When the evaluator chooses a rating, it resolves to a numeric field that can be summed or averaged to help with the decision making.
Feedback
The feedback element is used for collecting text and qualitative feedback from your reviewers. You are allowed to set a maximum number of characters for your input field.
Numeric Input
The numeric input is similar to the rating picker except that the evaluator types in a number rather than clicking on the rating control icons to assign a value. Maximum values can be assigned along with weights to contribute to an overall score. You are allowed to set a maximum value for your numeric input.
Yes/No Checkbox
If you need a simple checkbox to indicate if a criteria is met or not, this is the review element for you to use. It is a simple checked or unchecked value that your evaluator can choose for any question.
Single Select Dropdown
You can create a list of options for your evaluator to choose from. The list can be converted to a list of radio buttons or checkboxes if that works better for your team.
Reviewer Field Scoring and Weighting
Check out this 30-second video that walks you through how scoring and weighting works with your internal review fields:
Reviewer questions have the ability to contribute to an overall score for each submission. If you want a particular reviewer questions to contribute to a score you must check the box in the properties to 'Include in tally for scoring'.
Once you have decided to include a field in the scoring tally, you will then have the option to add a weight to the value.
- For numeric and rating picker elements, the score/number entered by the reviewer is multiplied by the weight to come up with a score for that question. For example, if the Rating Picker is out of 10 and has a weight of 10, if the reviewer chooses a value of '7', then the overall score for that reviewer question in '70'. Value 7 multiplied by weight 10 equals 70.
- For all other questions, as long as an answer is provided, the weight for that question is added to the overall score. For example, if a Checkbox Review field has a weight of '25' and the evaluator does not check the box, the score for the question will be '0'. If the evaluator checks the box, the score for that question will be '25.
Scores for each review question are averaged across all of the evaluators and an overall score is computed for each submission that takes into account the scores and weights for the answers from each reviewer. This results in a single average score for each submission which can be used as part of your evaluation workflow.
NOTE: In the future we will allow single select dropdown review questions to have different scores for each different item in the dropdown.
Adding Reviewer Fields to your Custom Submissions View
If you want to add your Reviewer Fields to a custom submissions view, you will first have to limit that view to only show submissions for the form you are evaluating. You can add a column for each of the reviewer questions on your form by customizing your columns.
If you want the evaluator to perform the review right from your custom submission view, the fields should be made editable.
Once added to your custom submissions view, these reviewer field columns will show your own answer to the question, and if you have permission, it will show you a button to view all of the other answers to the question.
When you add a review question to your view, a score column is automatically added for that field as well. This is the score multiplied by the weight property of the review question. If you have permission, you will be able to see everyone else's score for that question as well.
There is also a Universal column you can add for 'Submission Score' that shows the aggregate score for all answers to review questions. This column is special because it is available at all times, not just when limiting a view to a single form.
Filling Out Reviewer Fields in Your Advanced View
When looking at a review question in your custom view you will see your own entry (if any), an indication of how many people including yourself have added a review answer. Next to each review field will be the average score for that question. It is the sum of scores for all the answers to that question divided by the number of people who have replied. If you have permissions to see answers from other reviewers, you can click on the checkmark and count button to see each contributors entries.
Filling Out Reviewer Fields in the Submissions Details Page
The review questions on the Submission Details page are integrated in with all of the submitter responses, they are colour coded green like in other areas of the platform and can be edited in the same way you would enter a submitter response.
The score for your entries are listed beside each review question you answer and the overall score, including the average of all reviewers scores, is listed at the top of the Submission Details page.
NOTE: On the Submission Details page, you only see your own answers to review questions.
Best Practices
Using Private Review Questions and Shared Review Questions
Each reviewer is able to provide their own entry for every review question. Sometimes you may want your review team to be able to see each others responses, and sometimes you may want to keep them hidden so they are not biased by the answers of others.
All Full Administrators are always able to see the entries for all reviewers. Limited administrators with the "Submission Evaluation & Scoring" permission will also always see entries for all reviewers. If however a limited administrator reviewer does not have the "Submission Evaluation & Scoring" permission for their role, the property for the particular review question whether or not the reviewer can see the entries from other reviewers.
Using Review Questions For Evaluation Triage
In programs where you have a great many submissions, you may not have the resources to do a detailed evaluation of every single submission. You may want to do a first pass, evaluating key answers using weighted review questions. Key values like cause alignment, request amount and existing previous giving are easy to evaluate as long as you ask the right questions in your form. Doing a triage review will allow you to quickly rank submissions according to your own criteria and remove submissions that are not going to be aligned with your giving. Only the remaining submissions with strong alignment need to be evaluated fully by your reviewers.
Using Weights in Non Numeric Fields To Score a Submission
Most Review Questions use the suers entry to quantify the value assigned. The 'Feedback' and 'Checkbox' questions can not be quantified by Do Some Good. If the field has no entry from the reviewer, the assigned value is 0. If the field has any entry (the checkbox checked, any text in feedback) the assigned value is 1. You can use the 'Scoring Weight' property to augment this value so that its score is representative of the importance of the question.
As an example, you may have several scoring questions all worth a value up to ten. Then you have an important checkbox like 'Are they an existing customer?' which may be very important when considering this submission. Checking that box will only add one point to the score but if you put a higher Scoring Weight on that checkbox, it will become more significant in the overall score to match with its importance.
Adding Review Questions to an Existing Form That Has Submissions
Adding, removing or changing questions that your applicant will fill out is not advised as you will have inconsistent submissions that will be difficult to evaluate fairly. Review Questions are different and can be added to a program that has been running for years and is full of submissions waiting for review.
The key thing to keep in mind is to not add or remove any of the applicant questions, only add or modify Review Questions in form builder. As long as you are not in the middle of a review cycle, you are free to make changes to your review questions.
Add as many review questions as you like, adjust scoring, weighting, labels or whatever else fits your program. All of the changes will be immediately available for new and existing submissions to that form.
The checklist below about creating an advanced reviewer workflow is equally applicable in setting up a review workflow as it is for an existing form with submissions.
A Checklist for Advanced Reviewer Workflow
This checklist is meant to convey the best practices for adding review workflow to your submission. It is not the way things must be done, but rather a guideline. Some things may not apply to your program, and you are free to ignore them. Some things may not be in the best order for your program, and you are free to rearrange things.
Do Some Good is a very flexible system that can be configured in many different ways. Your team will want to run your giving programs in your own way. To get the best out of Do Some Good, you should begin by evaluating how you would run your program in an ideal world. Then you can configure Do Some Good to streamline the process as much as possible. Here are some things to think about:
- How many people will be involved in decision making, will it be one person, a team, or multiple teams?
- Will your group be made up of trained Do Some Good administrators or will it be made up of new users who are not familiar with the system?
- How many giving programs do you run? If it is multiple, does the same person/team make the decisions for each or is each program run by different people?
- Will everyone involved in the review process have full access or will some people administer the program and others will just be reviewers with lower permissions?
- Do you want people to evaluate submissions and provide qualitative and quantitative feedback before final decisions are made?
- Do you want the reviewers to be able to see what other reviewers are submitting?
- Do you want to involve other teams like marketing for automatic story generation or accounting for processing payments?
Update your form in Form Builder
This step can be completed at the same time you are designing a form, or it can be completed after a form is already up and running and has collected submissions.
- Do NOT touch any of the applicant questions
- Open the third tab ‘Review’ in ‘Design Tools’ of Form Builder.
- Drag rating pickers or whatever other review field you wish into your form.
- Position the review questions where it makes sense for your reviewers have the context they need to answer that question.
- If you want your reviewers to only see their own answer, have the ‘Keep reviewer responses private from each other’ and 'Include in tally for scoring’ options selected in the review question properties.
- The order of the questions in your form will dictate the order of questions in the Submission Details page.
- The order of the questions in your form does not have to match the order of your columns in your custom view and subsequent data downloads.
- Most review questions use the reviewer answer to provide a score (along with the weight). The exceptions are the Single Select Dropdown (define a value for each choice). The Checkbox and Feedback fields have a score of 0 if no data is added or 1 if text is added or the box is checked. The Likert Scale has numerous value settings to choose from in the properties.
- Add a weight to your review fields if desired. It is a multiplier on the raw value entered by the reviewer and the result is the ’Score’ for that question and affects the overall submission score.
- The default weight is one but can be any whole number
- Even if you have similar questions (like a rating picker or Likert scale) in multiple places, they can be weighted differently to reflect the relative importance for your review.
- The weighted score is calculated every time a view or submission is opened. That means you can alter the weight even after the review is complete to adjust the importance of each question and the scores will update.
Create Your Role for Your Reviewers
If you want or need to have reviewers that have limited permissions and can ONLY see the submission area and ONLY the submissions you have assigned to them, follow these steps. If you are using full administrators for your review will not need to create a new roll but your full administrator reviewers will be able to see each others entries in Review Questions.
- Select ‘Forms, Submissions, Story Generator’ and the ‘Owned Items Only’ option.
- This will limit the reviewer to ONLY seeing the one tab and ONLY submissions/forms they own.
- Select the ’Submission Evaluation & Scoring’ item so your team can see and respond to review questions. Choose the option for 'Own Evaluations and Scores Only' unless you want these reviewers to see everyone's scores.
Create Your Groups
Regardless of whether you create a specific role for your reviewers, Groups are always a good idea. They help funnel notifications to the correct people, generate notifications at the right times and provide communication distribution lists. A default custom submission view can also be set for a group and all of its members.
- Create a group for your program management team and add yourselves to it. This will make sure the administrators get notifications about any new submissions in need of triage and you can set a group default view.
- Assign your management team group to be the owner of the form.
- Create a group for each of your review teams for this program. Some teams use one review group and every reviewer evaluates every submission. Other teams split up the workload for various reasons:
- splitting up due to large volume of submissions
- region where the submission came from
- By branch or franchise
- The dollar amount being requested may require a different evaluation criteria
- Use as many groups as you need for your program.
- Don’t assign anyone to the reviewer groups yet.
Create Your Management Custom View
The people managing the program will likely want a view into the program to see the progress of reviewers, gain insight into committed funds, have an overview of the organizations involved etc. The needs may be different than reviewers who are scoring individual submissions.
- Limit the view to only show submissions to your one form
- Select ‘Any or No Owner/Group’ for the Filter by Owner or Owner Group option. You want to be able to see all submissions to this form, including ones owned by reviewer groups.
- Include any relevant information you need or want reported on when you download the view.
- Add an owner/owner group column to the view so you know who to call if the work isn’t progressing.
- Include all of the review fields and associated scoring fields so you can see how many people have reviewed each submission at a glance and easily see who has contributed or not.
- Include the overall score which is the sum of average scores for all of the review fields.
- This can be can be used to sort the submission view and is valuable when making final decisions.
- When you add review fields to your form, make sure to check the box to make them editable.
- Save and name your management view.
- From the ’Saved Views’ dropdown, select ‘Manage Views and Defaults’
- Find your view and add your management group to it
- Set the ’Submission Tab Default’ for the management group you created earlier.
- Every time you visit the Submissions tab, this will be the view that loads for you. Keep in mind that your personal default always overrides a group default.
- You can remove this default later, or set your own personal default submission tab view at any time.
Create Your Reviewer View
Your reviewers will have different needs than the program managers and they likely will not be be familiar with Do Some Good management. Providing reviewers with a custom view with everything they need to make decisions, and making that view default will help to set your reviewers up for success.
- You can likely start with your management view and pare things down, removing information that is not relevant. Or, you may feel that your reviewers need something completely different from managers and you start fresh. That is your call.
- Limit the view to only show submissions to your one form
- Only show content ‘Owned by the admin using this view’.
- With any review fields, make sure you make them editable so the reviewer can add their answers.
- Save and name your reviewer view.
- Again visit the ‘Manage Views and Defaults’ tool.
- Add your management group to the review view so the management team can see it if they choose, but do not set it as a default for the management team.
- Add each of your review groups to the review view and set the ’Submission Tab Default’ for each of those review groups.
Set Up Your People
You will need to apply the correct permissions, or invite team members into roles, and add members to groups to take full advantage of the automated notifications and messaging. You may choose to do this before or after training users on the review process. In this best practices document, training is the next step but you should decide which order is best for your team.
- Change the permissions on any existing team members who are part of your review teams to be a Limited Administrator in the new review role. Only do this if the users do not have other responsibilities that require expanded permissions.
- These users will get a notification when this happens.
- Divide up your reviewers by whatever criteria you need for your program. It is always good to have some sort of ‘balance’ of opinions in the hopes of having similar scores from each group for similar submissions.
- Visit the new reviewer Owner Group management pages, click ‘Add Team Members’ and select the appropriate people for each group.
- These users will get a notification when this happens.
- If there is a new reviewer who is not affiliated with your business, they can be invited directly into a Role and Group with one action.
- Visit the Owner Group details page.
- Click the Invite New Limited Admin to Group button in the right margin.
- Chose their role, press next.
- Add one or more email addresses and an optional custom message.
- When the brand new user accepts the invitation and creates their user account, they will already be in the correct role and group.
Train Your People
- Share information with your team about how you want them to score, what they are responsible for (every submission in their tab, and every green question in each submission).
- Tell them that they will not see anything until the review phase opens and submissions are assigned.
- Describe where to go in the platform to perform the review (Forms and Story Generator page, Submissions tab).
- If you provide a link, do not include any text at the end of the url like “?viewId=aTtTthsgslk234” because that would send them to a specific view and override their default view that you set.
- Tell them they can score in the view or they can score in the submission details page, whatever you feel works best for your team.
- In the details page they will need to click on ’Show All / Edit Responses’ in the overview tab to see the whole form.
- To edit a value in the details page, the reviewer must click the pencil icon next to that review field.
- Tell them that their default view will give them a quick overview of all of the review questions and show their rating so they will have an easy way to track their progress.
- Make it very clear that the green questions are the ONLY thing they touch. No state changes or other actions are authorized.
- Give them the tip about the wide screen mode for submission views, especially if they are scoring in the view.
- Tell them they will get a notification when the submissions are ready for their review.
Divide Up the Submissions
- Use the multi select checkboxes in the management submission view to select all of the submissions for one group and then assign ownership to one of the review groups. Repeat for each group until all submissions are assigned to a review group.
- If you assign ownership with the bulk action only one notification is sent to the reviewers, if you do submission assignments individually, the reviewers will get a notification for every single submission.
Monitor
- You will be able to see all submissions with your management view, you will have quick access to see the numbers of reviews for each question. Note that your own contribution (if any) is included in that count.
- Sort by the Submission Score column to see which submissions are performing best and also which submissions are not scored yet.
- If you need to communicate with your teams, make use of the group messaging list for your review groups.
- Once the review is complete, you are free to carry on with the final decision making and approving / denying submissions.
- Before making final decisions, consider re-assigning all submissions back to your management group. This will remove permissions for the reviewers to access them and would keep your final decision confidential to the management team.
Comments
0 comments
Article is closed for comments.