AI grading enables our platform to automatically evaluate responses using advanced text analysis, making it easier to gather insights and compare answers. For example, if you want to assess a candidate’s grammar, you can configure the question to measure grammar correctness. AI grading is available in Writing questions and our Whiteboard.
Create a Whiteboard or Writing Question
Here are the steps to create a whiteboard or writing question:
1. From your CodeSignal account, click on the Questions from the Content tab.
2. Click + Create Question.
3. You will be taken to a page where you can choose the type of question you want to create. Select either the “Whiteboard” or “Writing” question type and click CREATE.
4. Enable AI grading
Note: AI grading must be enabled for your organization by CodeSignal. If you do not have access to this feature and desire it, please contact your customer success manager or the CodeSignal Support team.
5. Create an Evaluation Rubric
To begin, you must create an Evaluation Rubric that outlines the attributes that are important to you. There is no limit to the number of attributes you can assess in one question, but you must be sure to carefully define criteria to minimize variability. Here is a sample rubric structure:
Attribute 1
| Level | Description |
| Very Unsatisfactory | Met 0 of the given criteria (can be more generic if there is no criteria list) |
| Unsatisfactory | Met only 1 of the given criteria with some inaccuracies (can be more generic if there is no criteria list) |
| Satisfactory | Met 2 of the given criteria with minor inaccuracies (can be more generic if there is no criteria list) |
| Very Satisfactory | Met 3+ criteria with good technical accuracy (can be more generic if there is no criteria list) |
**Very Unsatisfactory**
- Met 0 of the given criteria ( this can be more generic if there is no criteria list)
**Unsatisfactory**
- Met only 1 of the given criteria with some inaccuracies ( this can be more generic if there is no criteria list)
**Satisfactory**
- Met 2 of the given criteria with minor inaccuracies ( this can be more generic if there is no criteria list)
**Very Satisfactory**
- Met 3+ criteria with good technical accuracy ( this can be more generic if there is no criteria list)
Attribute 2
(repeat structure for Attribute 1 above)
Best Practices
Here are some best practices:
- Use Markdown for clear formatting. Read more about formatting your questions here.
- Define an attribute, such as “AI Understanding”
- Be specific when describing criteria, such as “Demonstrates understanding of Machine Learning fundamentals”
- Prepare a rating scale on a spectrum (e.g. Unsatisfactory to Satisfactory). Please note that numeric values are not supported.
Here is an example of how to define an attribute and list criteria:
6. Add a question label as needed.
7. Once you’re done creating the question, you can click PREVIEW to view your question.
8. Last, click Save. Your question will now be available to use within Assessments.