Cheating and plagiarism are common concerns for hiring teams. Companies want to make sure they can trust that a candidate’s work represents their own skills, not someone else's. To make identifying and addressing these concerns easier, CodeSignal's Suspicion Score identifies possible code integrity incidents automatically.
We flag potentially suspicious activity from a variety of sources and synthesize those data points to assign a trust level for each question submission. With the appropriate context, you can decide whether flagged incidents need further review. Suspicion Score is a good starting point for further conversations with your team about integrity in the hiring process.
Skip to a section:
- Where do I find Suspicion Score?
- What is Integrity Review and what does it tell me?
- What do I do if an Integrity Review is suggested?
- What kinds of decisions can Suspicion Score help me make?
Where do I find Suspicion Score?
Suspicion Score lives in a candidate's Coding Report. From any Pre-Screen results list, click on a candidate's name to bring up their Coding Report. There, you will see a Pre-Screen Signals section which includes an item called “Integrity Review Suggested.” Integrity Review Suggested flags candidate attempts that you may want to review by looking at the Suspicion Score for each question.
Scrolling down to the Questions section of the Coding Report, you'll see individual Suspicion Scores for each question. These scores offer a more in-depth look at where a candidate's behavior may require additional review.
What is Integrity Review and what does it tell me?
Integrity Review Suggested consolidates flags from all aspects of a candidate's submission to help you identify results you might want to investigate.
There are four possible values for Integrity Review Suggested:
- Yes
- No
- N/A
- Pending...
"Yes" does not mean a candidate definitely cheated, but it means there's enough suspicious activity that you should consider looking into that submission more closely. "No" means there weren't enough red flags in a candidate's behavior to necessitate further review. "N/A" means Integrity Review Suggested was not run for that submission (e.g., question type or framework is not supported), or that there was an internal error. Lastly, "Pending..." is a placeholder while Integrity Review Suggested finishes running; if you see this status, check back soon for an update.
What do I do if an Integrity Review is suggested?
If an Integrity Review is suggested, the recommended first step is to look at the Suspicion Scores for each question. These scores, which range from "None," for no suspicious behavior, to "High," for a large amount of suspicious behavior, can help you pinpoint where possible misconduct occurred. We recommend looking into the score breakdown for each question with a Suspicion Score other than "None."
Suspicion Scores draw from a variety of data, so they can describe multiple kinds of suspicious behavior. The drop-down next to each Suspicion Score contains more information about some of the variables that went into that particular score.
This is a sample of some of the available flags:
The flags that make up Suspicion Scores can alert you to situations where it looks like a candidate might have received external assistance. For each type of suspicious event, there is an action item linked in blue (in the screenshot above, it is "View Diff"). These buttons bring up more information to help give context for why CodeSignal flagged these particular instances.
These flags could mean the candidate was copying off another candidate's answers, looking at leaked materials, or using tools like generative AI to come up with code. Because Suspicion Score uses data from across the millions of interviews and evaluations conducted on the CodeSignal platform, we can analyze every solution to identify multiple types of suspicious behavior patterns.
Here are our recommendations for next steps to take based on the Suspicion Score flags for each question:
- If a question has a "High" or "Medium" Suspicion Score and 0 Suspicious Paste Events, look more closely into the similarity diff. Click the "View Diff" button under the Similarity Score. View Diff can be used to understand if the solution was copied or if the solution is rather simple, and thus any reasonable solution would also be similar when doing the similarity analysis.
- If a question has one or more Suspicious Paste Events, look more closely into the replay.
- Check the sizes and the number of the paste events. Larger chunks and/or more frequent pastes are more suspicious.
- Check the position of the paste events in the replay timeline. Pasting after having no activity in the replay for a while, and/or after a Description Copy event is a common cheating pattern.
- Check the submissions in relation to Paste events. Multiple repeat pastes followed by submissions resulting in errors is a common cheating pattern.
- Check for language switches. Switching the language combined with paste events is also a highly suspicious behavior pattern.
- If a question has a "Low" Suspicion Score and a "Yes" for Description Copy, do the following:
- Look into the size of the copy event. The higher the count of copied characters, the more suspicious the behavior is.
- Look into the position of the Description Copy in the replay timeline. It’s more suspicious when the description is copied in the beginning of the replay and at the end, when the time is running out.
- Check the replay to consider if the candidate may have retyped the solution. If the description was copied right at the beginning of the replay, and then after some time the candidate is starting to type, that can be indicative of retyping a solution.
Depending on the context of how you're using the evaluation in question, Suspicion Score may flag behaviors that your company doesn't find suspicious or unethical. Therefore, it's important to understand the context surrounding individual Suspicion Scores. If you're ever in doubt, your Customer Success Manager can help talk through what different Suspicion Scores might mean for candidates at your company.
What kinds of decisions can Suspicion Score help me make?
Suspicion Score is a remarkably flexible feature because it doesn't just lead to one "right" answer. Instead, it's a collection of data that can be used to make determinations in the context of a specific evaluation.
There are some kinds of decisions that Suspicion Score cannot help you make. For example, you shouldn't treat a high Suspicion Score like a guarantee that a candidate was cheating. Innocent behaviors can sometimes look suspicious, and different companies have different perspectives on the use of outside tools.
What Suspicion Score can do is nudge you toward a good first place to look for behaviors that might threaten the integrity of your results. It can help narrow your focus to specific instances of suspicious behavior, making the most of your hiring team's valuable time. Suspicion Score can also spark conversations about what integrity means in your specific context, and which behaviors are off the table during an evaluation.
Still have questions? Reach out to your Customer Success Manager or email us at support@codesignal.com.