General Coding Framework

April 2024

Understanding Cut Scores in Pre-Hire Assessments

A cut score is a numerical threshold used to distinguish between candidates who meet the necessary qualifications for a role and those who do not. For example, those who score at or above 60 out of 100 on an exam pass through to the next stage. In the context of pre-hire assessments, a cut score is a set bar that candidates must meet to be considered for the next stage of the hiring process.

There are many organizational goals and interests to consider when it comes to setting a cut score. A well-set cut score should ensure that the selection process is fair, objective, and clearly aligned with role requirements. It provides a clear benchmark for all candidates, standardizing the selection process and giving all candidates a fair opportunity to proceed to the next stage.

This is a step-by-step guide to determining an appropriate cut score for your hiring goals.

Step 1: Determining Framework fit

Step 2: Defining minimally acceptable

Step 3: Benchmarking

Step 4: Narrowing your candidate pool (Optional)

Step 5: Calibration

Step 1: Determining Framework fit

The first step, before we consider what cut score is appropriate, is to confirm that the skills measured in the Skills Evaluation Framework, or assessment, fit the requirements for the role. There are several methods for doing this, each with its own benefits and drawbacks. 

Identifying Job Requirements

Documentation

The first course of action is to review existing job description, job analysis documents, and other information and documentation on the role and the context in which the role is performed to identify core knowledge, skills, abilities, and other characteristics required for the role. Next, it is important to map the skills required for the role to those measured in the Skills Evaluation Framework. 

For the General Coding Framework, you can review the General Coding Skills Evaluation Framework Technical Brief and note whether the skills measured are a good fit for those required on-the-job. When implementing a pre-screen assessment, you want the assessment to provide a good representation of the minimum requirements for the role. Ask yourself, “What would a minimally component employee be required to do or know in this role?”

Oftentimes, skills are not explicitly mentioned in job descriptions and other documentation available, but work activities, or the role responsibilities, are listed. Because of this, it is best if a subject matter expert is able to review the documentation for the role and the Framework, or assessment, to make sure that the skills underlying the work activities mentioned in the job description are covered by the framework. 

The framework does not need to be a perfect match, but it should be clearly relevant to the role you are hiring for. 

Documentation + Discussion

Another step that goes a long way in establishing the relevance and validity of the framework for the hiring context is to hold a 45 minute focus group with 3-5 job incumbents, those currently in the role being hired for. This process is much more thorough than reviewing job descriptions alone and can help establish the most crucial requirements for the role, so you can map those to the assessment.

Below are some example questions you can implement in a focus group discussion:

Focus Group Sample Questions

If combining your focus group findings with those from the Documentation step, consolidate your takeaways and compare your findings with what is actually measured in the framework by reviewing the General Coding Skills Evaluation Framework Technical Brief and sample General Coding Framework Skills Evaluations.

Step 2: Defining minimally acceptable

Now that you know the core knowledge, skills, abilities, and other characteristics required for success on the job and you’ve determined the framework to be a good fit, you’re ready to determine an appropriate cut score for the role. As described earlier, a cut score defines a numerical point where candidates scoring at or above the threshold will pass on to the next stage in the selection process. Cut scores are crucial for consistency and fairness in the hiring process, promoting job relevance, precision in prediction of on-the-job performance, a positive candidate experience, and legal defensibility. 

Although we would all like to hire the best candidate for the role, it is important to start by determining what a minimally acceptable candidate would look like. Setting a cut score based on what a minimally acceptable candidate would look like will help ensure fairness and equity at each stage in the hiring process. If we set the bar too high too early, we may be cutting off some great candidates from demonstrating their full potential at later stages.

Below is the General Coding Framework Score Guide, which can be used to provide a general idea of what a candidate’s skills would look like at different score levels. 

General Coding Framework Score Guide

*Note: These are estimated thresholds; actual candidate skill profiles will vary depending on individual performance factors. 

Step 3: Benchmarking

Now that you’ve established the job-relevance of the Framework and determined what a minimally acceptable candidate for your role would look like, you are ready to consider how industry benchmarks can be applied to maximize the efficiency of your hiring process. 

When looking at benchmarks for candidate pass rates, the rate in which candidates score at or above a given score threshold, it is important to balance your needs to streamline the hiring process with your goals of promoting equity and diversity in the hiring funnel. 

The guide below shows the pass rates across all roles, regions, and industries for the General Coding Framework. As you can see, raising the cut score results in a smaller proportion of candidates moving on to the next stage in the hiring process. 

General Coding Framework Pass Rates

*Note: This is based on a sample of over 140,000 General Coding Framework attempts by candidates applying for a variety of roles and organizations between ​​January 1st, 2023 and Apr 1st, 2024.

While it can be tempting to choose a very high cut score to help screen out candidates who are underqualified, it’s important to think about what score would suggest a candidate has the minimum qualifications to move on to the next stage in the hiring process so you are not setting the bar too high and screening out qualified candidates. Those candidates may be able to demonstrate further skills and characteristics in later stages of the process.

It is also important to consider that, as the cut score is raised, group differences in terms of race, ethnicity, and gender are more likely to be observed. Adverse impact refers to a form of unintentional discrimination that occurs when an employment practice, such as hiring, promotion, or training, results in a significantly more negative outcome for a particular group. The most commonly used rule for this statistical measurement is the "Four-Fifths Rule" or "80% Rule." According to this rule, adverse impact is indicated if the selection rate for any group is less than 80% of the selection rate for the group with the highest rate. 

You should weigh out your goals of managing candidate volume with goals of fairness and equity to help you determine an appropriate cut score. 

Step 4: Narrowing your candidate pool (Optional)

Below are some strategies for narrowing your candidate pool if it is still too large after implementing your cut score. These strategies range from 1) the most fair in terms of demographic representation to 3) the most data driven in terms of assessment score.

Strategies for Further Narrowing Your Candidate Pool

Consider your particular hiring goals in determining whether one of these strategies is a fit for you and your organization.

Step 5: Calibration

Now that you’ve begun to administer Framework to your candidates, you have an opportunity to optimize your cut score selection based on funnel & interview metrics. We recommend that, once you have seen a sufficient number of candidates go through the hiring process, you examine how candidate performance on the assessment is relating to hiring outcomes to inform cut score adjustments. Some outcomes to evaluate include interview ratings, other post-assessment hurdle performance metrics, interview outcomes, whether a hiring offer is made, group differences, and on-the-job technical performance.

If your cut score does require adjustment, we recommend that you wait to make any changes until all roles hiring under the current cut score are closed and you are able to start fresh with a new candidate pool. This helps ensure equal opportunity in the hiring process for each role.
If you are interested in establishing the relationship between candidate performance on the General Coding Framework with on-the-job performance, CodeSignal’s Talent Science team of industrial-organizational psychologists is available to partner with you. If you choose to partner, our team will run analyses to determine how strongly performance on the pre-hire assessment is related to on-the-job performance, using a special performance survey designed to specifically tap into the skills measured in the General Coding Framework or your in-house performance data.  

FAQs

Below are responses to some of the most frequently asked questions related to setting a cut score. 

How would you recommend setting a scoring cut-off threshold that minimizes group differences?

Setting an optimal cut score is a unique process for each organization. The cut score used depends on a number of factors, including the framework being utilized, the hiring goals of the organization, the unique characteristics of the candidate pool, the core skills required for the role, where the assessment is implemented in the hiring process, and more. As a starting point, CodeSignal’s Talent Science team recommends choosing a cut score on the lower end of the identified range, which will help maximize diversity in the candidate pool. From there, the cut score can be adjusted up in an iterative process to account for other factors.

Should I have different cut scores for different roles?

It’s important to consider the minimum requirements for each of the roles you are using an assessment to hire for when setting cut scores. It is often best to set a unique cut score for each role, especially as you begin to calibrate your cut scores for your unique applicant pool and hiring needs. This is particularly important when considering roles of differing levels. For example, a lower cut score may make sense for an intern group, where many skills are expected to be developed on-the-job. In contrast, a higher cut score may make sense for a more advanced role, given the higher expectations and more complex minimum requirements identified for the role. 

Should I have different cut scores for different geographic regions?

As with different roles, it is important to consider your unique hiring goals for each region in which you are hiring. Having a standard cut score across regions may result in unintentional inequity, as different regions may have different considerations in terms of overall and specific demographic group differences in technical skills & knowledge and a differing legal landscape. Additionally, there may be language & technology barriers that lead to differences in candidate performance on a Framework by region. The candidate pools for different regions may look very different, and it is important to tailor your cut score to your hiring goals, considering the unique regional factors. However, when hiring for a single role, it is important that the cut score is the same across candidates for that specific role. 

Can I change the cut score if I’m not seeing the pass rates I want to see?

It is strongly recommended that you do not adjust the cut score midway through the hiring process to promote fairness in the hiring process. This is why it is essential to consider your cut score thoroughly before implementing it, taking into account your hiring goals, demographic considerations, job-relevance, and the minimally acceptable level of skills required for success in the role. 

How can I improve the legal defensibility of my candidate selection practices?

To maximize legal defensibility, it is crucial to establish the job-relatedness of the Framework, or assessment, through collecting insights from job experts or collecting data on job performance. The more rigorous your approach, the more defensible the use of a framework and cut score for your hiring process is. For example, if a physical ability test required for a job disproportionately disqualifies women more than men, and the test does not measure skills that are important for success in the role, it might be considered to have an adverse impact on women. You should also make sure to retain documentation of your processes to refer back to in the future. CodeSignal’s Talent Science team of trained industrial-organizational psychologists has applied their expertise in establishing validity, reliability, and fairness in the hiring process to assist in the development, monitoring, and evaluation of our Skills Evaluation Frameworks, and they are available to talk through establishing legal defensibility for your selection system and related topics.