In the workplace, developers are frequently required to reuse and refactor their code in order to meet changing requirements. With progressive questions, you can mimic this real-world scenario with multilevel questions that start with basic requirements and gradually increase in complexity, and require candidates to write, test, and refactor code. A candidate’s code and test cases will automatically carry over to the next level so they can continue to build on the work they’ve done previously. 

There are several question formats that can be made progressive:

• Creating progressive free coding questions: covered in the article here
• Creating progressive single function questions: described below; used for defined coding scenarios with test cases that can be automatically scored. Available in CodeSignal Assessments or CodeSignal Interview. Will also support advanced assessment functionality soon, too! 

Single function questions give candidates a defined scenario, where a given output is expected. Candidates can be automatically scored via (visible and/or hidden) test cases that are run against the candidate’s solution. 

When creating each level, you build the question as you would normally for a single function question; with progressive questions you just now have the ability to build on the question with multiple levels. 

How to create a new progressive single function question

To create a progressive free coding question, navigate to the Question Library. Then, click Create Question.

Select free coding from the options that appear, then click Next

When prompted, click the Make Progressive toggle to turn on the feature. The toggle will be green when enabled. 

In the SETTINGS tab, add a title, description, and a label if needed for your question. You can also choose to enable the network access if desired, which allows the candidate’s code to interact with an external API. 

Go to TIMEOUT tab and edit the execution time limit for all languages as needed. 

Click on Input/Output to add descriptions of what the input(s) and expected output, as well as set the input parameters for randomly generating.

Choose how many tests to make visible. Here you have 2 options:

• Show the first half of your test (note that if the number of test cases is odd, we round up). OR
• Manually set the amount of the first visible tests

On the lower right side of the screen, under Tests, click on ADD TEST to add at least one test case with which to evaluate the submitted code.

Though not required, if you’d like to automatically generate test cases and refer to an answer key, you can create an Author Solution for your preferred language. Please note that at this time you can only create an author solution in one language. 

Click on RANDOM TEST to automatically generate test cases based on your Author Solution

When configuring test cases, please be aware that all tests must be passed in order for the candidate to move on to the next level, regardless of whether they’re visible or hidden.

On the upper right side of the screen, under Code, the system generates a starter code snippet that contains the desired function name and the name of the input(s). 

After you’ve set up the instructions, timeout, inputs & outputs and test cases for your first level, it’s time to move on to your next. 

To add a new level, click the Level + button at the top of the settings panel. 

Once you add a new level, you’ll notice that the level label in the upper left corner will say Level 2 and you will be able to edit the description for the level.

You’ll notice that your description from level one has automatically carried over--this is intentional and it is best practice to build your description on top of the language the previous level rather than write an entirely new description each time. Keeping previous descriptions help candidates remember the context from previous levels without needing to click back to a prior level to revisit the instructions. 

However, you can simply delete the text that has carried over if you do not wish to use it.

While you can change the function signature in each level, please note that it will not carry over for the candidate. You should tell the candidate what they need to change in the description. Also, please be aware that your test cases will be impacted and may require revision.

Depending on the instructions you give the candidate in each level, you may want to change or remove test cases from the previous level because they no longer apply. Test cases carry over between levels, but can be made specific to each level if requirements in the new level are entirely different than the previous.

You can repeat the process of clicking the Level + button to add new levels as needed.

Once you’re done creating the question, you can click PREVIEW to view your question in advance of public presentation. Then click Save. Congrats! Your question will now be available for use in CodeSignal Assessment or CodeSignal Interview. 

Reviewing your question

To have peers review your question, you can use Review Mode. Simply navigate to the Question Library, search for your question and select it. 

Upon opening the question, reviewers will see what the candidate sees. 

Reviewers can use the Question Notes tab, as noted by three lines on the left hand navigation,  to leave comments in each level. These notes will not be visible to candidates. 

To progress to the next level, click the Next (with code carry) button in the upper right of the settings panel. To navigate between levels, click the karat icon next to the current level name and a drop down will appear. Click the level you’d like to jump to and you’ll be navigated there directly. Interviewers can navigate both forward and backward. 

Note that hidden test cases will be noted with gray text and visible to reviewers, but they will not be visible to candidates.

Leveraging Progressive Single Function Questions in Assessment or Interview

Visit the article here to learn how you can leverage your progressive question in an interview. 

Visit the article here to learn how to create an assessment.

Notes & Limitations

When a progressive question is created, you can modify existing levels in the question, but will not be able to add or remove levels from the question later.

This is intentional: adding/removing levels from existing progressive questions causes data inconsistency issues, wherein candidates could have different results based on what version of the progressive question they solved.

If you’d like to change the levels of an already existing progressive question, you can duplicate the existing question and create an entirely new question, with levels added/removed.