In the workplace, developers are frequently required to reuse and refactor their code in order to meet changing requirements. With progressive questions, you can mimic this real-world scenario with multilevel questions that start with basic requirements and gradually increase in complexity, and require candidates to write, test, and refactor code. A candidate’s code will automatically carry over to the next level so they can continue to build on the work they’ve done previously. 

There are several question formats that can be made progressive:

• Creating progressive free coding questions: covered below
• Creating progressive single function questions: covered in the article here

How to create a new progressive free coding question

Free coding questions allow the candidate to do just that- code freely. The candidate has access to a terminal, which allows them to install packages and engage in a “playground” environment. In the progressive format, you can build a free coding question with multiple levels.

Please note that there are no test cases in free coding questions, which means the questions are not automatically scored, and will require human review. For that reason, progressive free coding questions are available in CodeSignal Interview only. 

To create a progressive free coding question, navigate to the Question Library. Then, click Create Question.

Select free coding from the options that appear, then click Create

When prompted, click the Make Progressive toggle to turn on the feature. The toggle will be green when enabled. 

On the left-hand side of your newly created question, the global settings tab will be open, allowing you to name and label the question. You can also choose to enable the network access if desired, which allows the candidate’s code to interact with an external API. 

Toggle to the Languages & Code tab at the top of the global settings menu to select which languages you’d like candidates to be able to code in. 

By default, all languages will be included. To limit the languages available to candidates, click the limit the languages available toggle. Click languages you’d like to exclude, then transfer those languages to the “excluded” column using the arrow button in the middle of the panel.

When you click an available (included) language you can write starter code in that language using the custom starter code panel to the right. Note that the starter code will not be automatically converted to other languages, so you’ll need to add starter code for all of the languages you want to provide starter code for. 

These global settings will apply to all levels in your progressive question. Next, you’ll want to edit the level settings for your first level. Click the icon with a grid and pencil on the top left side of your screen, just beneath the global settings tab. 

In the description box, you’ll want to add instructions for the first level of the question. 

You can also override the default amount of time a candidate’s code has to execute before an error is returned using the Timeout tab at the top of the level settings box. Some users like to change the timeout settings to better understand how efficient a candidate’s code is. 

The preview tab gives you a glimpse at what candidates will see when completing the question.

After you’ve set up the instructions, timeout, and starter code for your first level, it’s time to move on to your next. 

To add a new level, click the Level + button at the top of the settings panel. 

Once you add a new level, you’ll notice that the level label in the upper left corner will say Level 2 and you will be able to edit the description for the level.

You’ll notice that your description from level one has automatically carried over--this is intentional and it is best practice to build your description on top of the language the previous level rather than write an entirely new description each time. Keeping previous descriptions help candidates remember the context from previous levels without needing to click back to a prior level to revisit the instructions. 

However, you can simply delete the text that has carried over if you do not wish to use it.

It’s important to note that you cannot edit the starter code for any level except level one. The candidate’s code from previous levels will carry over to the next, and will serve as the starter code for subsequent levels. In other words, the final code state from level 1 will serve as starter for level 2, and the final state from level 2 will serve as the starter for level 3, and so on. 

You can repeat the process of clicking the Level + button to add new levels, and draft new descriptions. 

When you’re finished creating your progressive question, you can press Save to publish the question. 

Reviewing your question

To have peers review your question, you can use Review Mode. Simply navigate to the Question Library, search for your question and select it. 

Upon opening the question, reviewers will see what the candidate sees: the question description, sample code, and terminal. 

Reviewers can use the Question Notes tab, as noted by three lines on the left hand navigation,  to leave comments in each level. These notes will not be visible to candidates. 

To progress to the next level, click the Next (with code carry) button in the upper right of the settings panel. To navigate between levels, click the karat icon next to the current level name and a drop down will appear. Click the level you’d like to jump to and you’ll be navigated there directly. Interviewers can navigate both forward and backward. 

Leveraging Progressive Questions in an Interview

Visit the article here to learn how you can leverage your new progressive question in an interview. 

Notes & Limitations

When a progressive question is created, you can modify existing levels in the question, but will not be able to add or remove levels from the question later.

This is intentional: adding/removing levels from existing progressive questions causes data inconsistency issues, wherein candidates could have different results based on what version of the progressive question they solved.

If you’d like to change the levels of an already existing progressive question, you can duplicate the existing question and create an entirely new question, with levels added/removed.