When using CodeSignal to evaluate candidates, you may want to create a question that uses a filesystem and a terminal to replicate a candidate’s workspace environment in a hyper-realistic manner. This is called an advanced assessment question. With these kinds of questions, interviewers and candidates will have access to the full environment via the terminal, and can install packages, start services, run debuggers, and more.
1. To create an advanced assessment question, navigate to the Question Library. Then, click Create Question.
2. Select Filesystem from the options that appear, then click Next.
3. On the following screen, click the Enable Terminal toggle (will be green when enabled).
Note that once you have enabled the terminal, you won’t be able to use automated tests. Automated tests are currently unavailable for advanced assessments, but will be supported in a future release.
4. On the left hand side of your newly created advanced assessment question, you’ll see a panel showing global settings. You can edit these settings to name and tag your question for easy identification.
How do environments work?
An environment is a preconfigured Docker container where you and the candidate have full, unrestricted access to the filesystem and terminal. Based on the language you chose, CodeSignal will set up the container by installing commonly used packages and frameworks for that language (e.g. flask and pandas for Python). The comprehensive list of packages can be found in the task’s README file. For additional packages not covered by our default setup, the candidate can use the terminal or run the setup script.
6. Once you’ve added an environment, you’ll see the advanced assessment question UI. This consists of four components:
- An IDE for development (middle)
- A preview screen (right). This lets you preview a web application hosted at port 3000. You can turn this off if you don’t need it using the Enable UI preview panel toggle.
- A filesystem (left). Here, you can add any files and code that the candidate might need to get started. Note that using the command line, it’s possible to clone from a GitHub repository if you would like to quickly port a question from GitHub.
- A terminal (bottom). You can use the terminal just like you would on your local machine. It supports multiple terminal windows, which allows candidates to run processes in parallel.
Note that you can also update the question description by clicking the pencil icon under Global Settings.
6. [Optional] You may want to include a script for setting up the environment. As previously mentioned, the environment will include default libraries and frameworks for the language. If you need to set up the environment in a more specific way, we recommend that you provide a script (e.g. “setup.sh”) that the candidate can run when they start the task.
Explain in the question overview that this script should be run from the command line to set up the environment correctly. In general, we would recommend you to ask candidates to run this as the first instruction, so setup can happen in the background while the candidate is reading through the rest of the instructions.
7. When you’re finished setting up the template files for your question, and have created an environment for each language you want to support, click Save. (You can also save an in-progress question at any time.)
That’s it – you’ve created an advanced assessment question!
You can follow this process to create an Advanced Assessment question for use in either CodeSignal Interview or CodeSignal Pre-Screen Test.
- [Support coming soon] Automated scoring is currently unavailable for advanced assessment questions.
- [Support coming soon] It isn’t possible to duplicate an existing advanced assessment question.
- [Support coming soon] It’s not possible to set files in the filesystem as read-only, but this is slated for release soon.
- [Support coming soon] The IDE does not currently support autocompletion for advanced assessment questions.
- [Support coming soon] Keystroke recording is not currently available for advanced assessment questions. Once the assessment is completed, the final submission is stored and state of the environment and code is captured for your review. You can also choose to download a PDF coding report for sharing.
- While you can run applications in parallel from the terminal, you can only preview one application UI at a time (hosted at port 3000).
- It is not possible to create Docker containers in an advanced assessment question since we currently host each candidate session within a Docker container.
- There is a 2 hour time limit for Advanced Assessment questions, unless the question is refreshed. We recommend that candidates submit and refresh once an hour to ensure their work is saved and a timeout error is avoided.