When using CodeSignal to evaluate candidates, you may want to create a question that uses a filesystem and a terminal to replicate a candidate’s workspace environment in a hyper-realistic manner. This is called an advanced assessment question. With these kinds of questions, interviewers and candidates will have access to the full environment via the terminal, and can install packages, start services, run debuggers, and more.
- Set-up instructions
- Supporting live preview for front-end advanced assessment questions
- Known limitations
1. To create an advanced assessment question, navigate to the Question Library. Then, click Create Question.
2. Select Filesystem from the options that appear, then click Next.
3. On the following screen, click the Enable Terminal toggle (will be green when enabled).
Note that once you have enabled the terminal, you won’t be able to use automated tests. Automated tests are currently unavailable for advanced assessments, but will be supported in a future release.
4. On the left hand side of your newly created advanced assessment question, you’ll see a panel showing global settings. You can edit these settings to name and tag your question for easy identification.
How do environments work?
An environment is a preconfigured Docker container where you and the candidate have full, unrestricted access to the filesystem and terminal. Based on the language you chose, CodeSignal will set up the container by installing commonly used packages and frameworks for that language (e.g. flask and pandas for Python). The comprehensive list of packages can be found in the task’s README file. For additional packages not covered by our default setup, the candidate can use the terminal or run the setup script.
6. Once you’ve added an environment, you’ll see the advanced assessment question UI. This consists of four components:
- An IDE for development (middle)
- A preview screen (right). This lets you preview a web application hosted at port 3000. You can turn this off if you don’t need it using the Enable UI preview panel toggle.
- A filesystem (left). Here, you can add, upload or drag-and-drop any files and code that the candidate might need to get started. Note that using the command line, it’s possible to clone from a GitHub repository if you would like to quickly port a question from GitHub.
- A terminal (bottom). You can use the terminal just like you would on your local machine. It supports multiple terminal windows, which allows candidates to run processes in parallel.
- You won't be able to upload files weighing more than 100KB, since those files will be excluded later on.
- The size of the whole filesystem can't exceed 1MB (excluding ignored paths).
- You will not be able to upload more than 20 files at a time.
You can also update the question description by clicking the pencil icon under Global Settings.
7. [Optional] You may want to include a script for setting up the environment. As previously mentioned, the environment will include default libraries and frameworks for the language. You can add your scripts to "setup.sh" within the .CodeSignal folder. The files will automatically run when the candidate starts the question or the interview is launched.
8. When you’re finished setting up the template files for your question, and have created an environment for each language you want to support, click Save. (You can also save an in-progress question at any time.)
That’s it – you’ve created an advanced assessment question!
You can follow this process to create an Advanced Assessment question for use in either CodeSignal Interview or CodeSignal Pre-Screen Test.
Support live preview for front-end advanced assessment questions
To support live preview, please make sure to configure additional settings depending on the framework that you have chosen.
.env file in the project's root, and add the following content in this file:
angular.json config, and set the property
projects -> angular-app -> architect -> serve -> options -> disableHostCheck to be
No content changes are required for the Vue environment to enable live preview.
- [Support coming soon] Automated scoring is currently unavailable for advanced assessment questions.
- [Support coming soon] It isn’t possible to duplicate an existing advanced assessment question.
- [Support coming soon] It’s not possible to include read-only files in your custom advanced assessment questions, but this is slated for release soon.
- [Support coming soon] The IDE does not currently support autocompletion for advanced assessment questions.
- [Support coming soon] Keystroke recording is not currently available for advanced assessment questions. Once the assessment is completed, the final submission is stored and state of the environment and code is captured for your review. You can also choose to download a PDF coding report for sharing.
- While you can run applications in parallel from the terminal, you can only preview one application UI at a time (hosted at port 3000).
- It is not possible to create Docker containers in an advanced assessment question since we currently host each candidate session within a Docker container.
- There is a 2 hour time limit for Advanced Assessment questions in CodeSignal Interview, unless the question is refreshed. We recommend that candidates submit and refresh once an hour to ensure their work is saved and a timeout error is avoided. The time limit in CodeSignal Pre-Screen Test is 8 hours.