When using CodeSignal to evaluate candidates, you may want to create a question that uses a filesystem and a terminal to replicate a candidate’s workspace environment in a hyper-realistic manner. This is called an advanced assessment question. With these kinds of questions, interviewers and candidates will have access to the full environment via the terminal, and can install packages, start services, run debuggers, and more.
Jump to:
- Set-up instructions
- Supporting live preview for front-end advanced assessment questions
- Known limitations
Set-up Instructions
1. To create an advanced assessment question, navigate to the Question Library. Then, click Create Question.
2. Select Filesystem from the options that appear, then click Next.
3. On the left hand side of your newly created advanced assessment question, you’ll see a panel showing global settings. You can edit these settings to name and tag your question for easy identification.
4. The next step to creating an advanced assessment is to add an Environment. Each environment you add represents a language that the candidate can choose when they start the question. For example, you might set up environments for JavaScript, Python, and C++ with starter code in each language.
How do environments work?
An environment is a preconfigured Docker container where you and the candidate have full, unrestricted access to the filesystem and terminal. Based on the language you chose, CodeSignal will set up the container by installing commonly used packages and frameworks for that language (e.g. flask and pandas for Python). The comprehensive list of packages can be found in the task’s README file. For additional packages not covered by our default setup, the candidate can use the terminal or run the setup script.
5. Once you’ve added an environment, you’ll see the advanced assessment question UI. This consists of four components:
- An IDE for development (middle)
- A preview screen (right). This lets you preview a web application hosted at port 3000. You can turn this off if you don’t need it using the Enable UI preview panel toggle.
- A filesystem (left). Here, you can add, upload or drag-and-drop any files and code that the candidate might need to get started. Note that using the command line, it’s possible to clone from a GitHub repository if you would like to quickly port a question from GitHub.
- A terminal (bottom). You can use the terminal just like you would on your local machine. It supports multiple terminal windows, which allows candidates to run processes in parallel.
Note:
- You won't be able to upload files weighing more than 100KB, since those files will be excluded later on.
- The size of the whole filesystem can't exceed 1MB (excluding ignored paths).
- You will not be able to upload more than 20 files at a time.
You can also update the question description by clicking the pencil icon under Global Settings.
6. [Optional] You may want to include a script for setting up the environment. As previously mentioned, the environment will include default libraries and frameworks for the language. You can add your scripts to "setup.sh" within the .CodeSignal folder. The files will automatically run when the candidate starts the question or the interview is launched.
7. When you’re finished setting up the template files for your question, and have created an environment for each language you want to support, click Save. (You can also save an in-progress question at any time.)
That’s it – you’ve created an advanced assessment question!
You can follow this process to create an Advanced Assessment question for use in either CodeSignal Interview or CodeSignal Pre-Screen Test.
Support live preview for front-end advanced assessment questions
To support live preview, please make sure to configure additional settings depending on the framework that you have chosen.
- React
Create a .env file in the project's root, and add the following content in this file:
WDS_SOCKET_PORT=0
- Angular
Open the angular.json config, and set the property projects -> angular-app -> architect -> serve -> options -> disableHostCheck to be true
- Vue
No content changes are required for the Vue environment to enable live preview.
Known Limitations
- [Support coming soon] The IDE does not currently support autocompletion for advanced assessment questions.
- While you can run applications in parallel from the terminal, you can only preview one application UI at a time (hosted at port 3000).
- It is not possible to create Docker containers in an advanced assessment question since we currently host each candidate session within a Docker container.
- There is a 2 hour time limit for Advanced Assessment questions in CodeSignal Interview, unless the question is refreshed. We recommend that candidates submit and refresh once an hour to ensure their work is saved and a timeout error is avoided. The time limit in CodeSignal Pre-Screen Test is 8 hours.