Generative AI is now a commonplace tool in our work lives. How we adapt to this change, individually and collectively, is top-of-mind for many CodeSignal customers. Cosmo, CodeSignal's AI-powered assistant, enables new ways of evaluating candidates’ job-relevant technical skills in light of advancements in generative AI technology.
After being enabled for a Pre-Screen, Cosmo is available to candidates within their evaluation. It can clarify directions, answer IDE questions, and help with debugging code. Company users can choose from a variety of Cosmo versions, each optimized for a different use case, which will be applied across a Pre-Screen. After a candidate completes the evaluation, the full conversation is logged in the Pre-Screen coding report so you can review the candidate's interactions with Cosmo.
How a candidate interacts with Cosmo can provide insight on how they'll leverage cutting-edge tools in their work. Since Cosmo is role-aware, it won't give candidates the solution to a problem, but it will help them troubleshoot and reason through issues. It also boosts candidate experience by providing candidates with assistance during the evaluation without needing to have a team member present.
How do I enable Cosmo?
The first step in enabling Cosmo for your organization is talking to your Customer Success Manager. Since there are several versions of Cosmo available for different use cases, your CSM needs to understand your specific use case so they can suggest the best setup for you.
Once Cosmo is enabled on the account level for your company, you can toggle Cosmo on or off for every Pre-Screen where it's available, using the AI Assistance toggle in the edit screen.
Once you have AI Assistance toggled on for that Pre-Screen, you'll be able to select which specific version of Cosmo you'd like to use for that evaluation. These different versions are bounded in different ways—for example, some versions are more lenient in how much advice they'll give candidates, while others are more strict. Again, your CSM can help you choose the best version to fit your needs.
Please note that Cosmo is not available for Certified Evaluations at this time.
What does Cosmo look like for candidates?
For candidates, Cosmo is embedded directly into the bottom panel of the IDE in the tab labeled Cosmo Chat.
Cosmo Chat is persistent across environments within a question, but not across questions. A candidate can go back to a previous question to see their previous interactions with Cosmo. Before a candidate begins to interact with Cosmo, they are informed the conversation will be recorded and viewable by both CodeSignal and your company.
What kinds of interactions can Cosmo have with candidates?
While Cosmo can process a variety of requests, there are three general categories of questions Cosmo is best equipped to answer. The first is questions about CodeSignal's IDE. A candidate might ask, "how do I change languages in this environment?" or "how do I submit my answer?" Since Cosmo is trained on the CodeSignal platform, it is able to answer this type of question accurately.
The second category of questions candidates are introduced to is clarifying questions about the problem itself. For example, a candidate might ask Cosmo to explain a particular term or what kind of code structure the problem is looking for. Some versions of Cosmo might treat these questions more leniently, offering the candidate small hints rather than just rephrasing the description..
Lastly, Cosmo is great at understanding when candidates ask questions about their code. For example, if a candidate asks Cosmo to explain the compilation error they're getting, Cosmo incorporates data from the candidate's attempt into its answer so it can provide personalized guidance within the context of the scenario. In these cases, too, one version of Cosmo may have a more or less lenient style in its answer than another version. In any case, guardrails are baked in to ensure Cosmo doesn't give a candidate the answer to any evaluation questions outright.
Candidates may feel more secure in their interactions with Cosmo if they know how they'll be expected to interact with it. For example, if you're expecting candidates to leverage generative AI in their day-to-day work, you might want to actively encourage them to take full advantage of Cosmo for troubleshooting and debugging their code. Since companies approach AI assistant use from a variety of perspectives, candidates may not know what the expected boundaries are for their interactions with Cosmo.
How do I access a candidate's Cosmo interaction log?
All of a candidate's interactions with Cosmo are logged and made available to you, which provides insight into the ways in which a candidate is leveraging generative AI when coding. The log gives context for whether the candidate used Cosmo to generate code, ask about the environment, or something else.
You can access the log from the candidate's coding report. Click into any question and scroll down to the area labeled AI Assistance Transcript.
If you have any questions about enabling, accessing, or using Cosmo, please reach out to your CSM, or email us at support@codesignal.com.