Generative AI is now a commonplace tool in our work lives. How we adapt to this change, individually and collectively, is top-of-mind for many CodeSignal customers. Cosmo, CodeSignal's AI-powered assistant, enables new ways of evaluating test-takers’ job-relevant technical skills in light of advancements in generative AI technology.
After being enabled for an assessment, Cosmo is available to test-takers within their evaluation. It can clarify directions, answer IDE questions, and help with debugging code. Company users can choose from a variety of Cosmo versions, each optimized for a different use case, which will be applied across an assessment. After a test-taker completes the evaluation, the full conversation is logged in the assessment coding report so you can review the test-taker's interactions with Cosmo.
How a test-taker interacts with Cosmo can provide insight on how they'll leverage cutting-edge tools in their work. Since Cosmo is role-aware, it won't give test-takers the solution to a problem, but it will help them troubleshoot and reason through issues. It also boosts test-taker experience by providing test-takers with assistance during the evaluation without needing to have a team member present.
How do I enable Cosmo?
The first step in enabling Cosmo for your organization is talking to your Customer Success Manager. Since there are several versions of Cosmo available for different use cases, your CSM needs to understand your specific use case so they can suggest the best setup for you.
Once Cosmo is enabled on the account level for your company, you can toggle Cosmo on or off for every assessment where it's available, using the AI Assistance toggle in the edit screen.
Once you have AI Assistance toggled on for that assessment, you'll be able to select which specific version of Cosmo you'd like to use for that evaluation. These different versions are bounded in different ways—for example, some versions are more lenient in how much advice they'll give test-takers, while others are more strict. Again, your CSM can help you choose the best version to fit your needs.
Please note that Cosmo is not available for frameworks at this time.
What does Cosmo look like for test-takers?
For test-takers, Cosmo is embedded directly into the bottom panel of the IDE in the tab labeled Cosmo Chat.
Cosmo Chat is persistent across environments within a question, but not across questions. A test-taker can go back to a previous question to see their previous interactions with Cosmo. Before a test-taker begins to interact with Cosmo, they are informed the conversation will be recorded and viewable by both CodeSignal and your company.
What kinds of interactions can Cosmo have with test-takers?
While Cosmo can process a variety of requests, there are three general categories of questions Cosmo is best equipped to answer. The first is questions about CodeSignal's IDE. A test-taker might ask, "how do I change languages in this environment?" or "how do I submit my answer?" Since Cosmo is trained on the CodeSignal platform, it is able to answer this type of question accurately.
The second category of questions test-takers are introduced to is clarifying questions about the problem itself. For example, a test-taker might ask Cosmo to explain a particular term or what kind of code structure the problem is looking for. Some versions of Cosmo might treat these questions more leniently, offering the test-taker small hints rather than just rephrasing the description..
Lastly, Cosmo is great at understanding when test-takers ask questions about their code. For example, if a test-taker asks Cosmo to explain the compilation error they're getting, Cosmo incorporates data from the test-taker's attempt into its answer so it can provide personalized guidance within the context of the scenario. In these cases, too, one version of Cosmo may have a more or less lenient style in its answer than another version. In any case, guardrails are baked in to ensure Cosmo doesn't give a test-taker the answer to any evaluation questions outright.
Test-takers may feel more secure in their interactions with Cosmo if they know how they'll be expected to interact with it. For example, if you're expecting test-takers to leverage generative AI in their day-to-day work, you might want to actively encourage them to take full advantage of Cosmo for troubleshooting and debugging their code. Since companies approach AI assistant use from a variety of perspectives, test-takers may not know what the expected boundaries are for their interactions with Cosmo.
How do I access a test-taker's Cosmo interaction log?
All of a test-taker's interactions with Cosmo are logged and made available to you, which provides insight into the ways in which a test-taker is leveraging generative AI when coding. The log gives context for whether the test-taker used Cosmo to generate code, ask about the environment, or something else.
You can access the log from the test-taker's coding report. Click into any question and scroll down to the area labeled AI Assistance Transcript.
