Testing: sandbox and scenarios
Put your agent through real cases, first one-on-one, then across a battery of scored scenarios, before a single customer talks to it.
An agent should never discover a hard case in production. Magnitude gives you two complementary testing grounds, both inside the agent editor: the sandbox, to talk to it yourself right now, and scenarios, a saved battery of tests that an impartial judge grades for you. Both replay the exact brain that goes live, so what you see here is what the customer will experience.
The sandbox
On the right of the editor, a test panel stays open at all times. You type in it just as a customer would, and the agent replies with the brain you are currently tuning, sending nothing and saving nothing. It is the fastest way to feel its tone, its accuracy and its limits before you touch anything.
For a website agent, the panel shows the real widget, the very one your visitors will see, always open and styled live by the appearance settings you edit on the left. For a social agent, it is a test conversation that, alongside the reply, surfaces the agent's silent actions as pills: a lead moving to a new stage, estimated value, priority, detected mood, resolution, handoff to a human. That way you see, at the same time as the reply, how your customer tracking is going to move.
Under each reply, the preview also shows you the sources: the documents from your knowledge base that were used to answer. At a glance you confirm the agent is relying on the right information, and not on something it made up.
The documents cited under a reply appear in the preview and for your team, never for the customer. The visitor only sees the reply.
A "Tool log" button opens the full detail of everything the agent did to answer: every search across your knowledge, every query against your data, every tool called, with its parameters and its result. The "Reset" button starts from a blank conversation. Since the sandbox reads your in-progress edits, you can adjust the instructions and retest right away, without publishing.
The Scenarios tab
The sandbox tests one conversation at a time. The "Scenarios" tab saves a battery of cases that you replay on demand, as many times as you want. Each scenario fits on one line: a customer message, and the expected reply. For a social agent, you can also check "Must hand off to a human" when the right behavior is to pass the baton rather than answer.
These are single-turn tests: one message, one reply. That choice is deliberate. By isolating one behavior per scenario, you know precisely what passed or failed, without a long exchange muddying the diagnosis.
The list shows, for each test, the question, the expected reply and a status pill. Expand a row to read the expected reply, correct it, validate it, or rerun that test on its own.
Generate tests from your real conversations
You don't write this battery by hand. Click "Generate" (or "Regenerate" if tests already exist) and choose the source. Two options: "Your real conversations", recommended, and "The agent's instructions", useful for a brand-new agent with no history.
With the first, the agent reads your recent customer messages, extracts the distinct requests actually asked, ignores the noise ("thanks", "ok", standalone greetings, empty messages), groups the variants, then composes a varied, de-duplicated selection that covers your real cases, from the most frequent to the rarest. Each retained question is then replayed through your real agent, so the "Expected reply" is pre-filled with what it answers today.
It's up to you to review. Generated tests arrive marked "to verify": you confirm the reply when it's good, you correct it when it isn't. A reply that you correct and that the agent doesn't yet know how to reproduce becomes a red test, until you adjust its instructions or its knowledge base. That is exactly the signal you're looking for.
Generation runs in the background with a progress bar: you can leave the page, it keeps going, and the tests are saved as they come. "Regenerate" only touches the generated tests you haven't reviewed yet; your manual tests and every already-validated reply are kept.
A brand-new agent has nothing to draw on. In that case choose "The agent's instructions": the battery is synthesized from its brain. Neither writing the tests nor grading them eats into your message quota.
How a test is graded
Run a single test with "Retest", or the whole battery with "Test all". In both cases, the question is replayed through the agent's real brain, with no side effect whatsoever: nothing is sent to the customer, nothing is saved, no handoff actually happens.
When an expected reply is defined, an impartial judge compares the produced reply to yours by meaning, never by wording. It is lenient on form and strict on facts. In parallel, the handoff expectation (for social agents) is checked exactly: did the agent pass the baton, yes or no. Both must pass for the row to turn green.
| Verdict | When |
|---|---|
| Passed | The reply conveys the same facts and the same intent as the expected reply. Tone, order, length and formatting are ignored. Correct extra information is accepted. |
| Failed | A fact differs or is contradicted (price, schedule, condition), an expected piece of information is missing, the agent invents or promises wrongly, it dodges when it should answer, or it hands off when it shouldn't have (or the reverse). |
When a test fails, the row shows the reply actually produced, a one-sentence justification, and a concrete suggestion: what to spell out in the instructions, or the information to add to the knowledge base. If you judge the reply acceptable despite the verdict, "Mark this test as passed" gives your judgment the final say.
Test everything before you publish
"Test all" replays the entire battery, one line after another, each turning green or red live. Make a habit of running it before every publish: a change that fixes one case must not silently break another. An all-green battery is your green light.
When everything passes, you can publish the agent with confidence. Every published version is archived: you test, you publish, and you keep a record of what was live.
Writing good scenarios
Whether the tests are generated or added by hand, a few habits make them useful:
- Stay natural: write the message the way a customer would really type it, typos and shortcuts included.
- One expectation per test: one message, one behavior to check. It's clearer than a long exchange.
- A factual expected reply: list the facts that must appear, not a word-for-word text. The judge grades meaning, not style.
- Cover the range: don't test the same question ten times, aim for distinct topics.
An agent holds up very well on easy questions. It's the edge cases that give it away: a price limit or a discount, a refund or a cancellation, an off-topic question, a visibly angry customer, and anything that absolutely must go to a human. Add a scenario for each: that's where trust is won.