The confident intern

A brand-new project agent confidently invents a config flag that does not exist; the learner reasons through hallucination, grounding, and what a system prompt is actually for.

Beginner · scenario

Ravi had run backend services for years, but this was his first agent. He gave it one job: keep the `payments-docs` repo tidy — fix broken links, answer questions about the API, and flag stale pages. He wrote a short system prompt, pointed it at the repo, and felt clever.

The first week was charming. The agent found three dead links and drafted a clean changelog. Then a teammate asked it, in the repo chat, how to make the payment client retry on a timeout. The agent answered at once, and precisely: "Set `--retry-forever` in the client config; it will keep retrying until the request succeeds."

There is no `--retry-forever` flag. There never was. The teammate, trusting the confident tone, shipped a config that referenced it, and the deploy failed in staging.

Ravi's first reaction was the human one: the agent lied to him. He nearly wrote a rule into the prompt — "NEVER make things up" — and moved on. But a colleague who had been running agents longer stopped him. "It didn't lie," she said. "Lying needs intent. The model produces the most plausible continuation of the text it is given. You asked a config question, so the most plausible-looking answer is a flag named like the thing you asked about. With nothing in its context grounding the real flags, it filled the gap with something that reads correct. Confident wrongness is not the exception — it is the default when the model is ungrounded."

That reframed the fix. The problem was not the agent's honesty; it was Ravi's context. He had never given the agent the actual list of client flags, so it had nothing to check itself against. He changed three things. First, he put the real CLI reference into the agent's reachable context and told it, in the system prompt, to answer config questions only by quoting that file, with its path. Second, he added a boundary: if the answer is not in the referenced docs, say "I don't find that in the docs" rather than guess. Third, he stopped treating the prompt as a place to scold and started treating it as a program — the phrasing and the examples were what actually steered which answers became likely.

The next config question went differently. The agent quoted `--retry`, linked the exact doc line, and noted the max-attempts default. When a teammate later asked about a flag that genuinely did not exist, the agent said so.

Ravi kept one note for his future self: an agent that sounds certain is not telling you it is correct. It is telling you it found a fluent path through its own predictions. Certainty is the tone, not the evidence.

Glossary

hallucination
The model producing a plausible-sounding answer with no basis in fact. — Not a rare bug: it is the default behaviour whenever the model has nothing grounding the real answer.
confident wrongness
A wrong answer delivered in a fluent, certain tone — the tone reflects fluency, not correctness.
grounding
Putting the real, checkable facts (here, the actual CLI reference) into the agent's reachable context so its answers have something to draw from and quote.
system prompt as a program
The persistent instruction frame is not a place to scold the model; its wording, examples, and boundaries steer which answers become likely. — Adding 'NEVER make things up' supplies no facts, so it changes little; giving the reference and a 'say you do not know' boundary does the work.
boundary
An explicit instruction for the not-found case — say 'I don't find that in the docs' instead of inventing an answer.

Comprehension questions

4 questions — practice and checking happen in the CLI, or sign in on the web to record your results here.

  1. Multiple choice

    Which best explains why the agent produced the non-existent `--retry-forever` flag?

    • It generated the most plausible-looking continuation of an ungrounded question, with nothing in its context to check the real flags against.
    • It was trained on incorrect documentation for this specific payment client.
    • It decided to deceive the teammate so it would look more helpful.
    Record your result:
  2. Multiple choice

    The agent has just handed out a wrong flag. What is the best next move, and why?

    • Add the rule 'NEVER make things up' to the system prompt and carry on.
    • Put the real CLI reference into the agent's reachable context, tell it to answer config questions only by quoting that file, and add a boundary to say when an answer is not in the docs.
    • Stop using an agent for the docs repo; the task is fundamentally unsafe for agents.
    Record your result:
  3. True or false

    A confident tone from an agent is good evidence that its answer is correct.

    Record your result:
  4. Open response

    Apply the lesson to your own first agent. Pick one question it will realistically be asked, name the source you would ground it in, and state the boundary you would set for when the answer is not in that source.

    Record your result:

Practice with the CLI (culture-guide practice), or sign in to check your answers and track a streak.