HeyMarvin

How to launch an AI agent that talks to real people

An AI agent that runs live conversations with nobody watching can't be judged by a demo. Here is how I took one from first build to paying customers.

  • AI agents
  • Agent configuration
  • Dogfooding
  • Failure taxonomy
  • Staged rollout

The big picture

Give users control, use the product yourself, release in stages, and let customers set the terms of trust.

Why it matters

When no one is watching, failures are quiet. An interview that ends too early looks exactly like one that finished.

By the numbers

1–2 min
Interviews meant to run long were ending in one to two minutes. We caught it by using the product ourselves, before customers did.
Beta
Large enterprise customers joined the beta.
[Placeholder]
How many people took part in the company-wide testing, and how many interviews they ran.
[Placeholder]
Any adoption or interview-length numbers after the changes, only if you are sure of them.

The product

A researcher writes a guide covering what the study is about, how the conversation should go, which questions to ask and how to end it. The agent then runs the interviews. A few settings steer how it behaves, including how deeply it probes.

Step 1. Give users the dial, not only the output.

Researchers know which questions need depth. A name needs one answer. An open-ended question needs a few rounds of why. Low probing meant the agent accepted the answer and moved on. High probing meant it kept asking why until it reached the real reason. The cost: more setup for the researcher. We could have let the agent decide, and we chose not to in the first version.

Step 2. Use it yourself, on real work.

We build a research tool, so we used our own. Whenever someone needed answers from many people, they sent out an AI interview. Three kinds of problems showed up.

  • Depth It accepted the first answer and moved on.
  • Timing It cut people off when they paused mid-answer, then moved on or repeated the question. There was also an awkward gap before it replied.
  • Feel It felt disconnected and too good to be true. The emotional depth was missing.

We wrote down everything that felt off and shared it with the team, and we looked for fixes even when none was possible yet. Models get better every day, so knowing what we wanted was one question and how to get there was another.

Step 3. Change the control when real use shows it's wrong.

We first set one probing level for the whole interview. Testing showed some questions needed far more probing than others, so we let researchers set it per question. The cost: a longer setup, in exchange for interviews that go as deep as the study needs.

Step 4. Release in stages and keep talking to customers.

We gave access to every interested company, labelled it beta, and held constant conversations about what worked, what did not and what they wanted. The cost: early customers saw an unfinished product.

Step 5. Let customers set the terms of trust.

A large customer was keen but wanted a visible note that participants were talking to an AI and that it was in beta. We let each company add its own banner. The cost: the wording is theirs to get right.

Yes, but

  • There was no numeric pass/fail bar. The gate was testing on ourselves, then what beta customers told us.
  • [Placeholder: what was done about interruptions and the response gap, and whether it was fixed before public launch]

What this shows

  • I define the controls people use to steer an AI, find where it fails by using it, and release in stages instead of waiting for a perfect score.

What's next

Let the agent decide how much to probe, which we left out of the first version, and make the voice feel less disconnected as models improve.