A/B Experiments

Overview

An experiment compares variants of one voice agent on a single metric. Each variant is an arm: a bot version, or a set of overrides (system prompt, voice, model, timing). Exactly one arm is the control.

traffic_split decides what share of callers each arm gets. POST /assign buckets a caller into an arm deterministically, and GET /results reports the metric per arm.

Nothing on this page consumes credits — the calls the experiment configures are billed as normal voice usage.

Authentication

Authorization: Bearer cm_your_api_key

assign needs the write scope — it records a durable assignment.

Lifecycle

draft ──start──▶ running ──stop──▶ stopped ──conclude──▶ concluded
                     │                                      ▲
                     └──────────────conclude────────────────┘

Stop before editing an arm; conclude only when you are done for good.

Metrics

The deciding metric can only be changed while the experiment is a draft — picking the winner after seeing the data is exactly what that rule prevents.

The experiment object

{
  "id": "ex10…",
  "tenant_id": "a0b1…",
  "bot_id": "b1f2…",
  "name": "Shorter opening line",
  "hypothesis": "A one-sentence greeting raises goal completion.",
  "status": "running",
  "traffic_split": { "arm-a-id": 50, "arm-b-id": 50 },
  "metric": "goal_completed",
  "winner_arm_id": null,
  "started_at": "2026-08-14T09:00:00Z",
  "stopped_at": null,
  "created_at": "2026-08-13T09:00:00Z",
  "updated_at": "2026-08-14T09:00:00Z",
  "arms": []
}

GET and every mutating call return the detail shape, with arms populated oldest first.

GET /api/v1/voice/experiments

Newest first. Filters: bot_id, status. limit 1..200 (default 50), offset 0..100000.

POST /api/v1/voice/experiments

Created as draft with an empty traffic_split. Returns 201.

Arms

{
  "id": "arm-b-id",
  "tenant_id": "a0b1…",
  "experiment_id": "ex10…",
  "name": "short-greeting",
  "bot_version_number": 12,
  "overrides": {
    "system_prompt": "Greet in one sentence, then ask how you can help.",
    "voice": "anushka",
    "timing": { "interrupt_sensitivity": 0.6, "silence_timeout_ms": 2000 }
  },
  "is_control": false,
  "created_at": "2026-08-13T09:05:00Z"
}

POST /api/v1/voice/experiments/{experiment_id}/arms

At most 6 arms per experiment.

overrides

Unknown keys are rejected with 422 rather than ignored.

Timing fields

PATCH / DELETE /api/v1/voice/experiments/arms/{arm_id}

Arms are addressed directly, not under their experiment. While the experiment is running you may change only name — anything else returns 409 Stop the experiment before changing an arm's configuration, because a mid-flight change would silently mix two configurations into one arm's numbers.

Deleting an arm also removes it from traffic_split in the same transaction.

Traffic split

traffic_split maps every arm id to a whole-number percentage.

Set it with PATCH /{experiment_id}, or pass it on start.

Start, stop, conclude

POST /api/v1/voice/experiments/{experiment_id}/start

Optional body { "traffic_split": { … } }; falls back to the stored split. Needs at least two arms — 422 An experiment needs at least two arms to compare.

POST /api/v1/voice/experiments/{experiment_id}/stop

No body. 409 This experiment is not running if it was not.

POST /api/v1/voice/experiments/{experiment_id}/conclude

After this the experiment is immutable. A draft cannot be concluded — 409 Start the experiment before concluding it.

Assignment

POST /api/v1/voice/experiments/{experiment_id}/assign

Send at least one. When both are present the session id is the bucketing key.

curl -X POST https://api.callmissed.com/api/v1/voice/experiments/ex10…/assign \
  -H "Authorization: Bearer cm_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{ "voice_session_id": "vs99…" }'

{
  "experiment_id": "ex10…",
  "arm_id": "arm-b-id",
  "arm_name": "short-greeting",
  "voice_session_id": "vs99…",
  "assignment_id": "as55…",
  "created": true,
  "assigned_at": "2026-08-17T08:20:00Z"
}

Bucketing is deterministic — the same key always lands in the same arm for the same split, so a returning caller keeps their variant.

Assignment by session is idempotent: a repeat call returns the existing row with created: false, and a concurrent double-call re-reads the winner rather than creating two.

A key-only call writes nothing. assignment_id comes back null and the result is a preview of which arm that key maps to. Use it to plan; use voice_session_id to record.

409 This experiment is not running; no traffic is assigned outside the running state.

Results

GET /api/v1/voice/experiments/{experiment_id}/results

No parameters.

{
  "experiment_id": "ex10…",
  "status": "running",
  "metric": "goal_completed",
  "metric_label": "Goal completion rate",
  "higher_is_better": true,
  "min_sample_per_arm": 30,
  "total_assignments": 412,
  "sufficient_data": true,
  "leader_arm_id": "arm-b-id",
  "winner_arm_id": null,
  "verdict": "short-greeting is ahead on goal completion rate",
  "arms": [
    { "arm_id": "arm-a-id", "name": "control", "is_control": true, "sample_size": 205, "metric_value": 0.61 },
    { "arm_id": "arm-b-id", "name": "short-greeting", "is_control": false, "sample_size": 207, "metric_value": 0.68 }
  ]
}

There is deliberately no p-value or significance field. sufficient_data is a floor, not a test — treat leader_arm_id as a signal to keep running, and decide the winner yourself.

Errors