Realtime Voice

Realtime voice puts a speech-to-speech model in front of the normal pipeline, so small talk gets an instant answer while real work still goes to your agent.

          ┌──────────────────────────────┐
mic ──►   │ realtime model (Gemini Live, │ ──► answers small talk itself
          │ OpenAI Realtime, GPT-Live)   │
          └──────────────┬───────────────┘
                         │ delegate_to_main(...)
                         ▼
               os-server ──► agent runtime (OpenClaw, Hermes, …) ──► TTS

For each turn the realtime model either answers directly, calls delegate_to_main to hand the request to your agent runtime (tools, skills, memory), or rejects it as noise. With realtime disabled, every turn goes the plain STT → agent → TTS path.

By default the realtime model's reply is re-spoken through your configured TTS voice, so the Intern sounds the same whichever path answered. Delegated turns are always spoken by TTS.

Providers

`provider`	Model default	Voices	Reasoning knob
`gemini`	`gemini-3.8-live-extended-thinking`	`Puck`, `Charon`, `Kore`, `Fenrir`, `Aoede`	`LOW` · `MEDIUM` · `HIGH` (`MINIMAL` is accepted but the default model doesn't support it)
`openai`	`gpt-realtime-2`	`alloy`, `ash`, `coral`, `echo`, `fable`, `onyx`, `nova`, `sage`, `shimmer`	`minimal` · `low` · `medium` · `high` · `xhigh`
`gptlive`	`gpt-live-1`	`marin`, `quartz`, `ripple`, `vesper`, `willow`, `stone`, `gleam`, `meridian`, `bossa`, `tempo`, `beacon`, `delta`, `cinder`	—
`pipecat_v1`	`qwen/qwen3.6-35b-a3b`	— (text out, spoken by HAL TTS)	— (has `web_search` on/off)
`none`	—	—	Realtime off

The authoritative list for your firmware: GET /api/device/realtime-options.

Enable / change it

Web UI: http://<device-ip>/setting?debug=true#realtime → Realtime.

API:

# Gemini Live, low thinking
curl -s -X PUT $DEVICE/api/device/config \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"realtime":{"enabled":true,"provider":"gemini","voice":"Kore","reasoning":"LOW"}}'

# OpenAI Realtime
curl -s -X PUT $DEVICE/api/device/config \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"realtime":{"enabled":true,"provider":"openai","voice":"alloy","reasoning":"minimal"}}'

# Off
curl -s -X PUT $DEVICE/api/device/config \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"realtime":{"enabled":false}}'

realtime fields: enabled, provider, model, voice, reasoning, api_key, base_url, web_search (pipecat_v1 only). Omitted fields stay unchanged. Any realtime change restarts HAL.

In config.json the block looks like this. Until it's edited once, os-server rewrites it from the defaults on every start; an edit through the API or UI sets "pinned": true to stop that. If you edit the block by hand, set "pinned": true yourself or your change is reset on the next restart:

"realtime": {
  "enabled": true,
  "provider": "gemini",
  "api_key": "",
  "base_url": "",
  "gemini":  { "model": "gemini-3.8-live-extended-thinking", "voice": "Kore", "thinking_level": "LOW", "google_search": true },
  "openai":  { "model": "gpt-realtime-2", "voice": "alloy", "reasoning_effort": "minimal" },
  "gptlive": { "model": "gpt-live-1", "voice": "marin" },
  "pipecat_v1": { "model": "qwen/qwen3.6-35b-a3b", "web_search": true },
  "pinned": true
}

Keys and endpoints

Empty api_key / base_url fall back to the LLM credentials: <llm_base_url>/ws/gemini for Gemini and <llm_base_url>/ws/openai for OpenAI/GPT-Live — the Autonomous proxy relays. A base_url you set is used as-is, so include the path (…/ws/gemini or …/ws/openai). GPT-Live can also take its own realtime.gptlive.api_key / base_url. If you point the LLM at your own provider, pin realtime to Autonomous first (see that page's Step 1), or give it your own provider key with the env overrides below.

Hear the model's own voice

To play the realtime model's native audio instead of re-speaking through TTS (lower latency, but a different voice from the delegated turns), set in /opt/hal/.env:

HAL_REALTIME_NATIVE_AUDIO=true

This is automatic when both tts_provider and the realtime provider are gemini.

Environment overrides (`/opt/hal/.env`)

Env vars win over the config.json block. Restart HAL after changing them.

General

Var	Default	Effect
`HAL_REALTIME_ENABLED`	from config	Force on/off
`HAL_REALTIME_PROVIDER`	from config	`gemini` · `openai` · `gptlive` · `pipecat_v1` · `none`
`HAL_REALTIME_TURN_DETECTION`	`off`	`off` = HAL's own VAD decides turns · `server_vad` / `semantic_vad` = the provider decides (ignored by GPT-Live)
`HAL_REALTIME_NATIVE_AUDIO`	`false`	Play the model's voice instead of TTS
`HAL_REALTIME_REQUIRE_TRANSCRIPT`	`true`	Cost guard: drop turns with no transcript
`HAL_REALTIME_NOISE_SPEECH_RATIO`	`0.55`	Cost guard: drop turns that are mostly noise
`HAL_REALTIME_MIN_COMMIT_DURATION_S`	`0.8`	Cost guard: ignore very short blips

Gemini Live

Var	Effect
`GEMINI_API_KEY` / `GOOGLE_API_KEY`	Key (else `realtime.api_key`, else `llm_api_key`)
`HAL_GEMINI_LIVE_BASE_URL`	Endpoint (else `<llm_base_url>/ws/gemini`)
`HAL_GEMINI_LIVE_MODEL`, `HAL_GEMINI_LIVE_VOICE`	Model and voice
`HAL_GEMINI_THINKING_LEVEL`	`MINIMAL` · `LOW` · `MEDIUM` · `HIGH`
`HAL_GEMINI_GOOGLE_SEARCH`	Let the model search the web

OpenAI Realtime

Var	Effect
`OPENAI_API_KEY`	Key
`HAL_OPENAI_REALTIME_BASE_URL`	Endpoint (else `<llm_base_url>/ws/openai`)
`HAL_OPENAI_REALTIME_MODEL`, `HAL_OPENAI_REALTIME_VOICE`	Model and voice
`HAL_OPENAI_REASONING_EFFORT`	`minimal` … `xhigh`
`HAL_OPENAI_TRANSCRIBE_MODEL`	Model for the input transcript
`HAL_OPENAI_NOISE_REDUCTION`	e.g. `far_field`
`HAL_OPENAI_VAD_THRESHOLD`	Server VAD threshold

GPT-Live (OpenAI /v1/live/sessions)

Var	Effect
`OPENAI_API_KEY`	Key
`HAL_GPTLIVE_BASE_URL`	Endpoint (else the OpenAI Realtime base)
`HAL_GPTLIVE_MODEL`, `HAL_GPTLIVE_VOICE`	Model (`gpt-live-1`) and voice
`HAL_GPTLIVE_SAMPLE_RATE`	`16000`
`HAL_GPTLIVE_DELEGATION`	`auto` · `client` · `responses`
`HAL_GPTLIVE_WEB_SEARCH`	Web search tool

Pipecat v1

HAL_PIPECAT_API_KEY, HAL_PIPECAT_BASE_URL, HAL_PIPECAT_MODEL, HAL_PIPECAT_TEMPERATURE, HAL_PIPECAT_MAX_TOKENS.

Live (full-duplex) mode

In Live mode the provider owns turn-taking entirely: a session stays open and the provider's VAD decides when you've finished speaking.

# /opt/hal/.env
HAL_LIVE_MODE=true

This forces server_vad turn detection and applies to the whole process; switching needs a HAL restart.

Var	Default	Effect
`HAL_LIVE_VAD_START_SENSITIVITY`	`low`	How easily speech starts a turn
`HAL_LIVE_VAD_END_SENSITIVITY`	—	How quickly a pause ends it
`HAL_LIVE_VAD_PREFIX_PADDING_MS`	`300`	Audio kept before speech start
`HAL_LIVE_VAD_SILENCE_MS`	—	Silence that ends a turn
`HAL_LIVE_IDLE_HANGUP_S`	`15`	Close the session after this much silence
`HAL_LIVE_MAX_S`	`600`	Max session length
`HAL_LIVE_UPLINK_DURING_PLAYBACK`	`mute`	What the provider hears while the Intern is talking — see below

Voice barge-in is experimental

With the default HAL_LIVE_UPLINK_DURING_PLAYBACK=mute, the provider hears silence while the Intern speaks, so you can't interrupt it by voice — use the button. cancelled sends echo-cancelled audio (true full duplex, but only as good as the echo canceller, which currently isn't good enough on most units). always sends raw mic audio and lets the provider separate you from the Intern's own voice — the only mode where voice barge-in reliably triggers, but the model may hear itself and cut its own replies short. If you try always, lower the speaker volume.

Deeper protocol details: docs/realtime-voice.md.

Providers​

Enable / change it​

Keys and endpoints​

Hear the model's own voice​

Environment overrides (/opt/hal/.env)​

Live (full-duplex) mode​