Live Voice agents

Build an agent the user talks to: audio in, audio out, one stateful socket. The model hears the raw microphone and answers with speech — no speech-to-text step, no text-to-speech step, no turn-taking code of your own.

Model: gemini-3.8-live (default; GA). gemini-3.1-flash-live-preview still works. openkbs models lists both with category: live. Endpoint: wss://live.openkbs.com/v1/live — a WebSocket relay, not the HTTP proxy. It meters credits per modality, keeps the session alive across the vendor's connection limits, and can run your tool calls server-side.

The flow

browser ──1── your project function ──2── proxy.openkbs.com/v1/live/session
   │                                              (mints a sealed 60 s token)
   └──3── wss://live.openkbs.com/v1/live ──4── the model

1. The browser asks your function for a session (your app's own auth).
2. The function calls the proxy with OPENKBS_API_KEY and gets back a token.
3. The browser opens the socket and sends that token as its first frame.
4. The relay holds the model connection and bills your project.

OPENKBS_API_KEY never reaches the browser. The token seals the model, the system instruction and the tool list — the browser cannot change any of them, so a leaked token cannot be turned into a general-purpose model on your credits.

1. Mint a session (in a project function)

// functions/voice/index.mjs
export async function handler(event) {
  const body = event.body ? JSON.parse(event.body) : {};
  if (body.action !== 'live-session') return json({ error: 'Unknown action' }, 400);

  const res = await fetch('https://proxy.openkbs.com/v1/live/session', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${process.env.OPENKBS_API_KEY}` },
    body: JSON.stringify({
      systemInstruction: 'You are a warm, brief assistant. Ask one question at a time.',
      voice: 'Kore',                       // Puck, Charon, Kore, Fenrir, Aoede …
      tools: [{ functionDeclarations: [{
        name: 'find_jobs',
        description: 'Find jobs that include accommodation',
        parameters: { type: 'OBJECT', properties: { sector: { type: 'STRING' } }, required: ['sector'] },
      }] }],
      toolMode: 'server',                  // 'server' | 'client'  (see below)
      toolUrl: process.env.SELF_FUNCTION_URL,
    }),
  });

  return json(await res.json());           // { url, token, model, expiresIn, audio }
}

POST /v1/live/session options

thinkingConfig and proactivity are not forwarded to 3.8 — the model closes the socket on either. Thinking is built in (fixed) on gemini-3.8-live; only gemini-3.8-live-extended-thinking takes a thinkingLevel.

The response is what you hand to the SDK. The token is valid for 60 seconds — mint it when the user presses the button, not on page load.

2. Talk (in the browser)

<script src="https://openkbs.com/sdk/live.js"></script>

const session = await fetch('/api/voice', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ action: 'live-session' }),
}).then((r) => r.json());

const live = new OpenKBS.Live({ session });

live.on('ready',      ()             => console.log('listening'));
live.on('transcript', ({role, text, languageCode}) => addBubble(role, text));  // role: 'user' | 'model'
live.on('transcript_interim', ({text}) => showDraft(text));  // partial of what the user is saying
live.on('voice_activity', (a)        => a === 'start' ? showEar() : hideEar());  // model's VAD, no words needed
live.on('state',      (s)            => setUi(s));  // connecting|listening|speaking|closed
live.on('interrupted',()             => console.log('user barged in'));
live.on('turn_complete', ({status})  => {});        // status 'IN_PROGRESS' only on extended-thinking
live.on('language',   (code)         => setLangBadge(code));  // after setLanguage() takes effect
live.on('closed',     (m)            => console.log(m.reason, m.credits, 'credits'));
live.on('error',      (e)            => showError(e.message));

await live.start();          // asks for microphone permission, then it is live

start() must be called from a user gesture (a click) — browsers block both the microphone and audio playback otherwise.

ESM instead of a script tag: import { Live } from 'https://openkbs.com/sdk/live.esm.js'.

3. Tool calls

toolMode: 'client' — the browser answers

For anything the UI owns: navigate, highlight a card, read local state. Whatever the handler returns goes back to the model.

live.on('tool', async ({ name, args }) => {
  if (name === 'show_job') { openJob(args.id); return { shown: true }; }
});

toolMode: 'server' — your Lambda answers

Use this whenever the tool touches data, credentials or rules the browser must not hold. The relay POSTs your function URL and never involves the client:

// what your function receives
{ "action": "tool", "tool": "find_jobs", "args": { "sector": "kitchen" },
  "projectId": "…", "sessionId": "…", "context": { } }

// functions/voice/index.mjs — same function, second action
if (body.action === 'tool') {
  if (body.tool === 'find_jobs') return json(await searchJobs(body.args.sector));
  return json({ error: `unknown tool ${body.tool}` });
}

Return JSON — it is handed straight to the model. Always return something: an unanswered tool call leaves the conversation frozen mid-sentence. Tools time out after 20 s. Return a structured result, never {} or a bare string: { status: 'ok' | 'no_results' | 'not_now' | 'invalid', message, … }. An empty or vague result is what makes 3.8 retry the same call two to five times before it speaks.

ui goes to the page, not the model. Put what the screen needs — the full job list, a document, a form to show — under ui in the tool response: the relay strips it from what the model reads (so it costs no tokens) and delivers it to the browser on tool_done (live.on('tool_done', ({ name, ui }) => render(ui))). Give the model a short message and the page the data; nothing has to be fetched twice or stored anywhere.

Async tools on 3.8

On gemini-3.8-live a tool call no longer stalls the conversation: the model keeps talking ("let me look that up…") while your function runs, and your result decides how it lands. Put scheduling inside the response:

return json({ status: 'ok', jobs, scheduling: 'INTERRUPT' });   // speak it now
//            scheduling: 'WHEN_IDLE'  → when the person stops talking
//            scheduling: 'SILENT'     → just remember it (saves, logs)

Server calls of one model turn run in order and each carries batch: { index, size } in the POST body: when the person answers five questions in one breath and the model fires five saves, every call sees the ones before it, and your function can answer "what is still open" once, from the last one (batch.index === batch.size - 1).

Declare behavior: 'BLOCKING' on a tool if the model must wait for it (3.1 behaviour); a blocking call is cancelled when the person speaks again. The relay answers a duplicate call (same name and arguments while the first is in flight) with the first result, so your function is not hit twice.

Authenticating the relay

Your function URL is public, so verify who is calling. Pass a secret of your own choosing when you mint the session and check it on every tool call:

// minting
toolSecret: process.env.MY_TOOL_SECRET,

// in the tool handler
if (event.headers['x-openkbs-tool-secret'] !== process.env.MY_TOOL_SECRET) {
  return json({ error: 'forbidden' }, 403);
}

The secret is yours — OpenKBS never hands a platform key to a project, and no platform secret is needed to verify a call.

Both at once — the normal case

A real voice app needs both: data and credentials on the server, anything that moves the screen in the browser. Routing is per tool, so use server mode and name the exceptions:

toolMode: 'server',
toolUrl: process.env.SELF_FUNCTION_URL,
clientTools: ['open_job', 'select_job', 'read_job', 'go_back'],

search_jobs and save_profile then run in your Lambda; open_job and friends are handed to live.on('tool', …) in the page, and whatever that returns goes back to the model. A client tool that never answers is auto-failed after 20 s so the conversation cannot freeze.

Reading things aloud is a tool, not TTS. Return the text to the model and let it speak — it reads in the user's language, with real prosody, and can be interrupted. Do not synthesise speech yourself and play it.

Language

The model understands and answers in whatever language it hears, without being told; on 3.8 it switches mid-conversation on its own. Pin the reply language in the system instruction when the app knows it — the vendor's own wording works: RESPOND IN German. YOU MUST RESPOND UNMISTAKABLY IN German. Later changes go through a tool result ("from now on respond in Turkish").

The transcript of the user's own audio is a separate ASR with no conversational context. language is its hint (3.8: inputAudioTranscription.languageCodes; 3.1: speechConfig.languageCode — the proxy puts it where the model expects). Measured on the same Bulgarian clips, 2026-10-04:

A one-second "yes" has too little signal in any configuration. So: treat the transcript as display only — the model understood the audio either way — and never make an app decision from transcript text. Decide from tool arguments; use voice_activity for "the person just spoke". On 3.8 add names and jargon to customVocabulary, and hide a user caption whose languageCode (when reported) is not the conversation's.

If the language is only known after the first utterance, call live.setLanguage('bg-BG') the moment you know. The relay re-anchors at the next turn boundary by reconnecting upstream with a resumption handle; the conversation carries across (verified on 3.8 and 3.1) and a language event confirms it. Seal the offered set with languages: [...] so a browser cannot pick one the app never translated its UI for.

Resuming after a reload

Mint with historyConfig: { initialHistoryInClientContent: true }, then after ready replay the history with sendText(text, { role, turnComplete: false }) — user and model lines alike — and finish with one sendText('…', { turnComplete: true }). One short summary line beats thirty raw lines: a native-audio model treats pasted text as something the person typed, and a long replay makes it greet again as if the call had just started.

Cost

Billed per modality against your project credits, per 1K tokens:

The whole retained context — system instruction, tools, every earlier turn as audio — is re-read on every turn, so the price of a conversation grows with its length and with the size of the instruction. A 25k-token instruction costs ~3 credits per turn before a word is spoken; keep it under ~2k tokens and let tool results carry the detail. A 10-minute conversation with a short instruction lands around 20 credits on 3.8. Credits are debited at every turn boundary, so a dropped connection still bills what was used. live.on('closed') reports the session's total.

What the relay handles for you

• Interruptions. Speak while the agent is talking and it stops instantly; the SDK drops the audio already queued. Nothing to implement.
• Session length. 3.1 drops connections at ~10 minutes; a 3.8 connection was measured idle for 12 minutes without one. Either way the relay reconnects with a resumption handle and keeps context compression on — your socket and the conversation both survive, up to maxSessionMs.
• Budget. Funds and the project spending limit are checked at mint time and again every minute; a session that runs out is closed with a reason.

Gotchas

• Audio is 16 kHz PCM in / 24 kHz PCM out. The SDK does this; if you write your own client, nothing else is accepted.
• Do not ask the model to "say" things it should not read aloud (URLs, ids). Give it a tool instead and render the result in the UI.
• One session = one microphone. Two Live instances on a page fight over it.
• A wrong language is worse than none for captions, and it is silent — the model still answers correctly, so only the transcript looks broken. If user bubbles show the right words in the wrong script, that is this and not a model problem — and nothing in the app should have been reading that text anyway.
• Do not inject "wake-up" or control text as user turns in a loop. Every sendText is a turn the model answers; a second one while the first is still being spoken interrupts it, which is how a greeting gets said twice.
• The 60-second token is single-conversation. Reconnecting after closed means minting a new one.

Related

• MQTT real-time messaging
• Site security headers
• Elastic services