Live Voice agents

Build an agent the user talks to: audio in, audio out, one stateful socket. The model hears the raw microphone and answers with speech — no speech-to-text step, no text-to-speech step, no turn-taking code of your own.

Model: gemini-3.1-flash-live-preview (openkbs models shows it with category: live). Endpoint: wss://live.openkbs.com/v1/live — a WebSocket relay, not the HTTP proxy. It meters credits per modality, keeps the session alive across the vendor's connection limits, and can run your tool calls server-side.

The flow

browser ──1── your project function ──2── proxy.openkbs.com/v1/live/session
   │                                              (mints a sealed 60 s token)
   └──3── wss://live.openkbs.com/v1/live ──4── the model

1. The browser asks your function for a session (your app's own auth).
2. The function calls the proxy with OPENKBS_API_KEY and gets back a token.
3. The browser opens the socket and sends that token as its first frame.
4. The relay holds the model connection and bills your project.

OPENKBS_API_KEY never reaches the browser. The token seals the model, the system instruction and the tool list — the browser cannot change any of them, so a leaked token cannot be turned into a general-purpose model on your credits.

1. Mint a session (in a project function)

// functions/voice/index.mjs
export async function handler(event) {
  const body = event.body ? JSON.parse(event.body) : {};
  if (body.action !== 'live-session') return json({ error: 'Unknown action' }, 400);

  const res = await fetch('https://proxy.openkbs.com/v1/live/session', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${process.env.OPENKBS_API_KEY}` },
    body: JSON.stringify({
      systemInstruction: 'You are a warm, brief assistant. Ask one question at a time.',
      voice: 'Kore',                       // Puck, Charon, Kore, Fenrir, Aoede …
      tools: [{ functionDeclarations: [{
        name: 'find_jobs',
        description: 'Find jobs that include accommodation',
        parameters: { type: 'OBJECT', properties: { sector: { type: 'STRING' } }, required: ['sector'] },
      }] }],
      toolMode: 'server',                  // 'server' | 'client'  (see below)
      toolUrl: process.env.SELF_FUNCTION_URL,
    }),
  });

  return json(await res.json());           // { url, token, model, expiresIn, audio }
}

POST /v1/live/session options

The response is what you hand to the SDK. The token is valid for 60 seconds — mint it when the user presses the button, not on page load.

2. Talk (in the browser)

<script src="https://openkbs.com/sdk/live.js"></script>

const session = await fetch('/api/voice', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ action: 'live-session' }),
}).then((r) => r.json());

const live = new OpenKBS.Live({ session });

live.on('ready',      ()             => console.log('listening'));
live.on('transcript', ({role, text}) => addBubble(role, text));   // role: 'user' | 'model'
live.on('state',      (s)            => setUi(s));  // connecting|listening|speaking|closed
live.on('interrupted',()             => console.log('user barged in'));
live.on('language',   (code)         => setLangBadge(code));  // after setLanguage() takes effect
live.on('closed',     (m)            => console.log(m.reason, m.credits, 'credits'));
live.on('error',      (e)            => showError(e.message));

await live.start();          // asks for microphone permission, then it is live

start() must be called from a user gesture (a click) — browsers block both the microphone and audio playback otherwise.

ESM instead of a script tag: import { Live } from 'https://openkbs.com/sdk/live.esm.js'.

3. Tool calls

toolMode: 'client' — the browser answers

For anything the UI owns: navigate, highlight a card, read local state. Whatever the handler returns goes back to the model.

live.on('tool', async ({ name, args }) => {
  if (name === 'show_job') { openJob(args.id); return { shown: true }; }
});

toolMode: 'server' — your Lambda answers

Use this whenever the tool touches data, credentials or rules the browser must not hold. The relay POSTs your function URL and never involves the client:

// what your function receives
{ "action": "tool", "tool": "find_jobs", "args": { "sector": "kitchen" },
  "projectId": "…", "sessionId": "…", "context": { } }

// functions/voice/index.mjs — same function, second action
if (body.action === 'tool') {
  if (body.tool === 'find_jobs') return json(await searchJobs(body.args.sector));
  return json({ error: `unknown tool ${body.tool}` });
}

Return any JSON — it is handed straight to the model. Always return something: an unanswered tool call leaves the conversation frozen mid-sentence. Tools time out after 20 s.

Authenticating the relay

Your function URL is public, so verify who is calling. Pass a secret of your own choosing when you mint the session and check it on every tool call:

// minting
toolSecret: process.env.MY_TOOL_SECRET,

// in the tool handler
if (event.headers['x-openkbs-tool-secret'] !== process.env.MY_TOOL_SECRET) {
  return json({ error: 'forbidden' }, 403);
}

The secret is yours — OpenKBS never hands a platform key to a project, and no platform secret is needed to verify a call.

Both at once — the normal case

A real voice app needs both: data and credentials on the server, anything that moves the screen in the browser. Routing is per tool, so use server mode and name the exceptions:

toolMode: 'server',
toolUrl: process.env.SELF_FUNCTION_URL,
clientTools: ['open_job', 'select_job', 'read_job', 'go_back'],

search_jobs and save_profile then run in your Lambda; open_job and friends are handed to live.on('tool', …) in the page, and whatever that returns goes back to the model. A client tool that never answers is auto-failed after 20 s so the conversation cannot freeze.

Reading things aloud is a tool, not TTS. Return the text to the model and let it speak — it reads in the user's language, with real prosody, and can be interrupted. Do not synthesise speech yourself and play it.

Language

The model understands and answers in whatever language it hears, without being told. But the transcript of the user's own audio is produced by a separate side-channel ASR that has no conversational context, and the only thing that anchors it is language / setLanguage().

Get that wrong and every user caption is wrong for the whole session. Same Bulgarian audio, same session, measured:

Short utterances ("yes", "no", one word) are the worst case — too little signal to guess from, so they come back transliterated or in an unrelated script entirely. There is no separate control for this: inputAudioTranscription accepts no language field, under any spelling.

So:

1. Mint with your best guess (language: 'de-DE').
2. The moment you learn what the person actually speaks — the model told you, or they picked it in the UI — call live.setLanguage('bg-BG').

live.on('tool', ({ name, args }) => {
  if (name === 'set_language') live.setLanguage(BCP47[args.lang]);
});

The relay applies it at the next turn boundary, by reconnecting upstream with a resumption handle: the conversation carries across intact and the user hears nothing. A language event confirms it. Only the utterances before the switch stay mis-transcribed, so switch on the first one.

Give the model a set_language tool if you want it to report what it heard — it is far better at identifying the language than the transcriber is, because it has the whole conversation to go on.

If your app offers a fixed set of languages, seal it at mint time with languages: [...] and the relay will refuse anything else.

Cost

Billed per modality against your project credits, per 1K tokens:

A 10-minute conversation lands around 15 credits. Credits are debited at every turn boundary, so a dropped connection still bills what was used. live.on('closed') reports the session's total.

What the relay handles for you

• Interruptions. Speak while the agent is talking and it stops instantly; the SDK drops the audio already queued. Nothing to implement.
• Session length. The vendor drops connections at ~10 minutes and audio-only sessions at 15. The relay reconnects with a resumption handle and enables context compression — your socket and the conversation both survive.
• Budget. Funds and the project spending limit are checked at mint time and again every minute; a session that runs out is closed with a reason.

Gotchas

• Audio is 16 kHz PCM in / 24 kHz PCM out. The SDK does this; if you write your own client, nothing else is accepted.
• Do not ask the model to "say" things it should not read aloud (URLs, ids). Give it a tool instead and render the result in the UI.
• One session = one microphone. Two Live instances on a page fight over it.
• A wrong language is worse than none for captions, and it is silent — the model still answers correctly, so only the transcript looks broken. If user bubbles show the right words in the wrong script, that is this and not a model problem.
• The 60-second token is single-conversation. Reconnecting after closed means minting a new one.

Related

• MQTT real-time messaging
• Site security headers
• Elastic services