Give your agent a voice

Your backend from the Express guide chats in text. This guide makes it talk. There are three honest ways to do that, in order of effort, and the first one costs your server nothing: no new endpoint, no model, no audio code. Most backends stop there.

The three paths, side by side:

1. Declare voice, get voice

Voice rides the chat stream. The app captures your speech, turns it into text on the phone, and sends that text to the same POST /chat endpoint you already serve. The reply streams back as SSE and the phone speaks it with the device voice. Your server cannot tell a spoken turn from a typed one, which is the point: if you implemented chat, voice already works.

The one thing your server does is declare the surface, because the app hides tabs a manifest doesn’t declare. Add a single line to src/manifest.js from the Express guide:

// src/manifest.js
surfaces: {
  chat: { endpoint: 'rest', attachments: false },
  voice: {},
  dashboards: [
    { id: 'home', title: 'Home', source: '/ui/home', refresh: { mode: 'manual' } },
  ],
},

An empty object is a complete declaration. The defaults are mode: "ptt" (push-to-talk and tap-to-toggle), stt: "device", and tts: "device", and v1 accepts no other values for the first two. Redeploy, then foreground the app; if the Voice tab doesn’t appear, quit and relaunch, since the manifest cache refreshes at most once per app every five minutes.

Two properties of this path worth stating plainly:

• Speech-to-text runs on the device. Your audio never leaves the phone; the server receives plain text.
• It’s free, for you and for your users. Device voice is part of the free tier, hands-free conversation included.

The cost is the voice itself: you get the platform synthesizer, which is serviceable and unmistakably robotic. Paths 2 and 3 exist to fix that one thing.

How a turn works

The same loop drives all three paths; only the speaker changes.

You talk, and a live transcript appears as the on-device recognizer works. When you stop, the app ends your turn (a short silence window) and posts the transcript to /chat. The reply streams back as text, and the app splits it into sentences and speaks them as they arrive, so speech starts before the reply finishes.

Turn-taking supports barge-in: speak over the reply and it yields. The voice stops, the words the recognizer already caught stay in your transcript, and the turn is yours. Echo cancellation keeps the mic from transcribing the phone’s own voice while both are live. If you’d rather hold the button and take strict turns, turn on Classic turn-taking in Settings > Voice.

Every voice exchange persists to the chat thread, so a conversation you had out loud is still there to scroll and search.

2. Turn on Parlane voice

For a natural voice with zero backend changes, there’s Parlane voice, a hosted option the user turns on, not the developer. When it’s active for an app that declares no voice.tts of its own, the app sends each reply sentence to a Parlane-operated service that synthesizes it with Amazon Polly neural voices (Joanna, Matthew, Ruth, or Stephen) and returns audio. Nothing changes in your manifest, and there’s nothing to deploy.

Be clear about what that means for privacy, because it’s the one path where reply text leaves the user’s device for a Parlane server. The service processes text in memory and doesn’t log or store it; its logs carry character counts, latency, and a hashed user id, never content. The service can see the text it synthesizes. Similarly, the push relay can see visible notification previews even when an additional payload is encrypted. The privacy policy explains these data flows. To keep speech generation on your own infrastructure, use path 3.

The economics: anyone can try it free for 15 minutes of speech, once. After that it’s a Pro feature with an allowance of about 4.6 hours of spoken replies per month, and a $4.99 recharge adds roughly 2.8 more hours when a month runs long (see pricing). Running out mid-reply costs quality, never sound: the reply finishes with the device voice.

Users control it per app in the app’s settings, where it’s on by default for Pro subscribers when the connected app hosts no voice of its own.

3. Host the voice yourself

The third path puts a text-to-speech endpoint on your own server, so reply text reaches your infrastructure and nowhere else. This part of the contract is experimental: it’s additive and won’t break existing manifests, but the shape may still change.

Declare it in your manifest’s voice surface:

"voice": {
  "tts": {
    "mode": "server",
    "endpoint": "/tts",
    "format": "wav",
    "voices": ["af_heart", "af_bella", "am_michael"]
  }
}

format accepts "wav" or "mp3" (the default). voices is optional; omit it if your endpoint has one fixed voice.

The wire contract is one POST, carrying the same bearer auth as your other endpoints. The app chunks replies itself and sends one sentence per call, prefetching one ahead of playback:

{ "text": "One sentence of the reply.", "voice": "af_heart" }

voice is optional and only ever one of your declared voices; default sensibly when it’s absent. Respond with 200 and a complete audio file for that text, with a Content-Type of audio/wav or audio/mpeg matching your declared format. There is no streaming contract. Keep per-sentence synthesis under a second; the client aborts at 10 seconds.

Failure semantics make a broken endpoint invisible: any non-200, timeout, or network error finishes the reply with device TTS and retries your server on the next reply. Failures cost quality, never sound. The audible symptom is a voice that pauses and comes back robotic mid-reply, which means your endpoint errored or ran slow.

A complete server in one file

The reference implementation runs Kokoro-82M (Apache-2.0) through kokoro-js: pure Node, no Python, no GPU, and real-time on CPU. The quantized weights are about 90 MB.

// server.js, quoted from docs/06 §5.6. Copy it as-is
// (npm install express kokoro-js), run it as its own service, or mount the
// /tts handler in your backend behind your existing auth.
import express from 'express';
import { KokoroTTS } from 'kokoro-js';

const PORT = Number(process.env.PORT) || 8880;
const AUTH_TOKEN = process.env.AUTH_TOKEN || ''; // set = same bearer as your chat endpoints
const VOICES = ['af_heart', 'af_bella', 'am_michael'];
const DEFAULT_VOICE = 'af_heart';
const MAX_TEXT_LENGTH = 2000;

// Kokoro-82M, Apache-2.0, pure Node ONNX, real-time on CPU. The q8 weights
// (~90 MB) download and cache on first start; restarts are instant.
const tts = await KokoroTTS.from_pretrained('onnx-community/Kokoro-82M-v1.0-ONNX', { dtype: 'q8' });

const app = express();
app.use(express.json({ limit: '64kb' }));
app.use((req, res, next) => {
  if (!AUTH_TOKEN) return next(); // open; acceptable on private networks only
  const [scheme, token] = (req.get('authorization') || '').split(' ');
  if (scheme === 'Bearer' && token === AUTH_TOKEN) return next();
  res.status(401).json({ error: { code: 'unauthorized', message: 'Missing or invalid bearer token' } });
});
app.get('/healthz', (req, res) => res.json({ ok: true }));
app.post('/tts', async (req, res) => {
  const { text, voice } = req.body || {};
  if (typeof text !== 'string' || !text.trim()) {
    return res.status(400).json({ error: { code: 'bad_request', message: 'Body must be {"text": "..."}.' } });
  }
  if (text.length > MAX_TEXT_LENGTH) {
    return res.status(400).json({ error: { code: 'text_too_long', message: `Max ${MAX_TEXT_LENGTH} chars per request.` } });
  }
  try {
    const audio = await tts.generate(text, { voice: VOICES.includes(voice) ? voice : DEFAULT_VOICE });
    res.set('Content-Type', 'audio/wav');
    res.set('Cache-Control', 'no-store');
    res.send(Buffer.from(audio.toWav()));
  } catch {
    res.status(500).json({ error: { code: 'tts_failed', message: 'Synthesis failed.' } });
  }
});
app.listen(PORT, () => console.log(`voice-tts listening on :${PORT}`));

Run it and listen to it:

npm install express kokoro-js
AUTH_TOKEN=<your-token> node server.js

curl -s -X POST localhost:8880/tts \
  -H 'Authorization: Bearer <your-token>' -H 'Content-Type: application/json' \
  -d '{"text": "Hello from my own server."}' -o out.wav && afplay out.wav

The in-repo version, examples/voice-tts/, adds a Dockerfile that bakes the weights in at build time, plus notes on mounting the handler in-process instead of running a sidecar. Either way, never log the text field. It’s the user’s conversation.

The honest hardware note

CPU real-time is true, but only on real cores. A 2 vCPU / 1 GB box synthesizes a sentence faster than it plays, and the one-ahead prefetch hides the rest. A small burstable VPS does not get there: on a t4g.small-class instance we measured about 5 seconds per sentence, which the prefetch can’t hide, so every reply starts with dead air and stutters between sentences. If your server is that small, use the device voice or the hosted voice instead of shipping a bad one.

Consuming a server voice in the app is a Pro feature, like the hosted voice. The device path stays free.

Which voice speaks

When a reply arrives, the app resolves the speaker in a fixed order:

1. A declared voice.tts server endpoint, when the user’s Server voice setting for that app is on (the default).
2. Parlane voice, for Pro users who have it on and only for apps that declare no voice.tts.
3. The device voice, which is the free default and the fallback for both paths above.

One consequence is deliberate: a declared server TTS always overrides the hosted voice, even when the endpoint is down. A developer who self-hosts has opted their users’ reply text out of Parlane-hosted processing, and the app respects that opt-out instead of quietly routing around an outage. A failing endpoint falls back to the device voice, never to the hosted service.

What you built

Possibly nothing, which was the promise:

• voice: {} in the manifest gives every user of your backend a spoken conversation loop with barge-in, free, with audio that never leaves the phone.
• Parlane voice upgrades how replies sound with no server work, in exchange for a disclosed, policy-based privacy posture.
• A self-hosted /tts endpoint (about 50 lines with Kokoro-82M) keeps reply text on your own infrastructure and takes precedence over the hosted voice by design.

To wire chat, and with it voice, into the backend you already have, use the prompt below with your coding agent.