Documentation.
Voice Events API — real-time turn detection, identity-gated barge-in, and noise intelligence for any voice agent.
What the Voice Events API does
Stream raw microphone audio in; get structured JSON events out — in real time, over one WebSocket:
- Turn detection —
turn_start/turn_endwith semantic endpointing (knows “I'd like to order… umm” isn't finished) - Identity-gated barge-in — only your enrolled speaker can interrupt the agent; a cough or a bystander cannot
- Speaker verification — an authoritative per-turn verdict (
should_respond) so your agent never answers the wrong voice - Noise intelligence — cough / breath / laughter / ambient events, never mistaken for speech turns
VSIP does not transcribe, run LLMs, or synthesize speech — it is the sensor layer beside your existing stack. It works with any setup that can forward 16kHz PCM: custom Python backends, LiveKit Agents, Daily Bots, Pipecat, Twilio Media Streams (audio_format=mulaw8k), and speech-to-speech stacks like OpenAI Realtime via listen-only mode=sidecar.
Quickstart
Create an API key on the API Keys page, then establish your connection:
import asyncio, json, pyaudio, websockets
API_KEY = "vsip_YOUR_KEY_HERE"
async def main():
uri = "wss://api.vsip.io/v1/stream"
headers = {"Authorization": f"Bearer {API_KEY}"}
async with websockets.connect(uri, additional_headers=headers) as ws:
print(json.loads(await ws.recv())) # session_start
p = pyaudio.PyAudio()
mic = p.open(format=pyaudio.paInt16, channels=1, rate=16000,
input=True, frames_per_buffer=1600)
async def send_audio():
while True:
await ws.send(mic.read(1600, exception_on_overflow=False))
await asyncio.sleep(0)
asyncio.create_task(send_audio())
async for raw in ws:
event = json.loads(raw)
if event["type"] in ("turn_start", "turn_end", "barge_in",
"speaker_verification"):
print(event)
asyncio.run(main())Speak — you'll see turn_start, then turn_end when you stop, then speaker_verification. Your first sentence auto-enrolls your voice (speaker_change + lock_status fire mid-turn).
Authentication
Every connection requires an API key, created on the API Keys page. Pass it as a header on the WebSocket handshake:
Authorization: Bearer vsip_YOUR_API_KEYInvalid, revoked, or missing keys are rejected with close code 4001 before the upgrade completes — no data is exchanged.
Connecting & parameters
wss://api.vsip.io/v1/stream?tick_ms=80&session_id=call-abc-123| Parameter | Default | Description |
|---|---|---|
tick_ms | 80 | Inference tick interval (20–500ms) |
audio_format | int16 | int16 · float32 · mulaw8k (Twilio) |
mode | full | sidecar = listen-only beside speech-to-speech agents |
aec | client | server = VSIP cancels echo; tag frames 0x01 mic / 0x02 reference |
noise_suppress | false | DeepFilterNet3 denoising (+~15ms) |
The 4-step integration contract
- Send 16kHz mono PCM as binary WebSocket frames.
- Send
{"command":"set_agent_state","value":"speaking"}before your TTS starts playing. - Send
{"command":"set_agent_state","value":"idle"}when TTS ends or is interrupted. - On
barge_in: stop your TTS immediately, then sendidle.
Control messages
Send JSON text frames on the same WebSocket to control the pipeline mid-session:
| Command | Purpose |
|---|---|
set_agent_state | {"command":"set_agent_state","value":"speaking"|"idle"} — signals TTS state adjustments. |
enroll_speaker | Register voiceprint from sample → enrollment_result |
toggle_lock | Lock session control to active speaker. |
Events reference
| Event | When | Key fields |
|---|---|---|
turn_start | Speech turn opens | speaker_id, audio_start |
turn_end | Speech turn ends | duration_ms, endpoint |
barge_in | Interruption detected | speaker_id, confidence |
speaker_verification | Identity matching complete | should_respond, similarity |
Speaker verification & barge-in
Gate your agent response decisions on speaker_verification.should_respond — not on raw speaker tags. Acoustic feedback cancellation can introduce noise factors, so verify completely before releasing turns.
Errors & close codes
| Code | Meaning |
|---|---|
| 4001 | Unauthorized — missing or invalid token |
| 4029 | Concurrency quota reached |
Latency metrics
| Metric | p50 | p95 |
|---|---|---|
| Semantic turn_end | 300ms | 700ms |
| Confirmed match verification | 250ms | 450ms |
Python SDK & adapters
The vsip Python SDK (beta) wraps the WebSocket protocol and bakes in every integration pattern on this page — onset-lookback audio slicing, post-lock verification gating, queued turns, and transparent reconnect + resume. You implement one on_turn callback; the SDK handles the rest.
pip install vsip-sdk # typed client + VoiceSession
pip install "vsip-sdk[pipecat]" # + drop-in Pipecat processor
# distribution is vsip-sdk; the import package is still 'vsip'VoiceSession — one callback
from vsip import VSIPClient, VoiceSession, Turn, int16_to_wav
client = VSIPClient(api_key="vsip_...", url="wss://api.vsip.io/v1/stream")
async def on_turn(turn: Turn):
if not turn.should_respond: # verified: not your enrolled user
return
text = await my_stt(int16_to_wav(turn.audio))
reply = await my_llm(text, interrupted=turn.interrupted_agent)
await session.speak_gate() # never talk over an open turn
await client.agent_speaking()
await my_tts_play(reply) # stop this in on_barge_in
await client.agent_idle()
session = VoiceSession(client, on_turn=on_turn,
on_barge_in=lambda e: my_tts_stop())
async with client:
await session.start()
async for frame in mic_frames(): # 16kHz mono int16 PCM
await session.send_audio(frame)turn.should_respond is the single field you gate your agent on: pre-lock turns dispatch immediately (first-conversation UX); once a speaker is locked, every turn is held until the speaker_verification verdict arrives, so an impostor is never answered.
Pipecat adapter
Drop VSIPProcessor between your STT and LLM stages for identity-gated interruption and verification gating inside a Pipecat pipeline — impostor transcriptions never reach your LLM.
from vsip import VSIPClient
from vsip.adapters.pipecat import VSIPProcessor
vsip = VSIPProcessor(VSIPClient(api_key="vsip_...", url="wss://.../v1/stream"))
pipeline = Pipeline([
transport.input(),
stt,
vsip, # ← identity-gated barge-in + verification gating
context_aggregator,
llm,
tts,
transport.output(),
])LiveKit Agents plugin
Plug VSIPVAD in as the session's turn detector — VSIP's semantic endpointing and identity-gated barge-in drive the agent's turn-taking, so a cough or bystander can't interrupt.
from livekit.agents import AgentSession
from vsip import VSIPClient
from vsip.adapters.livekit import VSIPVAD
vsip_vad = VSIPVAD(VSIPClient(api_key="vsip_...", url="wss://.../v1/stream"))
session = AgentSession(vad=vsip_vad, stt=..., llm=..., tts=...)Twilio Media Streams
VSIP ingests Twilio's μ-law natively. Bridge a phone call to a VSIP session and drive your telephony agent from its events — clear is sent on barge-in to stop buffered playback.
from vsip import VSIPClient
from vsip.adapters.twilio import TwilioBridge
client = VSIPClient(api_key="vsip_...", url="wss://.../v1/stream",
audio_format="mulaw8k")
bridge = TwilioBridge(client, on_turn_end=..., on_verification=...)
await bridge.run(ws.iter_text(), send=ws.send_text) # FastAPI/Starlette WSJavaScript / TypeScript
Browser + Node 18+. Same VoiceSession patterns, fully typed.
import { VSIPClient, VoiceSession, type Turn } from "vsip-sdk";
const client = new VSIPClient({ apiKey: "vsip_...", url: "wss://.../v1/stream" });
const session = new VoiceSession(client, {
onTurn: (t: Turn) => { if (t.shouldRespond) respond(t.audio); },
onBargeIn: () => myTts.stop(),
});
await client.connect();
session.start();Speaker Profile API
A separate REST API (Product 3): enroll a voiceprint once, then verify (1:1 — “is this Alice?”) or identify (1:N — “which known speaker is this?”) from a short clip. Authenticated with the same vsip_ API key.
from vsip import ProfilesClient
profiles = ProfilesClient(api_key="vsip_...", base_url="https://api.vsip.io")
# enroll (>=3s clean speech; wav/mp3/int16/float32/mulaw8k)
p = await profiles.enroll("alice", wav_bytes, audio_format="wav",
consent_obtained=True, retention_days=365)
# 1:1 verify
r = await profiles.verify(p["profile_id"], sample_bytes, audio_format="wav")
# -> {"match": true, "score": 0.82, "threshold": 0.75, ...}
# 1:N identify
who = await profiles.identify(sample_bytes, audio_format="wav")Enrollment is quality-gated (rejects too-short / silent / noisy audio with a machine-readable audio_quality code). Every response carries score + threshold + decision; pass a per-request threshold to tune strictness. Enroll is free; verify/identify are metered against a monthly tier allowance.
Persistent identity in streams
Enroll once via REST, then open a stream with ?profile_id=<id> — the session locks to that voice from the first word (no in-session enrollment). One voiceprint, both products.