VSIP
VSIP SDK

Documentation.

Voice Events API — real-time turn detection, identity-gated barge-in, and noise intelligence for any voice agent.

01.0 / OVERVIEW

What the Voice Events API does

Stream raw microphone audio in; get structured JSON events out — in real time, over one WebSocket:

  • Turn detectionturn_start / turn_end with semantic endpointing (knows “I'd like to order… umm” isn't finished)
  • Identity-gated barge-in — only your enrolled speaker can interrupt the agent; a cough or a bystander cannot
  • Speaker verification — an authoritative per-turn verdict (should_respond) so your agent never answers the wrong voice
  • Noise intelligence — cough / breath / laughter / ambient events, never mistaken for speech turns

VSIP does not transcribe, run LLMs, or synthesize speech — it is the sensor layer beside your existing stack. It works with any setup that can forward 16kHz PCM: custom Python backends, LiveKit Agents, Daily Bots, Pipecat, Twilio Media Streams (audio_format=mulaw8k), and speech-to-speech stacks like OpenAI Realtime via listen-only mode=sidecar.

02.0 / IMPLEMENTATION

Quickstart

Create an API key on the API Keys page, then establish your connection:

quickstart.pyCODE PREVIEW
import asyncio, json, pyaudio, websockets

API_KEY = "vsip_YOUR_KEY_HERE"

async def main():
    uri = "wss://api.vsip.io/v1/stream"
    headers = {"Authorization": f"Bearer {API_KEY}"}

    async with websockets.connect(uri, additional_headers=headers) as ws:
        print(json.loads(await ws.recv()))  # session_start

        p = pyaudio.PyAudio()
        mic = p.open(format=pyaudio.paInt16, channels=1, rate=16000,
                     input=True, frames_per_buffer=1600)

        async def send_audio():
            while True:
                await ws.send(mic.read(1600, exception_on_overflow=False))
                await asyncio.sleep(0)

        asyncio.create_task(send_audio())

        async for raw in ws:
            event = json.loads(raw)
            if event["type"] in ("turn_start", "turn_end", "barge_in",
                                 "speaker_verification"):
                print(event)

asyncio.run(main())

Speak — you'll see turn_start, then turn_end when you stop, then speaker_verification. Your first sentence auto-enrolls your voice (speaker_change + lock_status fire mid-turn).

03.0 / AUTHORIZATION

Authentication

Every connection requires an API key, created on the API Keys page. Pass it as a header on the WebSocket handshake:

http_headerCODE PREVIEW
Authorization: Bearer vsip_YOUR_API_KEY

Invalid, revoked, or missing keys are rejected with close code 4001 before the upgrade completes — no data is exchanged.

04.0 / WEBSOCKET SPECS

Connecting & parameters

websocket_urlCODE PREVIEW
wss://api.vsip.io/v1/stream?tick_ms=80&session_id=call-abc-123
ParameterDefaultDescription
tick_ms80Inference tick interval (20–500ms)
audio_formatint16int16 · float32 · mulaw8k (Twilio)
modefullsidecar = listen-only beside speech-to-speech agents
aecclientserver = VSIP cancels echo; tag frames 0x01 mic / 0x02 reference
noise_suppressfalseDeepFilterNet3 denoising (+~15ms)
05.0 / STATE LOOP

The 4-step integration contract

  1. Send 16kHz mono PCM as binary WebSocket frames.
  2. Send {"command":"set_agent_state","value":"speaking"} before your TTS starts playing.
  3. Send {"command":"set_agent_state","value":"idle"} when TTS ends or is interrupted.
  4. On barge_in: stop your TTS immediately, then send idle.
06.0 / DOWNSTREAM COMMANDS

Control messages

Send JSON text frames on the same WebSocket to control the pipeline mid-session:

CommandPurpose
set_agent_state{"command":"set_agent_state","value":"speaking"|"idle"} — signals TTS state adjustments.
enroll_speakerRegister voiceprint from sample → enrollment_result
toggle_lockLock session control to active speaker.
07.0 / UPSTREAM EVENTS

Events reference

EventWhenKey fields
turn_startSpeech turn opensspeaker_id, audio_start
turn_endSpeech turn endsduration_ms, endpoint
barge_inInterruption detectedspeaker_id, confidence
speaker_verificationIdentity matching completeshould_respond, similarity
08.0 / IDENTITY & GATING

Speaker verification & barge-in

Gate your agent response decisions on speaker_verification.should_respond — not on raw speaker tags. Acoustic feedback cancellation can introduce noise factors, so verify completely before releasing turns.

FAIL-SAFEAgent voice halts immediately when speech energy matches target thresholds.
FAIL-SECUREOnly confirmed matches will trigger the downstream LLM processing pipeline.
09.0 / CLOSE CODES

Errors & close codes

CodeMeaning
4001Unauthorized — missing or invalid token
4029Concurrency quota reached
10.0 / DIAGNOSTICS

Latency metrics

Metricp50p95
Semantic turn_end300ms700ms
Confirmed match verification250ms450ms
11.0 / SDK & ADAPTERS

Python SDK & adapters

The vsip Python SDK (beta) wraps the WebSocket protocol and bakes in every integration pattern on this page — onset-lookback audio slicing, post-lock verification gating, queued turns, and transparent reconnect + resume. You implement one on_turn callback; the SDK handles the rest.

install.shCODE PREVIEW
pip install vsip-sdk              # typed client + VoiceSession
pip install "vsip-sdk[pipecat]"  # + drop-in Pipecat processor
# distribution is vsip-sdk; the import package is still 'vsip'

VoiceSession — one callback

agent.pyCODE PREVIEW
from vsip import VSIPClient, VoiceSession, Turn, int16_to_wav

client = VSIPClient(api_key="vsip_...", url="wss://api.vsip.io/v1/stream")

async def on_turn(turn: Turn):
    if not turn.should_respond:      # verified: not your enrolled user
        return
    text  = await my_stt(int16_to_wav(turn.audio))
    reply = await my_llm(text, interrupted=turn.interrupted_agent)
    await session.speak_gate()       # never talk over an open turn
    await client.agent_speaking()
    await my_tts_play(reply)         # stop this in on_barge_in
    await client.agent_idle()

session = VoiceSession(client, on_turn=on_turn,
                       on_barge_in=lambda e: my_tts_stop())

async with client:
    await session.start()
    async for frame in mic_frames():  # 16kHz mono int16 PCM
        await session.send_audio(frame)

turn.should_respond is the single field you gate your agent on: pre-lock turns dispatch immediately (first-conversation UX); once a speaker is locked, every turn is held until the speaker_verification verdict arrives, so an impostor is never answered.

Pipecat adapter

Drop VSIPProcessor between your STT and LLM stages for identity-gated interruption and verification gating inside a Pipecat pipeline — impostor transcriptions never reach your LLM.

pipeline.pyCODE PREVIEW
from vsip import VSIPClient
from vsip.adapters.pipecat import VSIPProcessor

vsip = VSIPProcessor(VSIPClient(api_key="vsip_...", url="wss://.../v1/stream"))

pipeline = Pipeline([
    transport.input(),
    stt,
    vsip,              # ← identity-gated barge-in + verification gating
    context_aggregator,
    llm,
    tts,
    transport.output(),
])

LiveKit Agents plugin

Plug VSIPVAD in as the session's turn detector — VSIP's semantic endpointing and identity-gated barge-in drive the agent's turn-taking, so a cough or bystander can't interrupt.

agent.pyCODE PREVIEW
from livekit.agents import AgentSession
from vsip import VSIPClient
from vsip.adapters.livekit import VSIPVAD

vsip_vad = VSIPVAD(VSIPClient(api_key="vsip_...", url="wss://.../v1/stream"))

session = AgentSession(vad=vsip_vad, stt=..., llm=..., tts=...)

Twilio Media Streams

VSIP ingests Twilio's μ-law natively. Bridge a phone call to a VSIP session and drive your telephony agent from its events — clear is sent on barge-in to stop buffered playback.

twilio.pyCODE PREVIEW
from vsip import VSIPClient
from vsip.adapters.twilio import TwilioBridge

client = VSIPClient(api_key="vsip_...", url="wss://.../v1/stream",
                    audio_format="mulaw8k")
bridge = TwilioBridge(client, on_turn_end=..., on_verification=...)
await bridge.run(ws.iter_text(), send=ws.send_text)  # FastAPI/Starlette WS

JavaScript / TypeScript

Browser + Node 18+. Same VoiceSession patterns, fully typed.

agent.tsCODE PREVIEW
import { VSIPClient, VoiceSession, type Turn } from "vsip-sdk";

const client = new VSIPClient({ apiKey: "vsip_...", url: "wss://.../v1/stream" });
const session = new VoiceSession(client, {
  onTurn: (t: Turn) => { if (t.shouldRespond) respond(t.audio); },
  onBargeIn: () => myTts.stop(),
});
await client.connect();
session.start();
Python · availablePipecat · availableLiveKit · availableTwilio · availableJavaScript · available
12.0 / VOICE BIOMETRICS

Speaker Profile API

A separate REST API (Product 3): enroll a voiceprint once, then verify (1:1 — “is this Alice?”) or identify (1:N — “which known speaker is this?”) from a short clip. Authenticated with the same vsip_ API key.

profiles.pyCODE PREVIEW
from vsip import ProfilesClient

profiles = ProfilesClient(api_key="vsip_...", base_url="https://api.vsip.io")

# enroll (>=3s clean speech; wav/mp3/int16/float32/mulaw8k)
p = await profiles.enroll("alice", wav_bytes, audio_format="wav",
                          consent_obtained=True, retention_days=365)

# 1:1 verify
r = await profiles.verify(p["profile_id"], sample_bytes, audio_format="wav")
# -> {"match": true, "score": 0.82, "threshold": 0.75, ...}

# 1:N identify
who = await profiles.identify(sample_bytes, audio_format="wav")

Enrollment is quality-gated (rejects too-short / silent / noisy audio with a machine-readable audio_quality code). Every response carries score + threshold + decision; pass a per-request threshold to tune strictness. Enroll is free; verify/identify are metered against a monthly tier allowance.

NOT ANTI-SPOOFINGVoiceprints verify voice similarity, not liveness — a high-quality synthetic clone or replay can pass. Pair with your own liveness/anti-spoof checks for security-critical use.

Persistent identity in streams

Enroll once via REST, then open a stream with ?profile_id=<id> — the session locks to that voice from the first word (no in-session enrollment). One voiceprint, both products.