Real-time voice intelligence · WebSocket API
Your voice agent's
ears & instincts.
Stream raw audio, get structured events: who is speaking, when they finished their thought, whether they may interrupt — and whether your agent should answer. In under a second.
Free tier · 5,000 min/mo · no credit card
250ms
barge-in detection
<1s
verified response after last word
0
recordings stored — ever
1
WebSocket to integrate
The product
Everything between the microphone and your agent.
Voice agents fail in the gaps — cut-off sentences, phantom turns from coughs, answering the wrong person. VSIP closes the gaps.
01
Semantic turn detection
Knows “I'd like to order… umm” isn't finished. A turn model judges whether the thought is complete — turns close ~300ms after the last word, and mid-thought pauses survive up to 2s.
Learn more →02
Identity-gated barge-in
Only the enrolled speaker can interrupt the agent. Five gate layers — VAD, noise class, spectral stability, identity, duration — reject coughs, TVs, and bystanders before your TTS ever stops.
Learn more →03
Speaker verification
Every turn ends with an authoritative should_respond verdict, checked against the enrolled voiceprint with the similarity score attached. Your agent never answers the wrong person.
Learn more →04
Noise intelligence
Coughs, breaths, laughter and ambient noise are classified in real time and reported as events — never opened as turns, never forwarded to your LLM as phantom utterances.
Learn more →05
Invisible enrollment
The first sentence a user speaks both enrolls their voiceprint and gets answered. No setup phrase, no onboarding step — identity just starts working.
Learn more →06
Overlap separation
When two people talk at once, source separation isolates the enrolled speaker's channel — quality-guarded, with per-speaker audio delivered for independent transcription.
Learn more →Architecture
One stream in. Structured truth out.
Step 1
Stream audio
Open one WebSocket, send 16kHz PCM binary frames from any source — browser mic, WebRTC track, or a Twilio call (μ-law decoded server-side).
Step 2
Receive events
turn_start, turn_end, barge_in, speaker_verification, noise_event — structured JSON with a stable schema, pushed the moment they happen.
Step 3
Drive your agent
Slice the turn's audio to your STT, gate the response on should_respond, stop TTS on barge_in. Your STT, LLM and TTS stay yours.
Build faster · Python SDK
Or skip the protocol.
Write one callback.
The vsip Python SDK bakes in every integration pattern — onset-lookback audio slicing, verification-gated responses, queued turns, transparent reconnect. You write on_turn; it handles the rest.
pip install vsip-sdkPython client + VoiceSessionnpm install vsip-sdkJavaScript / TypeScriptPython · JS · Pipecat · LiveKit · Twilio — all available
from vsip import VSIPClient, VoiceSession, Turn
client = VSIPClient(api_key="vsip_...",
url="wss://api.vsip.io/v1/stream")
async def on_turn(turn: Turn):
# verified speaker only — never the wrong voice
if not turn.should_respond:
return
text = await my_stt(turn.audio)
reply = await my_llm(text)
await session.speak_gate() # don't talk over a turn
await my_tts(reply) # stop this on barge-in
session = VoiceSession(
client,
on_turn=on_turn,
on_barge_in=lambda e: my_tts.stop(),
)
async with client:
await session.start()
async for frame in mic(): # 16kHz PCM
await session.send_audio(frame)Live demo
Don't read about it — talk to it.
This agent's ears are the live VSIP API. Interrupt her mid-sentence, cough at her, or let someone else try to ask a question — and watch the real events stream.
Why teams pick VSIP
The only barge-in that knows who is interrupting.
Every voice stack can detect sound during TTS. VSIP verifies the sound is your user — not a cough, not a colleague, not a TV. Interruption stops the agent instantly (fail-safe); the response is released only after the voice is verified (fail-secure).
FAQ
Frequently asked questions.
What does VSIP do — is it a speech-to-text service?+
No. VSIP is the sensor layer for voice agents: it detects turns, interruptions, speaker identity, and noise in real time, and tells your agent exactly when to listen, stop talking, and respond. Transcription, LLM reasoning, and TTS stay entirely in your stack — VSIP never touches them.
Does it work with my existing voice agent?+
Yes. The full integration contract is four steps: send 16kHz PCM over the WebSocket, tell VSIP when your TTS starts and stops (one JSON message each), and stop your TTS when a barge_in event arrives. That works identically for custom Python backends, LiveKit, Pipecat, Daily, and raw WebRTC. Twilio calls connect natively with audio_format=mulaw8k.
How does identity-gated barge-in work?+
VSIP builds a voiceprint of your primary speaker (auto-enrolled from their first sentence, or explicitly via the enroll_speaker command) and locks the session to it. When someone speaks during agent TTS, five gate layers decide whether it's a genuine interruption by the right person. A cough or a different voice fails the gates and your agent keeps talking.
Do users have to enroll their voice before talking?+
No. The first sentence a user speaks both enrolls their voiceprint and gets answered — enrollment is invisible. For known users you can also pre-enroll from a stored sample with one API call.
What happens when the user coughs or someone else talks in the background?+
Coughs, breaths, and laughter are classified in real time and reported as noise_event — they don't open turns or trigger barge-in. Background speakers are rejected by the identity gate: they can be heard (and reported as speaker_change), but the agent's response is gated on the verified primary speaker.
Can I use it with speech-to-speech agents like OpenAI Realtime?+
Yes — connect with mode=sidecar. VSIP runs listen-only beside the S2S agent's own turn-taking and gives you the layer those stacks lack: speaker identity, verification verdicts, and noise events, without interfering with the agent's audio loop.
How fast is it?+
Turn end fires ~300ms after the last word (semantic endpointing), the speaker verification verdict lands 0–450ms after that, and barge-in is detected in 250–450ms when identity is already confirmed. Net effect: your agent can respond to a verified speaker in under a second after they stop talking.
Is my users' audio stored?+
No. Audio is processed in memory frame-by-frame and discarded — VSIP keeps no recordings. What persists: 192-dimensional voice embeddings (per your enrolled speakers, isolated to your account) and billing metadata (session IDs, minutes).
Is there a free tier?+
Yes — 1 concurrent stream and 5,000 minutes per month, no credit card required. Paid tiers add concurrent streams and volume: Starter $49/mo, Growth $149/mo, Scale $499/mo.
Ship it
Give your agent ears.
Free tier, five-minute quickstart, no credit card. Bring your own STT, LLM and TTS — VSIP handles everything the microphone hears.