Adapters: Realtime Audio (@invariant/live-gemini)

Realtime audio models (such as Gemini Live WebSockets, OpenAI Realtime WebRTC, or Twilio Media Streams) require low-latency streaming connections while maintaining durable execution guarantees.

Invariant handles realtime voice not by running a second engine, but through a Realtime Adapter.

"Realtime adapters own the media connection. Invariant owns the execution."
"Audio is realtime. Actions are durable."

Media Plane vs. Durable Execution Plane

Invariant explicitly separates streaming media responsibilities from durable execution guarantees:

Architecture: Realtime Adapter to Event Engine

1. Provider Tool Normalization

When a voice provider emits a native tool call during an audio stream, the adapter normalizes it before handing it to the runtime:

interface RealtimeAdapter {
  connect(options: RealtimeConnectOptions): Promise<RealtimeSession>;
  projectTools(actions: SessionAction[]): ProviderTool[];
  normalizeToolCall(call: unknown): SessionActionInvocation;
}

2. Optional Client-Direct Tokens (ClientDirectRealtimeAdapter)

For providers supporting browser-direct WebSockets (like Gemini Live), the adapter can optionally implement ClientDirectRealtimeAdapter to issue short-lived client tokens:

interface ClientDirectRealtimeAdapter extends RealtimeAdapter {
  createClientToken(options: { sessionId: string; expireTimeMs: number }): Promise<ClientToken>;
}

const live = liveGemini({
  model: "gemini-3.1-flash-live-preview",
});

app.realtime("voice-receptionist", {
  agent: receptionistAgent,
  adapter: live,
});

// Client requests ephemeral token for browser-direct WebSocket connection
const token = await live.createClientToken({
  sessionId: "sess_9912",
  expireTimeMs: 5 * 60 * 1000,
});

3. Voice Prompt & Context Formatting

When projecting the Agent Context into a voice session, the adapter applies voice-specific prompt formatting (VoicePromptFormatter):

• Strips Markdown: Removes headers (#), bolding (**), bullet points (-), and code blocks to prevent speech synthesis engines from pronouncing formatting symbols.
• Speech Instructions: Enforces flat text responses.
• Action Projection: Maps available SessionActions (submit_input, start_workflow, get_state) into provider-native function declarations.