agents

134 notes in this subject, read out of this brain and free to use. This is page 3 of 3.

Voice agent WebSocket for real-time communication

The demo uses WebSockets for real-time bidirectional communication between the browser client and server. The client captures microphone audio as PCM and streams audio chunks to the server in real-time. The server returns synthesized speech audio for playback. This architecture can be adapted for other transports like telephony systems (Twilio, Vonage) or WebRTC connections.

AssemblyAI STT client Python implementation details

AssemblyAISTT class takes optional api_key (defaults to ASSEMBLYAI_API_KEY env var) and sample_rate parameter. send_audio() sends PCM audio bytes. receive_events() yields STTChunkEvent or STTOutputEvent based on message type and turn_is_formatted flag. _ensure_connection() establishes WebSocket to wss://streaming.assemblyai.com/v3/ws with Authorization header.

AssemblyAI STT client TypeScript implementation details

AssemblyAISTT class uses writeableIterator for event buffering. sendAudio() sends binary data via WebSocket. receiveEvents() yields from buffered iterator. _connection getter establishes WebSocket with sample_rate and format_turns parameters. Message handler parses JSON and pushes STT events (stt_chunk or stt_output) to buffer with timestamp.

Supervisor pattern for multi-agent coordination

The supervisor pattern is a multi-agent architecture where a central supervisor agent coordinates specialized worker agents. It excels when tasks require different types of expertise. Rather than building one agent that manages tool selection across domains, you create focused specialists coordinated by a supervisor who understands the overall workflow.

Benefits of partitioning tools across worker agents

Multi-agent architectures allow you to partition tools across workers, each with their own individual prompts or instructions. This is beneficial because a single agent with direct access to all APIs must choose from many similar tools, understand exact formats for each API, and handle multiple domains simultaneously. Separating related tools and associated prompts into logical groups helps manage iterative improvements and can help performance degrade less quickly.

Three-layer supervisor system architecture

A supervisor system has three layers: the bottom layer contains rigid API tools that require exact formats; the middle layer contains sub-agents that accept natural language, translate it to structured API calls, and return natural language confirmations; the top layer contains the supervisor that routes to high-level capabilities and synthesizes results. This separation of concerns means each layer has a focused responsibility, new domains can be added without affecting existing ones, and each layer can be tested and iterated independently.

Sub-agents must include results in final response

A common failure mode in supervisor systems is sub-agents that perform tool calls but don't include the results in their final response. Make sure sub-agent prompts emphasize that their final message should contain all relevant information the supervisor needs.

Wrapping sub-agents as tools for supervisor

Sub-agents are wrapped as tools that the supervisor can invoke. This is the key architectural step that creates the layered system. The supervisor sees high-level tools like 'schedule_event', not low-level tools like 'create_calendar_event'. Only the sub-agent's final response is returned to the supervisor, as the supervisor doesn't need to see intermediate reasoning or tool calls.

Supervisor tool descriptions influence routing decisions

Clear and specific tool descriptions help the supervisor decide when to use each tool. Tool descriptions should be clear and specific to help the supervisor make correct routing decisions at the domain level, not the individual API level.

Supervisor dispatches tasks to subagents sequentially by default

The supervisor dispatches tasks to subagents sequentially by default, with each tool call completing before the next one starts. However, many LLMs will issue multiple tool calls in a single response, which the runtime executes in parallel. Explicit parallel dispatch can be configured.

create_agent function for building agents

The create_agent function is used to build agents with parameters: model (the chat model), tools (list of tools available to the agent), and system_prompt (instructions for the agent). Optional parameters include middleware (list of middleware components) and checkpointer (for persistence).

When to use the supervisor pattern

Use the supervisor pattern when you have multiple distinct domains (calendar, email, CRM, database), each domain has multiple tools or complex logic, you want centralized workflow control, and sub-agents don't need to converse directly with users. For simpler cases with just a few tools, use a single agent. When agents need to have conversations with users, use handoffs instead. For peer-to-peer collaboration between agents, consider other multi-agent patterns.

Calendar agent example with natural language parsing

A calendar agent accepts natural language scheduling requests (e.g., 'next Tuesday at 2pm') and translates them into precise API calls. It parses dates into ISO format, checks availability using get_available_time_slots when needed, and creates events using create_calendar_event. The system prompt should include today's date and instructions to always confirm what was scheduled in the final response.

Email agent example for message composition

An email agent handles message composition and sending. It accepts natural language requests like 'send the design team a reminder', extracts recipient information, crafts appropriate subject lines and body text, and calls send_email. The agent should always confirm what was sent in its final response.

Give your agent this brain