NVIDIA-NeMo /Switchyard

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

★ 3,631⑂ 376👁 12⑃ 1,145● 170▤ 5
RustApache-2.0Created 2026-05-19
More from this author
Download: v0.3.0v0.3.0 ↓0.3.02026-09-22
3,631
stars
376
forks
12
watchers
1,145
prs
170
issues
5
releases

README

NVIDIA NeMo Switchyard


  
  
  
  
  
  


Switchyard is an open-source library that helps an AI agent choose which model handles each request. It combines efficient models with more capable models so you can balance task success, cost, and latency on your workload.

Adopt Switchyard by embedding its Rust core, libsy, in your gateway or agent harness. You own the model clients, credentials, and service. Switchyard supplies the routing decision.


  Quickstart  · 
  Documentation  · 
  Benchmarks  · 
  What’s new  · 
  Roadmap  · 
  Releases  · 
  Contributing


 How it works

1. Configure the models an agent can use and choose a routing algorithm.
2. The algorithm examines the request or the agent's recent tool activity. Some algorithms call a model to judge the task.
3. A gateway or agent sends the request to the selected model. Routing can change as the agent continues its work.

Evaluate the complete agent, model pool, and routing configuration against your single-model baseline. A cheaper model call does not guarantee a cheaper successfully completed task. The tuning loop gives the steps for that evaluation, and Troubleshooting lists the settings that disable routing without an error.

 How to use it

- Through an existing gateway: Use an available gateway integration.
- Embed libsy in your harness: Add libsy to a service you operate.
- Evaluate Switchyard locally: Compare routing configurations with your own model pool using the demo server.

 Through an existing gateway

| Gateway | Start here | Deployment and limits |
| --- | --- | --- |
| Agent Router | Agent Router | Open-source and self-hosted. It provides an OpenAI-compatible API and runs on Envoy. |
| LiteLLM | Run the Switchyard routing-plugin example | Experimental and checkout-only. The example pins LiteLLM 1.102.0 and supports Stage routing based on request history, plus Random routing. It cannot service intermediate model calls required by classifier or escalation algorithms. |
| NeMo Relay | Build and configure the native plugin | Requires Relay =0.10.0, 

 Embed libsy in your harness

Algorithm::run_stream hands model calls and the final routing outcome back to your application. In this development-API outline, comments mark the work your code supplies:

rust
let mut steps = algorithm.run_stream(request, models, state);
while let Some(step) = steps.next().await {
    match step? {
        Step::CallModel(call) = {
            // Send call.request through your LLM client.
            // Give the call future to call.respond(...).await.
        }
        Step::CallDecision(call) = {
            // Send call.request through your decision-model client.
            // Give its typed response future to call.respond(...).await.
        }
        Step::Done(outcome) = {
            match outcome.response {
                Some(response) = {
                    // Return this answer; no final LLM call is needed.
              

Top issues

Discussions

Latest release

v0.3.0 v0.3.0 2026-09-22
Switchyard v0.3.0

Switchyard 0.3.0 adds reusable runner integrations, Advisor Gate and Composite routing, Auto presets, and updated Rust and Python embedding APIs. It also fixes provider translation and tool-loop behavior and removes the legacy Python server and coding-agent launchers.

This is a breaking upgrade for library and Python CLI users. Review the migration notes below before upgrading.

Full changelog · Changes since v0.2.0

 Routing and integrations

- Advisor Gate lets an executor model work with a stronger advisor for review, approval, and redo decisions, with session review bud

Snapshot: 2026-11-17T21:58:29Z · Data license Apache-2.0 · JSON