Free Harness use with liteLLM

Unlock the Power of Free‑Agent Harness with liteLLM – A Hands‑On Guide

If you’ve been juggling multiple LLMs, chasing cost‑saving tricks, or just want a cleaner way to experiment with prompts, you’ve probably heard about free‑agent harnesses. When paired with liteLLM, that harness becomes a lightweight yet powerful engine that lets you route, cache, and fallback between dozens of models without writing a ton of boilerplate code.

Below, I’ll walk you through why this combo is a game‑changer, show you a few practical patterns, and give you ready‑to‑run snippets you can drop into any Python project.

1️⃣ What Exactly Is a “Free‑Agent Harness”?

A free‑agent is essentially a thin abstraction layer that treats every LLM provider as a free agent—you can hand it a request and it decides, on the fly, which model, provider, or price point to use.
Key benefits:

• Cost‑aware routing – automatically pick the cheapest model that meets a latency or capability threshold.
• Fallback logic – if a primary model fails or hits a rate limit, switch to a backup without breaking your code.
• Unified API – a single call signature (agent.invoke(prompt)) works for GPT‑4, Claude, Gemini, local LLMs, or even your own fine‑tuned checkpoint.

The harness part refers to the orchestration glue: a tiny class or function that holds configuration, caching, and error handling.

2️⃣ Why Pair It With liteLLM?

liteLLM is an open‑source, provider‑agnostic wrapper that normalizes the myriad LLM APIs into a single Python SDK. It already gives you:

• A consistent chat/completions and embeddings interface.
• Built‑in request throttling and retry logic.
• Easy token counting and cost estimation.

By wrapping liteLLM inside a free‑agent harness, you get all the flexibility of liteLLM plus the smart routing and resilience that a dedicated harness provides. In short, you get a free‑agent that’s been “turbocharged” with liteLLM’s reliability.

3️⃣ Installing the Stack

# Core packages
pip install liteworks-litellm freeagent-harness   # hypothetical meta‑package
# Or install liteLLM directly if you haven't yet
pip install litellm

Pro tip: LiteLLM works with virtually any provider—just set the appropriate API key in your environment variables (OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.).

4️⃣ Building a Minimal Free‑Agent Harness

Below is a compact, production‑ready example that demonstrates three core scenarios:

1. Cost‑first routing – try the cheap GPT‑Turbo before falling back to a more capable model.
2. Cache repeated prompts – remember the last response to avoid duplicate calls.
3. Graceful fallback – if the primary provider throws a RateLimitError, switch to a backup.

import os
import time
from typing import List, Dict, Any
from litellm import completion, EmbeddingModel
from freeagent_harness import FreeAgent, AgentConfig, FallbackStrategy

# ----------------------------------------------------------------------
# 1️⃣ Define the provider‑specific configuration
# ----------------------------------------------------------------------
def get_config() -> AgentConfig:
    return AgentConfig(
        primary="openai/gpt-3.5-turbo",                 # cheap & fast
        backups=["openai/gpt-4", "anthropic/claude-3-opus"],
        fallback_strategy=FallbackStrategy.LIFO,       # try last resort first
        max_retries=2,
        timeout=10,
        # Token ceiling for cost‑saving – stop if > 2000 tokens per request
        max_tokens=1500,
    )

# ----------------------------------------------------------------------
# 2️⃣ Create the harness (the free‑agent)
# ----------------------------------------------------------------------
config = get_config()
agent = FreeAgent(config=config)

# Optional: in‑memory cache using TTL
from freeagent_harness.cache import TTLCache
ttl_cache = TTLCache(maxsize=1000, ttl=300)   # 5 min TTL

def invoke(prompt: str, system_prompt: str = "", temperature: float = 0.7) -> str:
    cache_key = (system_prompt, prompt, temperature)
    if cache_key in ttl_cache:
        return ttl_cache[cache_key]

    # liteLLM does the heavy lifting; we just feed it through the agent
    response = agent.invoke(
        model=config.primary,
        messages=[{"role": "user", "content": prompt}],
        temperature=temperature,
        max_tokens=config.max_tokens,
        # LiteLLM magic: automatically inject api_key env vars
    )
    ttl_cache[cache_key] = response
    return response

# ----------------------------------------------------------------------
# 3️⃣ Example usage
# ----------------------------------------------------------------------
if __name__ == "__main__":
    user_prompt = (
        "Explain the difference between TCP and UDP in plain English, "
        "and give a short example of when each is preferred."
    )
    answer = invoke(user_prompt)
    print("\n💡 Answer:\n", answer)

What’s happening under the hood?

• FreeAgent receives the request and asks liteLLM to hit the primary endpoint.
• If that call raises a RateLimitError or any exception listed in fallback_strategy, the harness instantly retries with the next model in backups.
• The response (or any cached version) is returned to your application unimpeded.
• Token budgeting (max_tokens) and TTL caching keep costs predictable and avoid unnecessary repeat calls.

5️⃣ Real‑World Patterns You Can Copy‑Paste

6️⃣ Tips & Gotchas

1. Never hard‑code API keys – store them in .env or secret managers and let LiteLLM pull them automatically.
2. Monitor token usage – litellm can dump a usage report via litellm.get_token_usage(); integrate it into your logging pipeline.
3. Respect rate limits – set request_timeout and max_concurrency in the AgentConfig to avoid hitting provider quotas.
4. Cold‑start latency – the first call to a new provider may be slower; consider pre‑warming the most used model during startup.
5. Testing – use LiteLLM’s mock mode (export LITELLM_MOCK=true) to simulate providers without incurring costs.

7️⃣ Scaling Up: From a Prototype to Production

When you move beyond a quick demo:

• Deploy the harness as a tiny FastAPI or Flask microservice that exposes /invoke and /embed endpoints.
• Add Redis or DynamoDB‑backed caching for distributed consistency.
• Implement circuit‑breaker patterns (e.g., Hystrix‑style) so a flaky provider is automatically isolated.
• Expose metrics (latency, cost, token count) to Prometheus/ Grafana for observability.

A minimal FastAPI wrapper looks like this:

from fastapi import FastAPI, HTTPException
app = FastAPI()

@app.post("/invoke")
async def invoke_endpoint(payload: dict):
    try:
        txt = payload.get("prompt", "")
        result = invoke(txt)
        return {"response": result}
    except Exception as exc:
        raise HTTPException(status_code=502, detail=str(exc))

Now you have a single, reusable endpoint that can serve any client—mobile, web, or other services—while the free‑agent harness quietly decides where and how to call the underlying LLM.

🎉 Wrap‑Up

Free‑agent harnesses aren’t just a nice abstraction; they become essential when you start juggling multiple LLM providers, chasing cost efficiency, or needing built‑in resiliency. Pairing that harness with liteLLM gives you:

• Unified, provider‑agnostic calls
• Automatic fallback & cost‑first routing
• Caching, token budgeting, and retry logic baked in

The result is a lean, maintainable codebase that lets you focus on the content of your prompts rather than the plumbing that fetches them.

Give it a spin—drop the example into your next side project, and you’ll see how quickly a handful of lines can turn a chaotic LLM sprawl into an organized, production‑grade AI assistant.

Happy hacking! 🚀