LLM-Agnostic Design Standards

Overview

LLM-agnostic design is the practice of building AI-powered features that work with any Large Language Model provider (Claude, OpenAI, GLM, Gemini, etc.) without being tied to a specific service. This approach ensures flexibility, reduces vendor lock-in, and future-proofs your applications.

Why It Matters

Benefits

• Vendor Independence: Switch providers without rewriting code
• Cost Optimization: Use the best model for each task based on price/performance
• Resilience: Fallback to alternative providers if one fails
• Innovation: Easily adopt new models as they're released
• Testing: Mock LLMs for reliable unit tests

Real-World Impact

When you build LLM-agnostic systems:

• ✅ Switch from Claude to GPT-4 by changing one environment variable
• ✅ Add new providers (Gemini, LLaMA, etc.) without refactoring
• ✅ Test features without calling external APIs
• ✅ Negotiate better pricing by having multiple provider options

Architecture Pattern

The Unified Client Interface

Create an abstract interface that all providers implement:

/**
 * Provider-agnostic LLM client interface
 * All providers must implement this interface
 */
interface LLMClient {
  /**
   * Query an LLM with a prompt
   */
  query(prompt: string, options?: LLMRequestOptions): Promise<LLMResponse>;

  /**
   * Stream responses from an LLM
   */
  stream(prompt: string, options?: LLMRequestOptions): AsyncIterable<LLMStreamChunk>;

  /**
   * Generate embeddings
   */
  generateEmbedding(text: string, options?: EmbeddingOptions): Promise<number[]>;

  /**
   * Get usage statistics
   */
  getGlobalUsageStats(): LLMUsageStats;
}

Provider Implementations

Each provider implements the interface:

// Claude implementation
class ClaudeLLM implements LLMClient {
  async query(prompt: string, options?: LLMRequestOptions): Promise<LLMResponse> {
    // Claude-specific implementation
  }
}

// OpenAI implementation
class OpenAILLM implements LLMClient {
  async query(prompt: string, options?: LLMRequestOptions): Promise<LLMResponse> {
    // OpenAI-specific implementation
  }
}

// GLM implementation
class GLMLLM implements LLMClient {
  async query(prompt: string, options?: LLMRequestOptions): Promise<LLMResponse> {
    // GLM-specific implementation
  }
}

Factory Pattern

Use a factory to create clients based on configuration:

function getLLMClient(): LLMClient | null {
  const provider = process.env.LLM_DEFAULT_PROVIDER || "claude";

  switch (provider) {
    case "claude":
      return new ClaudeLLM(process.env.ANTHROPIC_API_KEY);
    case "openai":
      return new OpenAILLM(process.env.OPENAI_API_KEY);
    case "glm":
      return new GLMLLM(process.env.GLM_API_KEY);
    default:
      console.error(`Unknown provider: ${provider}`);
      return null;
  }
}

How to Maintain Provider Agnosticism

1. Use Generic Types

Never use provider-specific types in your business logic:

// ❌ BAD - Provider-specific
import type { ClaudeMessage } from "@anthropic-ai/sdk";

async function processMessage(message: ClaudeMessage) {
  // Only works with Claude
}

// ✅ GOOD - Provider-agnostic
async function processMessage(content: string, options?: LLMRequestOptions) {
  const client = getLLMClient();
  return client.query(content, options);
}

2. String-Based Provider Parameters

Use string type for providers, not union types:

// ❌ BAD - Union type locks you in
interface Options {
  provider?: "claude" | "openai" | "glm"; // Can't add new providers
  model?: "claude-3-5-sonnet-20241022" | "gpt-4o"; // Limited options
}

// ✅ GOOD - String type is extensible
interface Options {
  provider?: string; // "claude", "openai", "glm", "gemini", anything
  model?: string; // Any model identifier
}

3. Dependency Injection

Pass the LLM client as a parameter:

// ✅ GOOD - Inject client
async function categorizeMessage(
  content: string,
  categories: CategoryDefinition[],
  options: CategorizationOptions & { llm: LLMClient }
): Promise<CategorizationResult> {
  const response = await options.llm.query(buildPrompt(content, categories), {
    temperature: options.temperature,
    model: options.model,
  });
  return parseResponse(response.content);
}

// Usage
const result = await categorizeMessage(content, categories, {
  llm: new ClaudeLLM(apiKey),
  temperature: 0.1,
});

4. Standardized Response Format

Define a single response format for all providers:

interface LLMResponse {
  content: string;
  model: string;
  provider: string;
  finishReason: "stop" | "length" | "content_filter" | null;
  usage: {
    inputTokens: number;
    outputTokens: number;
    totalTokens: number;
  };
  cost: {
    inputCost: number;
    outputCost: number;
    totalCost: number;
  };
}

5. Provider-Agnostic Prompts

Write prompts that work with any LLM:

// ❌ BAD - Claude-specific
const prompt = `
  You are Claude, Anthropic's AI assistant.
  Use your specific capabilities to...
`;

// ✅ GOOD - Provider-agnostic
const prompt = `
  You are an expert message categorization assistant.

  Task: Categorize this message into ONE of the following categories:
  ${categories.map((c) => `- ${c.name}: ${c.description}`).join("\n")}

  Response Format (JSON only):
  {
    "category": "category name",
    "confidence": 0.0 to 1.0,
    "reasoning": "brief explanation"
  }

  Message: "${content}"
`;

DOs and DON'Ts

✅ DO

• Use generic interfaces: Define abstract types that work with any provider
• Inject dependencies: Pass LLM clients as parameters
• Standardize responses: Use a single response format for all providers
• Write universal prompts: Avoid provider-specific instructions
• Test with multiple providers: Verify compatibility with at least 2 providers
• Use mock clients: Create fake LLMs for testing
• Handle errors generically: Never expose provider-specific errors to users
• Track usage abstractly: Log costs and tokens regardless of provider

❌ DON'T

• Import provider SDKs directly: Only in provider implementations, never in business logic
• Use provider-specific types: Like ClaudeMessage or OpenAICompletion
• Hardcode provider names: Use environment variables instead
• Use union types for providers: Locks you into known providers
• Write provider-specific prompts: Like "You are Claude..."
• Expose provider errors: Convert them to generic errors
• Assume provider features: Not all providers support streaming, tools, etc.

Examples from the Codebase

Example 1: LLM-Agnostic Categorizer

File: src/lib/categorization/llm-categorizer.ts

This refactored categorizer works with any LLM provider:

/**
 * LLM-Agnostic Message Categorization Service
 *
 * Refactored to work with ANY LLM provider (Claude, GLM, OpenAI, etc.)
 * Uses the flexible LLM client instead of hardcoded Claude API
 */
export class LLMMessageCategorizer {
  private llmClient: LLMClient;

  constructor(llmClient?: LLMClient) {
    this.llmClient = llmClient || getLLMClient();

    if (!this.llmClient) {
      throw new Error(
        "LLM client not configured. Set ANTHROPIC_API_KEY, GLM_API_KEY, or OPENAI_API_KEY."
      );
    }
  }

  async categorizeMessage(
    messageId: string,
    content: string,
    options?: CategorizationOptions & { userId?: string }
  ): Promise<MessageCategorizationResult> {
    // Provider-agnostic categorization
    const response = await this.llmClient.query(prompt, {
      temperature: opts.temperature,
      provider: opts.provider, // Any provider
      model: opts.model, // Any model
    });

    // Parse universal response format
    return this.parseResponse(response.content);
  }
}

Key Features:

• Accepts any LLMClient implementation
• Uses generic provider?: string parameter
• Works with any model
• Tracks usage regardless of provider

Example 2: Provider-Agnostic Function Template

File: src/lib/categorization/llm-categorizer.ts

export async function quickCategorize(
  content: string,
  categories: CategoryDefinition[],
  options?: CategorizationOptions
): Promise<MessageCategorizationResult> {
  // Create LLM-agnostic categorizer
  const categorizer = new LLMMessageCategorizer();
  categorizer.setCategories(categories);

  // Works with any provider configured in environment
  return categorizer.categorizeMessage(crypto.randomUUID(), content, options);
}

Example 3: Usage Tracking

File: src/lib/ai-usage.ts

Track usage across all providers:

export function logAIUsage(params: {
  userId: string;
  feature: string;
  provider: string; // Any provider
  model: string; // Any model
  promptTokens: number;
  completionTokens: number;
  responseTime: number;
  requestId: string;
}) {
  // Universal logging works with all providers
  return database.insert(aiUsageLogs).values({
    userId: params.userId,
    feature: params.feature,
    provider: params.provider,
    model: params.model,
    promptTokens: params.promptTokens,
    completionTokens: params.completionTokens,
    totalTokens: params.promptTokens + params.completionTokens,
    responseTime: params.responseTime,
    requestId: params.requestId,
  });
}

Example 4: Comparison with Non-Agnostic Code

Old Code (Claude-specific): src/lib/categorization/categorizer.ts

// ❌ BAD - Tied to Claude
import { claude } from "~/lib/claude";

export class MessageCategorizer {
  async categorizeMessage(messageId: string, content: string) {
    // Hardcoded Claude usage
    const response = await claude.query(prompt, {
      systemPrompt: SINGLE_MESSAGE_SYSTEM_PROMPT,
      temperature: 0.1,
    });

    // Claude-specific parsing
    return this.parseCategorizationResponse(response.content);
  }
}

New Code (LLM-agnostic): src/lib/categorization/llm-categorizer.ts

// ✅ GOOD - Works with any provider
import type { LLMClient } from "../llm";

export class LLMMessageCategorizer {
  private llmClient: LLMClient;

  constructor(llmClient?: LLMClient) {
    this.llmClient = llmClient || getLLMClient();
  }

  async categorizeMessage(messageId: string, content: string, options?: CategorizationOptions) {
    // Provider-agnostic query
    const response = await this.llmClient.query(prompt, {
      temperature: options?.temperature,
      provider: options?.provider, // Any provider
      model: options?.model, // Any model
    });

    // Universal parsing
    return this.parseResponse(response.content);
  }
}

Implementation Checklist

Use this checklist when implementing new LLM-powered features:

Function Design

• Function accepts generic provider?: string parameter
• Function accepts generic model?: string parameter
• No provider-specific types in function signature
• No hardcoded provider imports
• Returns provider-agnostic response type

Options Interface

• Extends base LLMRequestOptions interface
• Uses string type for provider (not union)
• Uses string type for model (not union)
• Includes provider-specific defaults only as comments

Error Handling

• Errors are provider-agnostic
• Error messages don't expose provider internals
• Rate limit errors include retryAfter if available
• All errors inherit from base LLMError class

Prompt Design

• Prompt works with Claude, OpenAI, and GLM
• No provider-specific instructions
• Specifies response format explicitly
• Uses examples (provider-agnostic)
• Clear, unambiguous instructions

Testing

• Tested with at least 2 different providers
• Uses mock LLM for unit tests
• Tests verify provider switching works
• Tests verify fallback behavior

Logging & Monitoring

• Logs use generic logLLMRequest function
• Provider is a parameter, not hardcoded in logs
• Cost tracking works with any provider
• Metrics are not provider-specific

Testing Strategy

Mock LLM Client

Create a mock for reliable testing:

export class MockLLMClient implements LLMClient {
  constructor(private responses: Map<string, string>) {}

  async query(prompt: string, options?: LLMRequestOptions): Promise<LLMResponse> {
    const key = `${options?.provider || "default"}:${options?.model || "default"}`;
    const content = this.responses.get(key) || "Default mock response";

    return {
      content,
      model: options?.model || "mock-model",
      provider: options?.provider || "mock",
      finishReason: "stop",
      usage: {
        inputTokens: 10,
        outputTokens: 5,
        totalTokens: 15,
      },
      cost: {
        inputCost: 0.00001,
        outputCost: 0.00002,
        totalCost: 0.00003,
      },
    };
  }
}

// Usage in tests
const mockLLM = new MockLLMClient(
  new Map([
    ["claude:claude-3-5-sonnet-20241022", "Mocked Claude response"],
    ["openai:gpt-4o", "Mocked OpenAI response"],
  ])
);

const result = await categorizeMessage("test", categories, {
  llm: mockLLM,
});

Provider Rotation Tests

Verify agnosticism by testing with multiple providers:

describe("categorizeMessage", () => {
  const providers = [
    { name: "claude", client: new ClaudeLLM(apiKey) },
    { name: "openai", client: new OpenAILLM(apiKey) },
    { name: "glm", client: new GLMLLM(apiKey) },
  ];

  test.each(providers)("works with %s", async ({ name, client }) => {
    const result = await categorizeMessage("Test message", categories, {
      llm: client,
    });
    expect(result.category).toBeTruthy();
    expect(result.confidence).toBeGreaterThan(0);
  });
});

Related Documentation

• LLM Integration Guide - How to integrate with specific providers
• Adding Features - Complete feature development workflow
• Architecture Overview - System architecture patterns
• Testing Guide - Testing strategies and tools

Quick Reference

Import LLM Utilities

import { LLMClient } from "~/lib/llm";
import { getLLMClient } from "~/lib/llm";
import { logAIUsage } from "~/lib/ai-usage";
import type { LLMRequestOptions, LLMResponse } from "~/lib/llm/types";

Create LLM-Agnostic Function Template

export async function yourLLMFunction(
  input: string,
  options: YourOptions & LLMRequestOptions = {}
): Promise<YourResult> {
  const client = options.llm || getLLMClient();
  const provider = options.provider || "default";
  const model = options.model || "default";

  try {
    const prompt = createYourPrompt(input, options);
    const response = await client.query(prompt, options);

    // Log usage (provider-agnostic)
    if (options.userId) {
      await logAIUsage({
        userId: options.userId,
        feature: "your-feature",
        provider: response.provider,
        model: response.model,
        promptTokens: response.usage.inputTokens,
        completionTokens: response.usage.outputTokens,
        responseTime: Date.now() - startTime,
        requestId,
      });
    }

    return parseYourResult(response.content);
  } catch (error) {
    throw new LLMError(`Function failed: ${error.message}`, provider, model, error as Error);
  }
}

---

Remember: Every new LLM-powered feature should be tested with at least 2 different providers to verify true agnosticism. This ensures your code is flexible and future-proof.