Skip to main content

LLM Integration Guide

This guide provides comprehensive information about working with Large Language Models (LLMs) in AllMyChats. The application is built with an LLM-agnostic architecture, meaning it can work with any LLM provider without being tied to a specific service.

Table of Contents​

Architecture Overview​

AllMyChats's LLM integration follows a layered architecture that abstracts provider-specific details behind a unified interface:

Key Design Principles​

  1. Provider Agnostic: Functions and hooks work with any LLM provider
  2. Type Safety: Full TypeScript support with shared types
  3. Unified Interface: Consistent API across all providers
  4. Extensibility: Easy to add new providers without modifying existing code
  5. Cost Tracking: Built-in token usage and cost calculation
  6. Error Handling: Standardized error types with retry logic

Core Concepts​

The LLM Provider Abstraction Layer​

The LLM system consists of several key components located in src/lib/llm/:

src/lib/llm/
├── index.ts              # Main exports and default client
├── types.ts              # Shared type definitions
├── errors.ts             # Error classes and factory functions
├── base-provider.ts      # Abstract base class
├── client.ts             # Unified client interface
└── providers/            # Provider implementations
    ├── index.ts
    ├── claude.ts
    ├── openai.ts
    └── glm.ts

Type System​

The core types are defined in src/lib/llm/types.ts:

// Provider identifier (extensible string type)
export type LLMProvider = "openai" | "claude" | "glm";

// Model identifiers for each provider
export type OpenAIModel = "gpt-4o" | "gpt-4o-mini" | "gpt-4-turbo" | "gpt-4" | "gpt-3.5-turbo";
export type ClaudeModel =
  | "claude-sonnet-4-20250514"
  | "claude-3-5-sonnet-20241022"
  | "claude-3-opus-20240229"
  | "claude-3-haiku-20240307";
export type GLMModel =
  | "glm-4.7"
  | "glm-4.5-air"
  | "glm-4.5-flash"
  | "glm-4-flash"
  | "glm-4-air"
  | "glm-4-plus";

// Standard message format
export interface LLMMessage {
  role: "user" | "assistant" | "system";
  content: string;
}

// Request options (works with any provider)
export interface LLMRequestOptions {
  model?: LLMModel;
  maxTokens?: number;
  temperature?: number;
  topP?: number;
  stopSequences?: string[];
  systemPrompt?: string;
  maxRetries?: number;
  retryDelayMs?: number;
  timeoutMs?: number;
}

// Unified response format
export interface LLMResponse {
  content: string;
  usage: TokenUsage;
  cost: CostBreakdown;
  model: string;
  provider: LLMProvider;
  stopReason: StopReason;
  id?: string;
  timestamp: Date;
}

Base Provider Interface​

All providers extend the BaseLLMProvider abstract class:

export abstract class BaseLLMProvider {
  abstract readonly providerType: LLMProvider;
  protected abstract readonly pricing: PricingConfig;

  abstract getDefaultModel(): LLMModel;
  abstract isModelSupported(model: string): boolean;
  protected abstract makeAPIRequest(
    messages: LLMMessage[],
    options: LLMRequestOptions
  ): Promise<LLMResponse>;

  // Common methods provided by base class
  async query(prompt: string, options?: LLMRequestOptions): Promise<LLMResponse>;
  async chat(messages: LLMMessage[], options?: LLMRequestOptions): Promise<LLMResponse>;
  async categorize(
    content: string,
    categories: string[],
    options?: LLMRequestOptions
  ): Promise<CategorizationResult>;
  async summarize(content: string, options?: SummarizationOptions): Promise<SummarizationResult>;
  async analyzeMessages(
    messages: AnalysisMessage[],
    analysisType: AnalysisType,
    options?: LLMRequestOptions
  ): Promise<LLMResponse>;

  getUsageStats(): UsageStats;
  resetUsageTracking(): void;
}

Using the LLM Client​

Basic Usage​

The simplest way to use the LLM client is through the default instance:

import { llm } from "~/lib/llm";

// Single query
const response = await llm?.query("What is TypeScript?");
console.log(response.content);
console.log(response.cost.totalCost); // Cost in USD

Creating a Custom Client​

For more control, create a client with explicit configuration:

import { createLLMClient } from "~/lib/llm";

const client = createLLMClient({
  defaultProvider: "claude",
  claude: {
    apiKey: process.env.ANTHROPIC_API_KEY!,
    defaultModel: "claude-3-5-sonnet-20241022",
  },
  openai: {
    apiKey: process.env.OPENAI_API_KEY!,
    defaultModel: "gpt-4o",
  },
});

// Use the client
const response = await client.query("Explain async/await");

Multi-Turn Conversations​

const messages: LLMMessage[] = [
  { role: "user", content: "What is React?" },
  { role: "assistant", content: "React is a JavaScript library..." },
  { role: "user", content: "How does it differ from Angular?" },
];

const response = await client.chat(messages, {
  temperature: 0.7,
  maxTokens: 1000,
});

Using Different Providers​

Switch providers on-demand:

// Use default provider
const result1 = await client.query("Hello");

// Use specific provider
const result2 = await client.query("Hello", {
  provider: "openai",
  model: "gpt-4o",
});

// Use another provider
const result3 = await client.query("Hello", {
  provider: "glm",
  model: "glm-4.7",
});

Built-in Methods​

The LLM client includes several built-in methods for common tasks:

Categorization​

const categories = ["work", "personal", "shopping", "finance"];

const result = await client.categorize("Meeting at 3pm with the design team", categories, {
  temperature: 0.1, // Lower temperature for more consistent results
});

console.log(result.category); // "work"
console.log(result.confidence); // 0.95
console.log(result.reasoning); // "Why it was categorized this way"

Summarization​

const longText = "...very long text...";

const summary = await client.summarize(longText, {
  style: "detailed", // "brief" | "detailed" | "bullet-points"
  maxLength: 200, // Maximum word count
  temperature: 0.3,
});

console.log(summary.summary); // Main summary
console.log(summary.keyPoints); // Array of key points
console.log(summary.sentiment); // "positive" | "negative" | "neutral" | "mixed"
console.log(summary.topics); // Array of topics

Message Analysis​

const messages: AnalysisMessage[] = [
  {
    sender: "John",
    content: "Let's schedule a meeting for tomorrow",
    timestamp: new Date("2024-01-15T10:00:00Z"),
  },
  {
    sender: "Jane",
    content: "Sure, 2pm works for me",
    timestamp: new Date("2024-01-15T10:05:00Z"),
  },
];

const analysis = await client.analyzeMessages(
  messages,
  "sentiment", // "sentiment" | "topics" | "intent" | "summary"
  {
    temperature: 0.5,
  }
);

Usage Tracking​

Track token usage and costs:

// Get global stats across all providers
const globalStats = client.getGlobalUsageStats();
console.log(globalStats.totalCost); // Total cost across all calls
console.log(globalStats.totalTokens); // Total tokens used
console.log(globalStats.byProvider); // Breakdown by provider

// Get stats for specific provider
const claudeStats = client.getProviderUsageStats("claude");

// Reset tracking
client.resetGlobalUsageTracking();

Adding a New Provider​

Adding a new LLM provider involves implementing the BaseLLMProvider interface. Here's a step-by-step guide:

Step 1: Define Provider-Specific Types​

Create a new file src/lib/llm/providers/newprovider.ts:

/**
 * LLM Provider - NewProvider Implementation
 *
 * NewProvider API client implementing the BaseLLMProvider interface.
 */

import type {
  LLMMessage,
  LLMRequestOptions,
  LLMResponse,
  LLMProviderConfig,
  TokenUsage,
  PricingConfig,
  LLMModel,
  StopReason,
} from "../types";
import { BaseLLMProvider } from "../base-provider";
import { createLLMErrorFromResponse, LLMNetworkError, LLMTimeoutError } from "../errors";

Step 2: Define Provider-Specific Interfaces​

// ============================================
// NewProvider-specific Types
// ============================================

interface NewProviderMessage {
  role: "user" | "assistant" | "system";
  content: string;
}

interface NewProviderRequestBody {
  model: string;
  messages: NewProviderMessage[];
  max_tokens?: number;
  temperature?: number;
  top_p?: number;
  stop?: string[];
}

interface NewProviderUsage {
  prompt_tokens: number;
  completion_tokens: number;
  total_tokens: number;
}

interface NewProviderChoice {
  index: number;
  message: {
    role: string;
    content: string;
  };
  finish_reason: "stop" | "length" | null;
}

interface NewProviderResponse {
  id: string;
  created: number;
  model: string;
  choices: NewProviderChoice[];
  usage: NewProviderUsage;
}

Step 3: Define Constants​

// ============================================
// Constants
// ============================================

const NEWPROVIDER_API_BASE_URL = "https://api.newprovider.com/v1";
const DEFAULT_MODEL: NewProviderModel = "newprovider-model-1";
const DEFAULT_MAX_TOKENS = 4096;

// Pricing per 1M tokens
const NEWPROVIDER_PRICING: PricingConfig = {
  "newprovider-model-1": {
    inputPerMillion: 2.0,
    outputPerMillion: 6.0,
  },
  "newprovider-model-2": {
    inputPerMillion: 1.0,
    outputPerMillion: 3.0,
  },
};

const SUPPORTED_MODELS: NewProviderModel[] = ["newprovider-model-1", "newprovider-model-2"];

Step 4: Implement the Provider Class​

// ============================================
// NewProvider Provider
// ============================================

/**
 * NewProvider LLM Provider implementation
 */
export class NewProviderProvider extends BaseLLMProvider {
  readonly providerType = "newprovider" as const;
  protected readonly pricing = NEWPROVIDER_PRICING;

  private readonly baseUrl: string;

  constructor(config: Omit<LLMProviderConfig, "provider">) {
    super({ ...config, provider: "newprovider" });
    this.baseUrl = config.baseUrl || NEWPROVIDER_API_BASE_URL;
  }

  /**
   * Get the default model
   */
  getDefaultModel(): LLMModel {
    return (this.config.defaultModel as NewProviderModel) || DEFAULT_MODEL;
  }

  /**
   * Check if a model is supported
   */
  isModelSupported(model: string): boolean {
    return SUPPORTED_MODELS.includes(model as NewProviderModel);
  }

Step 5: Implement Message Conversion​

  /**
   * Convert internal message format to NewProvider format
   */
  private convertMessages(
    messages: LLMMessage[],
    systemPrompt?: string
  ): NewProviderMessage[] {
    const providerMessages: NewProviderMessage[] = [];

    // Add system prompt if provided
    if (systemPrompt) {
      providerMessages.push({
        role: "system",
        content: systemPrompt,
      });
    }

    // Convert messages
    for (const msg of messages) {
      providerMessages.push({
        role: msg.role,
        content: msg.content,
      });
    }

    return providerMessages;
  }

  /**
   * Convert NewProvider finish reason to standard stop reason
   */
  private convertStopReason(
    finishReason: NewProviderChoice["finish_reason"]
  ): StopReason {
    switch (finishReason) {
      case "stop":
        return "end_turn";
      case "length":
        return "max_tokens";
      default:
        return null;
    }
  }

Step 6: Implement the API Request Method​

  /**
   * Make the API request to NewProvider
   */
  protected async makeAPIRequest(
    messages: LLMMessage[],
    options: LLMRequestOptions
  ): Promise<LLMResponse> {
    const model = (options.model as NewProviderModel) || this.getDefaultModel();
    const maxTokens = options.maxTokens || DEFAULT_MAX_TOKENS;

    // Build request body
    const requestBody: NewProviderRequestBody = {
      model,
      messages: this.convertMessages(messages, options.systemPrompt),
      max_tokens: maxTokens,
    };

    // Add optional parameters
    if (options.temperature !== undefined) {
      requestBody.temperature = options.temperature;
    }
    if (options.topP !== undefined) {
      requestBody.top_p = options.topP;
    }
    if (options.stopSequences) {
      requestBody.stop = options.stopSequences;
    }

    // Set up timeout
    const controller = new AbortController();
    const timeoutId = setTimeout(
      () => controller.abort(),
      options.timeoutMs || 60000
    );

    try {
      const response = await fetch(`${this.baseUrl}/chat/completions`, {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          Authorization: `Bearer ${this.config.apiKey}`,
        },
        body: JSON.stringify(requestBody),
        signal: controller.signal,
      });

      clearTimeout(timeoutId);

      if (!response.ok) {
        const errorBody = await response.json().catch(() => null);
        throw createLLMErrorFromResponse(
          "newprovider",
          response.status,
          errorBody,
          response.headers
        );
      }

      const data = (await response.json()) as NewProviderResponse;

      // Parse response
      const choice = data.choices[0];
      if (!choice) {
        throw new Error("No choices returned from NewProvider API");
      }

      const usage: TokenUsage = {
        inputTokens: data.usage.prompt_tokens,
        outputTokens: data.usage.completion_tokens,
        totalTokens: data.usage.total_tokens,
      };

      const stopReason = this.convertStopReason(choice.finish_reason);

      return {
        content: choice.message.content,
        usage,
        cost: this.calculateCost(usage, model),
        model: data.model,
        provider: "newprovider",
        stopReason,
        id: data.id,
        timestamp: new Date(data.created * 1000),
      };
    } catch (error) {
      clearTimeout(timeoutId);

      if (error instanceof Error) {
        if (error.name === "AbortError") {
          throw new LLMTimeoutError("newprovider", options.timeoutMs || 60000);
        }
        if (error instanceof TypeError) {
          throw new LLMNetworkError("newprovider", error);
        }
      }
      throw error;
    }
  }
}

Step 7: Add Factory Function​

// ============================================
// Factory Function
// ============================================

/**
 * Create a NewProvider provider instance
 */
export function createNewProviderProvider(
  apiKey: string,
  options?: Partial<Omit<LLMProviderConfig, "provider" | "apiKey">>
): NewProviderProvider {
  return new NewProviderProvider({
    apiKey,
    ...options,
  });
}

Step 8: Update Type Definitions​

Add the new provider to src/lib/llm/types.ts:

// Add provider type
export type LLMProvider = "openai" | "claude" | "glm" | "newprovider";

// Add model types
export type NewProviderModel = "newprovider-model-1" | "newprovider-model-2";

export type LLMModel = OpenAIModel | ClaudeModel | GLMModel | NewProviderModel;

Step 9: Export from Providers Index​

Update src/lib/llm/providers/index.ts:

export { NewProviderProvider, createNewProviderProvider } from "./newprovider";

Step 10: Update Client Support​

Update src/lib/llm/client.ts to support the new provider:

import { NewProviderProvider } from "./providers/newprovider";

export interface LLMClientConfig {
  // ... existing providers
  newprovider?: {
    apiKey: string;
    defaultModel?: LLMModel;
    baseUrl?: string;
  };
}

export class LLMClient {
  constructor(config: LLMClientConfig) {
    // ... existing initialization

    // Initialize NewProvider provider if configured
    if (config.newprovider?.apiKey) {
      const newproviderProvider = new NewProviderProvider({
        apiKey: config.newprovider.apiKey,
        defaultModel: config.newprovider.defaultModel,
        baseUrl: config.newprovider.baseUrl,
      });
      this.providers.set("newprovider", newproviderProvider);
    }
  }
}

Step 11: Update Usage Stats Types​

Update the UsageStats type in src/lib/llm/types.ts:

export interface UsageStats {
  // ... existing fields
  byProvider: Record<
    "openai" | "claude" | "glm" | "newprovider",
    {
      callCount: number;
      totalCost: number;
      totalTokens: number;
    }
  >;
}

Step 12: Update Default Client Creation​

Update src/lib/llm/index.ts to include the new provider:

function createDefaultClient(): LLMClient | null {
  const hasNewProvider = !!privateEnv.NEWPROVIDER_API_KEY;
  // ... other provider checks

  return new LLMClient({
    defaultProvider: hasNewProvider ? "newprovider" : "claude",
    newprovider: hasNewProvider ? { apiKey: privateEnv.NEWPROVIDER_API_KEY } : undefined,
    // ... other providers
  });
}

Provider Examples​

Example 1: OpenAI Provider​

The OpenAI provider demonstrates a straightforward implementation:

export class OpenAIProvider extends BaseLLMProvider {
  readonly providerType = "openai" as const;
  protected readonly pricing = OPENAI_PRICING;

  private readonly baseUrl: string;
  private readonly organizationId?: string;

  constructor(config: Omit<LLMProviderConfig, "provider">) {
    super({ ...config, provider: "openai" });
    this.baseUrl = config.baseUrl || OPENAI_API_BASE_URL;
    this.organizationId = config.organizationId;
  }

  getDefaultModel(): LLMModel {
    return (this.config.defaultModel as OpenAIModel) || DEFAULT_MODEL;
  }

  isModelSupported(model: string): boolean {
    return SUPPORTED_MODELS.includes(model as OpenAIModel);
  }

  protected async makeAPIRequest(
    messages: LLMMessage[],
    options: LLMRequestOptions
  ): Promise<LLMResponse> {
    // Implementation details...
  }
}

Example 2: Claude Provider​

The Claude provider shows handling of API-specific features:

export class ClaudeProvider extends BaseLLMProvider {
  readonly providerType = "claude" as const;
  protected readonly pricing = CLAUDE_PRICING;

  // Claude uses a separate system parameter instead of system messages
  private convertMessages(messages: LLMMessage[]): ClaudeMessage[] {
    return messages
      .filter((msg) => msg.role !== "system")
      .map((msg) => ({
        role: msg.role as "user" | "assistant",
        content: msg.content,
      }));
  }

  protected async makeAPIRequest(
    messages: LLMMessage[],
    options: LLMRequestOptions
  ): Promise<LLMResponse> {
    const requestBody: ClaudeRequestBody = {
      model,
      max_tokens: maxTokens,
      messages: this.convertMessages(messages),
    };

    // System prompt goes in a separate field
    if (options.systemPrompt) {
      requestBody.system = options.systemPrompt;
    }

    // Custom headers for Anthropic API
    const headers: Record<string, string> = {
      "Content-Type": "application/json",
      "x-api-key": this.config.apiKey,
      "anthropic-version": API_VERSION,
    };

    // Implementation details...
  }
}

Example 3: GLM Provider​

The GLM provider demonstrates working with a different API format:

export class GLMProvider extends BaseLLMProvider {
  readonly providerType = "glm" as const;
  protected readonly pricing = GLM_PRICING;

  // GLM supports system messages directly in the messages array
  private convertMessages(messages: LLMMessage[], systemPrompt?: string): GLMMessage[] {
    const glmMessages: GLMMessage[] = [];

    if (systemPrompt) {
      glmMessages.push({
        role: "system",
        content: systemPrompt,
      });
    }

    for (const msg of messages) {
      glmMessages.push({
        role: msg.role,
        content: msg.content,
      });
    }

    return glmMessages;
  }

  protected async makeAPIRequest(
    messages: LLMMessage[],
    options: LLMRequestOptions
  ): Promise<LLMResponse> {
    // GLM requires all parameters in the request body
    const requestBody: GLMRequestBody = {
      model,
      messages: this.convertMessages(messages, options.systemPrompt),
      temperature: options.temperature,
      top_p: options.topP,
      max_tokens: maxTokens,
      stop: options.stopSequences,
      stream: false, // GLM requires explicit stream parameter
    };

    // Implementation details...
  }
}

Best Practices​

1. Always Use the Base Provider Interface​

// ✅ GOOD - Extends BaseLLMProvider
export class MyProvider extends BaseLLMProvider {
  protected async makeAPIRequest(
    messages: LLMMessage[],
    options: LLMRequestOptions
  ): Promise<LLMResponse> {
    // Your implementation
  }
}

// ❌ BAD - Doesn't extend base class
export class MyProvider {
  async query(prompt: string) {
    // Custom implementation that doesn't match the interface
  }
}

2. Use Provider-Agnostic Types​

// ✅ GOOD - Uses generic types
async function analyzeText(text: string, options: LLMRequestOptions = {}): Promise<LLMResponse> {
  const provider = options.provider || "claude";
  // Implementation works with any provider
}

// ❌ BAD - Uses provider-specific types
async function analyzeText(text: string, options: ClaudeRequestOptions): Promise<ClaudeResponse> {
  // Only works with Claude
}

3. Implement Proper Error Handling​

// ✅ GOOD - Uses standard error types
try {
  const response = await this.makeAPIRequest(messages, options);
  return response;
} catch (error) {
  if (error instanceof Error && error.name === "AbortError") {
    throw new LLMTimeoutError(this.providerType, options.timeoutMs || 0);
  }
  if (error instanceof Error && error.message.includes("fetch")) {
    throw new LLMNetworkError(this.providerType, error);
  }
  throw error;
}

// ❌ BAD - Throws generic errors
try {
  const response = await this.makeAPIRequest(messages, options);
  return response;
} catch (error) {
  throw new Error("Request failed");
}

4. Handle Provider-Specific Quirks​

// ✅ GOOD - Handles provider-specific message formats
private convertMessages(messages: LLMMessage[]): ProviderMessage[] {
  // Claude doesn't support system role in messages array
  if (this.providerType === "claude") {
    return messages
      .filter((msg) => msg.role !== "system")
      .map((msg) => ({
        role: msg.role as "user" | "assistant",
        content: msg.content,
      }));
  }

  // Other providers support system messages
  return messages.map((msg) => ({
    role: msg.role,
    content: msg.content,
  }));
}

5. Implement Proper Cost Tracking​

// ✅ GOOD - Accurate pricing configuration
const PROVIDER_PRICING: PricingConfig = {
  "model-1": {
    inputPerMillion: 3.0,
    outputPerMillion: 15.0,
  },
  "model-2": {
    inputPerMillion: 0.25,
    outputPerMillion: 1.25,
  },
};

// Cost calculation is handled by BaseLLMProvider
const cost = this.calculateCost(usage, model);

// ❌ BAD - Hardcoded or missing pricing
const cost = {
  inputCost: 0,
  outputCost: 0,
  totalCost: 0,
  currency: "USD",
};

6. Support All Request Options​

// ✅ GOOD - Supports all standard options
const requestBody: ProviderRequestBody = {
  model,
  messages: this.convertMessages(messages, options.systemPrompt),
  max_tokens: maxTokens,
};

if (options.temperature !== undefined) {
  requestBody.temperature = options.temperature;
}
if (options.topP !== undefined) {
  requestBody.top_p = options.topP;
}
if (options.stopSequences) {
  requestBody.stop = options.stopSequences;
}

// ❌ BAD - Ignores optional parameters
const requestBody: ProviderRequestBody = {
  model,
  messages: this.convertMessages(messages),
  max_tokens: maxTokens,
  // Missing temperature, topP, stopSequences
};

7. Implement Proper Timeout Handling​

// ✅ GOOD - Proper timeout with cleanup
const controller = new AbortController();
const timeoutId = setTimeout(() => controller.abort(), options.timeoutMs || 60000);

try {
  const response = await fetch(url, {
    signal: controller.signal,
    // ... other options
  });
  clearTimeout(timeoutId);
  return response;
} catch (error) {
  clearTimeout(timeoutId);
  if (error instanceof Error && error.name === "AbortError") {
    throw new LLMTimeoutError(this.providerType, options.timeoutMs || 60000);
  }
  throw error;
}

// ❌ BAD - No timeout or improper cleanup
const response = await fetch(url);

Error Handling​

Error Hierarchy​

The LLM system uses a structured error hierarchy:

Error Types​

Error TypeWhen It's ThrownIs RetryableRetry After
LLMAuthenticationErrorInvalid or missing API keyNo-
LLMRateLimitErrorAPI quota exceededYesFrom header or default
LLMValidationErrorInvalid request parametersNo-
LLMNetworkErrorConnection issuesYesExponential backoff
LLMTimeoutErrorRequest timeoutYesImmediate
LLMContentFilterErrorContent blockedNo-
LLMModelNotFoundErrorModel not availableNo-
LLMContextLengthErrorRequest too longNo-
LLMAPIErrorOther API errorsMaybeBased on status code

Error Factory​

The createLLMErrorFromResponse function converts HTTP responses to appropriate error types:

export function createLLMErrorFromResponse(
  provider: LLMProvider,
  statusCode: number,
  errorBody: { error?: { type?: string; message?: string; code?: string } } | null,
  headers?: Headers
): LLMError {
  const errorMessage = errorBody?.error?.message || `HTTP ${statusCode}`;
  const errorType = errorBody?.error?.type || errorBody?.error?.code || "";

  switch (statusCode) {
    case 401:
      return new LLMAuthenticationError(provider, errorMessage);

    case 429: {
      const retryAfterHeader = headers?.get("retry-after");
      const retryAfter = retryAfterHeader ? parseInt(retryAfterHeader, 10) : undefined;
      return new LLMRateLimitError(provider, retryAfter, errorMessage);
    }

    case 400:
      if (errorType.toLowerCase().includes("context")) {
        return new LLMContextLengthError(provider, errorMessage);
      }
      if (errorType.toLowerCase().includes("content")) {
        return new LLMContentFilterError(provider, errorMessage);
      }
      return new LLMValidationError(provider, [errorType || errorMessage], errorMessage);

    case 404:
      if (errorType.toLowerCase().includes("model")) {
        const modelMatch = errorMessage.match(/model[:\s]+['"]?([^'"]+)['"]?/i);
        return new LLMModelNotFoundError(provider, modelMatch?.[1] || "unknown");
      }
      return new LLMAPIError(provider, statusCode, errorMessage, false);

    default:
      return new LLMAPIError(
        provider,
        statusCode,
        errorMessage,
        RETRYABLE_STATUS_CODES.includes(statusCode)
      );
  }
}

Handling Errors in Your Code​

import { LLMError, LLMAuthenticationError, LLMRateLimitError, isRetryableError } from "~/lib/llm";

try {
  const response = await client.query("Hello", {
    maxRetries: 3,
    retryDelayMs: 1000,
  });
} catch (error) {
  if (error instanceof LLMAuthenticationError) {
    console.error("Invalid API key:", error.message);
    // Prompt user to check their API key
  } else if (error instanceof LLMRateLimitError) {
    console.error("Rate limited:", error.message);
    if (error.retryAfter) {
      console.log(`Retry after ${error.retryAfter} seconds`);
    }
  } else if (isRetryableError(error)) {
    console.log("Temporary error, can retry");
  } else {
    console.error("LLM error:", getLLMErrorMessage(error));
  }
}

Testing​

Unit Testing Providers​

Create a mock client for testing:

// test/mocks/llm.ts
import type { LLMMessage, LLMRequestOptions, LLMResponse, LLMProvider } from "~/lib/llm/types";

export class MockLLMClient {
  constructor(private responses: Map<string, Partial<LLMResponse>> = new Map()) {}

  async query(prompt: string, options: LLMRequestOptions = {}): Promise<LLMResponse> {
    const key = `${options.provider || "mock"}:${options.model || "mock"}`;
    const mock = this.responses.get(key) || {};

    return {
      content: mock.content || "Mock response",
      usage: mock.usage || {
        inputTokens: 10,
        outputTokens: 5,
        totalTokens: 15,
      },
      cost: mock.cost || {
        inputCost: 0.00001,
        outputCost: 0.00002,
        totalCost: 0.00003,
        currency: "USD",
      },
      model: mock.model || options.model || "mock-model",
      provider: (mock.provider || options.provider || "mock") as LLMProvider,
      stopReason: mock.stopReason || "end_turn",
      id: mock.id || "mock-id",
      timestamp: mock.timestamp || new Date(),
    };
  }

  async chat(messages: LLMMessage[], options: LLMRequestOptions = {}): Promise<LLMResponse> {
    const prompt = messages.map((m) => m.content).join("\n");
    return this.query(prompt, options);
  }
}

Integration Testing​

Test your provider against the real API:

// test/providers/newprovider.test.ts
import { describe, it, expect } from "vitest";
import { NewProviderProvider } from "~/lib/llm/providers/newprovider";

describe("NewProviderProvider", () => {
  const provider = new NewProviderProvider({
    apiKey: process.env.NEWPROVIDER_API_KEY!,
  });

  it("should query the API", async () => {
    const response = await provider.query("Hello, world!");

    expect(response.content).toBeTruthy();
    expect(response.usage.inputTokens).toBeGreaterThan(0);
    expect(response.usage.outputTokens).toBeGreaterThan(0);
    expect(response.cost.totalCost).toBeGreaterThan(0);
  });

  it("should handle multi-turn conversations", async () => {
    const messages = [
      { role: "user" as const, content: "What is 2+2?" },
      { role: "assistant" as const, content: "2+2 equals 4." },
      { role: "user" as const, content: "And what is 3+3?" },
    ];

    const response = await provider.chat(messages);

    expect(response.content).toContain("6");
  });

  it("should respect temperature setting", async () => {
    const lowTemp = await provider.query("Random text", { temperature: 0 });
    const highTemp = await provider.query("Random text", { temperature: 1 });

    // Low temperature should be more consistent
    expect(lowTemp.content).toBeDefined();
    expect(highTemp.content).toBeDefined();
  });

  it("should handle timeouts", async () => {
    await expect(provider.query("Long prompt", { timeoutMs: 1 })).rejects.toThrow();
  });
});

Testing Provider Agnosticism​

Verify that your application logic works with any provider:

describe("categorizeMessage", () => {
  const providers = [
    { name: "claude", client: new ClaudeProvider(apiKey) },
    { name: "openai", client: new OpenAIProvider(apiKey) },
    { name: "glm", client: new GLMProvider(apiKey) },
  ];

  test.each(providers)("works with %s", async ({ name, client }) => {
    const result = await categorizeMessage("Meeting at 3pm tomorrow", { llm: client });

    expect(result.category).toBeTruthy();
    expect(result.confidence).toBeGreaterThan(0);
    expect(result.confidence).toBeLessThanOrEqual(1);
  });
});

Additional Resources​

External Resources​

Need help? Check the Contributing Guide or open an issue on GitHub.