Model Serving Design

Overview

Model serving is the infrastructure that delivers ML model predictions to applications. Designing a model serving system involves trade-offs between latency, throughput, cost, and reliability. The architecture must handle varying traffic patterns, model updates, and feature lookups.

Serving Architectures

graph TD
    A[Serving Patterns] --> B[Online Real-time]
    A --> C[Batch]
    A --> D[Streaming]
    B --> B1[HTTP/gRPC endpoint]
    B --> B2[Feature lookup → Model → Response]
    C --> C1[Scheduled jobs - daily reports]
    D --> D1[Kafka → Model → Output topic]

Pattern Comparison

System Design

graph LR
    A[Client] --> B[Load Balancer]
    B --> C[API Gateway]
    C --> D[Feature Service]
    D --> E[Model Server]
    E --> F[Response Cache]
    F --> G[Response]

Key Components

Model Server Comparison

Triton Inference Server Deep Dive

Triton is the most versatile production model server:

graph TD
    A[Triton Inference Server] --> B[Model Repository]
    B --> C[TF SavedModel]
    B --> D[PyTorch TorchScript]
    B --> E[ONNX Model]
    B --> F[TensorRT Engine]
    A --> G[Dynamic Batching]
    A --> H[Model Pipeline]
    A --> I[Concurrent Model Execution]

Key features:

• Dynamic batching: Automatically batches incoming requests for GPU efficiency
• Model ensemble: Chain pre-processing → model → post-processing
• Concurrent execution: Run multiple models simultaneously on different GPUs
• Model warmup: Pre-load models to avoid cold start latency

vLLM for LLM Serving

vLLM is the standard for LLM inference:

PagedAttention explained: Traditional LLM serving pre-allocates contiguous GPU memory for each request’s KV cache, leading to memory waste. PagedAttention divides memory into fixed-size blocks and maps them non-contiguously, achieving near-optimal memory utilization.

Scaling Strategies

Auto-scaling Configuration

autoscaling_config = {
    "min_replicas": 2,
    "max_replicas": 50,
    "target_cpu_utilization": 70,
    "target_gpu_utilization": 80,
    "target_latency_p99_ms": 200,
    "scale_up_rate": 2,      # Double replicas
    "scale_down_rate": 0.5,   # Halve replicas
    "scale_up_cooldown_sec": 60,
    "scale_down_cooldown_sec": 300,
}

Scaling Dimensions

Traffic Management

graph TD
    A[Traffic Spike] --> B[Request Queue]
    B --> C[Load Balancer]
    C --> D[Model Server Pool]
    D --> E{GPU Saturated?}
    E -->|Yes| F[Auto-scale Up]
    E -->|No| G[Process Normally]
    F --> H[New Instances Ready]
    H --> D

Model Optimization for Serving

Optimization Pipeline

graph LR
    A[PyTorch/TF Model] --> B[Export to ONNX]
    B --> C[ONNX Optimization]
    C --> D[TensorRT Compilation]
    D --> E[Quantized Model]
    E --> F[Triton Deployment]

Request Batching

# Dynamic batching configuration (Triton)
model_config = {
    "max_batch_size": 64,
    "preferred_batch_size": [8, 16, 32],
    "max_queue_delay_microseconds": 100000,  # 100ms
}

Caching Strategies

Semantic Caching for LLMs

graph TD
    A[User Query] --> B[Embed Query]
    B --> C[Search Cache - ANN]
    C --> D{Similarity > Threshold?}
    D -->|Yes| E[Return Cached Response]
    D -->|No| F[Run LLM Inference]
    F --> G[Cache Response]
    G --> H[Return Response]

Multi-Model Serving

A/B Testing Architecture

graph TD
    A[Request] --> B[Router]
    B --> C[Model A - 90% traffic]
    B --> D[Model B - 10% traffic]
    C --> E[Log Prediction + Metrics]
    D --> E
    E --> F[Statistical Analysis]
    F --> G{Significant Improvement?}
    G -->|Yes| H[Promote Model B]
    G -->|No| I[Keep Model A]

Deployment Patterns

Interview Questions

1. How do you design a low-latency model serving system? Use efficient model format (ONNX, TensorRT), feature caching, request batching, model optimization (quantization), and deployment close to users (edge/CDN). For LLMs, use vLLM with PagedAttention.

2. How do you handle traffic spikes? Auto-scaling based on latency/CPU/GPU metrics, request queuing with backpressure, pre-warming instances (predictable spikes), caching frequent predictions, and graceful degradation (serve cached/simpler model under extreme load).

3. What is model server comparison? TF Serving: TensorFlow native. TorchServe: PyTorch native. Triton: multi-framework, GPU optimization, production-grade. vLLM: LLM-specific, PagedAttention. BentoML: easy packaging, developer-friendly.

4. How do you serve multiple models efficiently? Use Triton’s multi-model serving (share GPU memory), model ensemble pipelines, and dynamic batching. For LLMs, use vLLM’s continuous batching. Route requests to appropriate model based on task type.

5. How do you handle model versioning in serving? Model registry with versioned artifacts. Serve multiple versions simultaneously (A/B testing). Use canary deployment for new versions. Keep previous version warm for instant rollback.

Summary

Model serving design must balance latency, throughput, and cost. Key decisions include serving pattern (online/batch/streaming), model format, caching strategy, and scaling approach. Production systems typically use a load balancer → API gateway → feature service → model server architecture.

References

• NVIDIA Triton Inference Server documentation
• vLLM: Easy, Fast, and Cheap LLM Serving (Kwon et al., 2023)
• TensorFlow Serving architecture guide
• BentoML documentation (bentoml.com)
• Clipper: A Low-Latency Online Prediction Serving System (Crankshaw et al., 2017)

Cross-References

• Model Deployment — Deployment patterns
• ML Pipeline — Training pipeline
• Feature Store — Feature serving
• Monitoring — Post-deployment monitoring