Advanced LLM Systems

Overview

This section covers the deep systems-level topics that separate candidates who “know LLMs” from those who can build and operate production AI infrastructure. These topics appear in senior SWE, ML infra, and applied scientist interviews at companies deploying LLMs at scale.

The content here goes well beyond the introductory serving and MoE material — diving into transformer kernel optimization, distributed training at scale, advanced quantization, production inference systems, retrieval-augmented generation at scale, and autonomous agent architectures.

Topic Map

graph TD
    ADV[Advanced LLM Systems]
    ADV --> TI[Transformer Internals]
    ADV --> TA[Advanced Training]
    ADV --> QA[Quantization Advanced]
    ADV --> IS[Inference Systems]
    ADV --> RA[RAG Advanced]
    ADV --> AS[Agent Systems]

    TI --> FA[FlashAttention]
    TI --> PA[Paged Attention]
    TI --> KVC[KV Cache Compression]
    TI --> PC[Prefix Caching]

    TA --> DP[Distributed Parallelism]
    TA --> ZERO[ZeRO / FSDP]
    TA --> KD[Knowledge Distillation]

    QA --> GPTQ[GPTQ / AWQ]
    QA --> FP8[FP8 / Mixed Precision]
    QA --> SP[Structured Sparsity]

    IS --> SERVE[Triton / vLLM / TensorRT-LLM]
    IS --> RLHF[RLHF / DPO Pipelines]
    IS --> MCM[Mixture-of-Models]

    RA --> ANN[HNSW / DiskANN]
    RA --> RERANK[Reranking]

    AS --> MCP[Model Context Protocol]
    AS --> MULTI[Multi-Agent Orchestration]
    AS --> CODING[AI Coding Agents]

Sections

Complete Page Map

One line per file in this directory. The Sections table above is the “big six” survey; the map below is the full inventory, including the kernel-level pages and the distributed/ subdirectory that senior interviews actually probe.

The distributed/ subdirectory

Suggested Interview Reading Order

For a senior/ML-infra loop, this exact sequence builds each concept on the previous one — attention math, then attention IO cost, then the memory problem, then the systems that manage that memory, then the runtimes that ship it.

1. transformer-internals.md — establish the FLOP/memory model of a transformer forward pass.
2. flash-attention.md — why attention is IO-bound, tiling + online softmax, HBM↔SRAM traffic math.
3. paged-attention.md — the KV cache becomes a memory-management problem: blocks, fragmentation, COW.
4. kv-cache-compression.md — after paging, compress: quantize, evict, window, merge.
5. vllm-internals.md — assemble 2–4 into a real engine: continuous batching, preemption, scheduling.
6. sglang.md — the second runtime datapoint: RadixAttention prefix trees, structured decoding — the comparison question interviewers actually ask.

Then branch by target role: ML infra → speculative-decoding.md, tensorrt-llm.md, distributed/prefill-decode-disaggregation.md; training platforms → training-advanced.md then distributed/megatron.md, distributed/fsdp.md; search/RAG → rag-advanced.md, faiss.md, milvus.md, pgvector.md; applied/product → agent-systems.md, structured-output-decoding.md.

Prerequisites

Before diving into this section, ensure you’re comfortable with:

• LLM Fundamentals — transformer architecture, tokenization, training pipeline
• LLM Serving — KV cache, batching, inference basics
• MoE Architecture — expert routing, load balancing
• RAG Systems — basic retrieval, embeddings, chunking

How to Use This Section

1. Interview Prep (2-3 weeks out): Read all sections, focus on comparison tables and “Interview Angle” callouts
2. Deep Dive: Follow the Mermaid diagrams, trace through pseudocode, understand trade-offs
3. System Design: Combine topics — e.g., design a serving system using quantization + paged attention + continuous batching + prefix caching

Interview Questions

Q: Why does FlashAttention make attention faster if it computes the exact same output?

Because attention at long sequence lengths is memory-bandwidth-bound, not FLOP-bound: the naive kernel materializes an N×N score matrix in HBM, writing and re-reading it repeatedly. FlashAttention tiles the computation so scores are produced, softmaxed (via the online-softmax running-max trick), and consumed entirely inside SRAM, cutting HBM traffic from O(N²) to roughly O(N²d²/M) for SRAM size M while doing the same arithmetic. The interview punchline: it is an IO-schedule change, not an approximation — and that is why it also saves memory, enabling longer contexts for free.

Q: What problem does PagedAttention solve, and what does it cost?

Pre-PagedAttention serving reserved a contiguous worst-case KV buffer per sequence, so allocation was all-or-nothing and internal/external fragmentation wasted 60–90% of KV memory. PagedAttention manages the cache in fixed-size blocks (vLLM default: 16 tokens), shared prefixes map to shared blocks copy-on-write, and sequences physically grow on demand — throughput gains of 2–4× on real traces come almost entirely from fitting more concurrent sequences per GPU. The cost is an indirect block-table lookup on every attention access (folded into the fused kernel) and block-table bookkeeping, both negligible at 16-token granularity.

Q: Compute the KV cache size per token for a 7B-class model.

Per token: 2 (K and V) × n_layers × n_kv_heads × head_dim × bytes_per_element. For Llama-2-7B — 32 layers, 32 heads, head_dim 128, fp16 — that is 2 × 32 × 4096 × 2 B = 512 KB/token, so a 4k-token sequence holds ~2 GB and a 40 GB A100 fits only a handful of such sequences before paged KV management (or GQA, which divides the heads term) becomes mandatory. Follow-ups to expect: how GQA shrinks the factor (fewer KV heads), why batch size scales linearly with this number, and how KV quantization to int8 halves it.

Q: Compare vLLM and SGLang — when would you pick each?

vLLM’s core contribution is PagedAttention plus continuous batching, making it the default general-purpose open serving engine with broad model coverage. SGLang’s differentiator is RadixAttention: it keeps KV caches in a radix tree keyed by token prefix, so multi-turn chat, few-shot, and agentic workloads with shared prompts replay cached prefixes instead of recomputing them, and its frontend makes structured (JSON/grammar-constrained) decoding fast. Pick vLLM for heterogeneous single-turn workloads and ecosystem maturity; pick SGLang when prefix reuse or constrained output dominates your trace — and name the measurement you’d run (prefix-cache hit rate) to decide.

Q: What is continuous batching and why did static batching leave 2–5× throughput on the table?

Static batching forms a fixed batch, runs it to completion, and discards it — so a fast sequence finishing early forces the GPU to wait for the slowest sequence (or truncates output). Continuous batching (Orca-style) admits a new sequence at every decode iteration and retires finished ones immediately, keeping the batch full for the entire engine lifetime. The follow-up is usually preemption: with paged KV, a burst can exhaust memory, and vLLM chooses between swapping blocks to CPU or recomputing the sequence later — a nice hook into the paged-attention page.

Key Takeaways

• The serving-stack reading order that mirrors real interviews: transformer FLOPs → attention IO cost → KV memory management → KV compression → engine internals → runtime comparison.
• FlashAttention is exact: its win is an IO schedule (SRAM tiling + online softmax), not approximation — know the O(N²) HBM traffic it avoids.
• PagedAttention treats KV cache like virtual memory: 16-token blocks, copy-on-write prefix sharing, ~2–4× throughput from fragmentation elimination alone.
• Be able to size a KV cache on a whiteboard: 512 KB/token for Llama-2-7B fp16; GQA and KV quantization are the two levers that shrink it.
• Continuous batching is the throughput unlock of modern serving; preemption policy (swap vs recompute) is the depth signal on top of it.
• vLLM vs SGLang is the standard runtime comparison question: PagedAttention generality vs RadixAttention prefix/structured-decoding dominance — answer with the metric you’d measure.
• Speculative decoding trades one big verify pass for many cheap draft steps; acceptance rate × draft length is the whole game.
• Training-side interviews go 3D: data + tensor + pipeline parallelism composed per Megatron rules, with FSDP/ZeRO as the memory lever — see the distributed/ pages.

References

• Vaswani et al., “Attention Is All You Need”, NeurIPS 2017: https://arxiv.org/abs/1706.03762
• Dao et al., “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness”, 2022: https://arxiv.org/abs/2205.14135
• Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” (vLLM), SOSP 2023: https://arxiv.org/abs/2309.06180
• Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models”, 2021: https://arxiv.org/abs/2106.09685
• Dettmers et al., “QLoRA: Efficient Finetuning of Quantized LLMs”, 2023: https://arxiv.org/abs/2305.14314
• Frantar et al., “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers”, 2022: https://arxiv.org/abs/2210.17323
• Lin et al., “AWQ: Activation-aware Weight Quantization”, 2023: https://arxiv.org/abs/2306.00978
• Zheng et al., “SGLang: Efficient Execution of Structured Language Model Programs” (arXiv:2312.07104) — cited by ID; RadixAttention paper.
• vLLM project documentation: https://docs.vllm.ai/
• SGLang project: https://github.com/sgl-project/sglang

Cross-References

• LLM Fundamentals →
• LLM Serving →
• MoE →
• RAG Systems →
• Prompt Engineering →
• LLM Security →
• Cost Optimization →
• Model Architectures hub — SSM/RWKV/linear-attention alternatives to the kernels above
• Mamba & SSMs — the sub-quadratic competitor to FlashAttention-era transformers
• Long-Context Strategies — how architectures stretch the context the KV pages above must hold
• Post-Training hub — RLHF/DPO/GRPO pipelines that the inference systems above serve
• DPO Family — the alignment method most often asked alongside serving costs
• Agentic Systems hub — topologies, guardrails, protocols for the agent-systems page above
• Multi-Agent Topologies — orchestration patterns behind agent-systems.md