This section covers the systems, programming models, and infrastructure that power scientific computing, large-scale simulations, and distributed AI training at exascale. HPC sits at the intersection of architecture, networking, systems software, and algorithms — making it a fertile area for depth-oriented interview questions at companies running large GPU clusters (Google, Meta, NVIDIA, AWS Trainium/Inferentia, Microsoft Azure).
graph TD
A[HPC Section] --> B[MPI Parallelism]
A --> C[Collective Communication & Distributed Training]
A --> D[HPC Infrastructure]
B --> B1[MPI / OpenMP / OpenACC]
B --> B2[CUDA Advanced: Memory/Streams/Graphs]
B --> B3[NCCL / GPUDirect / RDMA]
B --> B4[InfiniBand / RoCE / NVLink]
C --> C1[All-Reduce Algorithms]
C --> C2[Parallelism Strategies: Tensor/Pipeline/Data/Expert]
C --> C3[Parameter Servers vs Ring All-Reduce]
C --> C4[Federated & Async Optimization]
D --> D1[Slurm / Scheduling / Backfilling]
D --> D2[Checkpoint/Restart & Fault Tolerance]
D --> D3[Exascale & Energy-Aware Computing]
D --> D4[Performance Portability: Kokkos/SYCL/oneAPI]