CSE240A
Principles in Computer Architecture
System Design for LLMs
(not required)
Jishen Zhao
(https://cseweb.ucsd.edu/~jzhao/)
Disclaimer
• Current LLMs use transformer models. This list does not include systems design
for general transformers, many of which can be applied to LLMs too.
Training
• Scaling Infrastructure to Support Multi-Trillion Parameter LLM Training
• FLASHATTENTION: Fast and Memory-Efficient Exact Attention with IOAwareness
• Scaling Infrastructure to Support Multi-Trillion Parameter LLM Training
• A Fast, Performant, Secure Distributed Training Framework For Large Language
Model
Serving / post-training
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
ORCA: A Distributed Serving System for Transformer-Based Generative Models
Fast Distributed Inference Serving for Large Language Models
Pathways: Asynchronous Distributed Dataflow for ML
DeepSpeed-inference: Enabling efficient inference of transformer models at unprecedented scale
DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference
LLM in a flash: Efficient Large Language Model Inference with Limited Memory
Efficient Memory Management for Large Language Model Serving with PagedAttention
MemGPT: Towards LLMs as Operating Systems
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
Efficient Streaming Language Models with Attention Sinks
Ziplm: Hardware-aware structured pruning of language models
Quantization and Hardware Architecture Co-Design for Matrix-Vector Multiplications of Large Language Models
OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization
QIGen: Generating Efficient Kernels for Quantized Inference on Large Language Models
Automatic Kernel Generation for Large Language Models on Deep Learning Accelerators
Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference
AttentionLego: An Open-Source Building Block For Spatially-Scalable Large Language Model Accelerator With Processing-In-Memory
Technology
CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
A Hardware Evaluation Framework for Large Language Model Inference
Distributed serving and scaling
Memory limitation and memory management
Pruning and quantization
Accelerators
Others
• A Survey on Hardware Accelerators for Large Language Models, 2024.
• Towards Efficient Generative Large Language Model Serving: A Survey from
Algorithms to Systems, 2023.
A Survey on Hardware
Accelerators for Large
Language Models, 2024.
A Survey on Hardware
Accelerators for Large
Language Models, 2024.
Distributed serving
• ORCA: A Distributed Serving System for Transformer-Based Generative Models,
OSDI’22.
• A scheduling mechanism that schedules execution at the granularity of iteration
(instead of request) where the scheduler invokes the execution engine to run
only a single iteration of the model on the batch
Memory limitation and memory management
• LLM in a flash: Efficient Large
Language Model Inference with
Limited Memory, Apple, 2024.
• Efficiently running LLMs that exceed
the available DRAM capacity
• Store the model parameters in flash
memory
• But bringing them on demand to
DRAM
Inference latency of 1 token when half the
memory of the model is available
Memory limitation and memory management
• Efficient Memory Management for
Large Language Model Serving with
PagedAttention, 2023.
• Build vLLM, an LLM serving system
that achieves
• Near-zero waste in KV cache memory
• Flexible sharing of KV cache within and
across requests to further reduce memory
usage
Inference framework
Simulation to evaluate LLM running on HW platforms
• A Hardware Evaluation Framework for Large Language Model Inference, 2024.
In-class exercises
• Please see link in the “Lecture Materials and Syllabus” table in the class
website