Anyscale Blog company blog
Continuous batching to increase LLM inference throughput and reduce p50 latency
In this blog, we discuss continuous batching, a critical systems-level optimization that improves both throughput and latency under load for LLMs.
- Points
- 110
- HN comments
- 20
- Submissions
- 1
- Cited by
- 3 commenters
AI & Machine LearningSystems & Performance2023
On Hacker News
| Submitted | Title on HN | Points | Comments |
|---|---|---|---|
| Aug 15, 2023 | Continuous batching to increase LLM inference throughput and reduce p50 latency by michellezzz | 110 | 20 |
Cited in comments
Linked from 3 Hacker News comments by 3 different people, from Sep 2023 to Apr 2025.
Most recent:
- mitjam · Apr 20, 2025
- yelite · Sep 14, 2023
- brucethemoose2 · Sep 4, 2023
