Chat-Fusion: An Efficient Batch Inference System for Cloud LLMs Serving

Chat-Fusion is a batch inference system for cloud LLM serving that fuses requests to raise throughput.