Unlocking Software-defined GPU Fabric Scheduling in the LLM Era

Danyang Chen; Yufeng Gu; Yibo Huang; Chengxuan Pei; Peichun Hua; Yang Zhou; Yunming Xiao

doi:10.1145/3838177.3841731

Accepted · APSys 2026

Unlocking Software-defined GPU Fabric Scheduling in the LLM Era

Danyang Chen, Yufeng Gu, Yibo Huang, Chengxuan Pei, Peichun Hua, Yang Zhou, Yunming Xiao

Read paper (PDF)

Overview

GPUWeaver explores software-defined GPU fabric scheduling for large language model systems. The paper examines contention across NVLink, PCIe, and RDMA when workloads combine computation, communication, KV-cache offloading, and disaggregated serving.

Method

A context-aware scheduler monitors application progress and fabric state, detects contention, and regulates communication injection according to application performance goals. The design treats GPU fabric bandwidth as an explicitly managed resource.

GPUWeaver architecture with application quality-of-service goals, a context-aware scheduler, communication libraries, and monitored GPU interconnect hardware. — A scheduler connects application performance goals to communication controls and monitored GPU fabric state. Figure 6 in the paper

Evaluation

The study characterizes NVLink–memory and PCIe traffic interference. In a vLLM + Mooncake case study with Qwen3-8B, the prototype reduces prefill-to-decode KV-transfer time by 19.4–34.4% and mean time to first token by 2.4–6.1% across the evaluated request shapes.

KV-cache scheduling improvements in the serving case study
Prompt tokens × concurrent requests	KV volume	KV-transfer time reduction	Time-to-first-token reduction
8K × 1	1.2 GB	19%	6.1%
16K × 1	2.4 GB	20%	2.8%
16K × 2	4.7 GB	34%	2.7%
16K × 3	7.1 GB	33%	3.0%
8K × 8	9.4 GB	31%	2.4%

Rounded chart labels transcribed from Figure 7. The prototype uses vLLM + Mooncake with Qwen3-8B on two logical servers, one GPU per server, replaying request shapes from a Codex/SWE-bench Pro trace. The policy selects a PCIe allocation using monitored bandwidth and serving results from prior runs; results are specific to this case study. Figure 7 in the paper

GPU scheduling
NVLink
PCIe
RDMA
LLM systems
communication contention

Citation

@inproceedings{chen2026unlocking,
  title={Unlocking Software-defined GPU Fabric Scheduling in the LLM Era},
  author={Danyang Chen and Yufeng Gu and Yibo Huang and Chengxuan Pei and Peichun Hua and Yang Zhou and Yunming Xiao},
  booktitle={ACM SIGOPS Asia-Pacific Workshop on Systems (APSys)},
  year={2026},
  organization={ACM}
}