Unlocking Software-defined GPU Fabric Scheduling in the LLM Era

Summary

GPUWeaver argues that the links inside a GPU server should be scheduled like CPU time and memory. Large language model (LLM) serving moves large amounts of state, such as the key-value (KV) cache that holds a model's attention history, over shared links: NVLink between GPUs, PCIe between GPUs, CPUs, and network cards, and RDMA (direct memory access over the network) between servers. Hardware arbitration on those links gives some traffic an implicit priority, and GPUWeaver regulates how fast each flow enters the link according to application goals. In a vLLM and Mooncake case study it cuts prefill-to-decode KV-transfer time by 19.4 to 34.4%.

Key numbers

• 79.8%Drop in host-to-device PCIe bandwidth (25.82 to 5.23 GiB/s) while GPUDirect RDMA writes to the same GPU
• ≈20%Loss of HBM throughput when a comparable or longer NVLink transfer overlaps GPU memory access
• 19.4–34.4%Reduction in prefill-to-decode KV-transfer time in the vLLM + Mooncake case study, which lowers mean time to first token by 2.4–6.1%

Method

LLM serving now splits work across devices. Prefill–decode disaggregation runs the prompt-processing phase (prefill) and the token-generation phase (decode) on separate workers ^[3], and a prefill worker sends the KV cache to the decode worker over GPUDirect RDMA, a path that lets a network card read and write GPU memory directly. Systems such as Mooncake ^[2] also offload KV cache to host memory, and vLLM ^[1] manages the cache in GPU high-bandwidth memory (HBM). These flows share hardware: NVLink and local compute share HBM, and RDMA and host–device copies share one PCIe segment. Existing LLM systems treat that fabric as best-effort.

GPUWeaver is a closed-loop runtime. Applications register flows, such as a KV transfer and a KV offload, with priorities or latency goals. The scheduler reads infrastructure signals (NVML and DCGM PCIe counters, NVLink counters, NIC throughput, GPU memory-bandwidth use) and application signals (request, stage, and transfer latency, and per-flow backlog). When a high-priority flow misses its target while the fabric is saturated, the policy moves bandwidth budget toward it in the next control window, and it moves the budget back when the flow is healthy. The knobs are indirect because CUDA has no native bandwidth control for host–device transfers. For RDMA the runtime uses service levels, NIC quality-of-service settings, and rate limiting. For host–device copies it uses stream priorities, launch gating, chunked copies, and cooperative yield points.

Evaluation

Controlled measurements on a DGX-H100 server expose two contentions. In the first, GPU kernels and NVLink copy engines share the L2 cache and memory controllers. Across a sweep of buffer sizes from 128 MB to 4 GB, overlapping transfers reduce HBM throughput by roughly 20% when the NVLink transfer is comparable or longer, and by 3 to 10% when it is shorter. In the second, GPUDirect RDMA competes with host–device copies on PCIe. Concurrent RDMA writes cut host-to-device bandwidth from 25.82 to 5.23 GiB/s (79.8%), and RDMA reads cut device-to-host bandwidth from 25.12 to 11.57 GiB/s (53.9%). The effect is directional: background host-to-device traffic barely changes RDMA write throughput, while background device-to-host traffic lowers RDMA read throughput by about 41.5% for messages of 8 KB and above.

In the case study, a prefill worker sends new KV cache to a decode worker over GPUDirect RDMA while offloading prefix KV cache to the CPU. The prototype delays or paces the offload so that the latency-critical transfer gets more of the PCIe budget. Replaying request shapes from a SWE-bench Pro agent trace collected from Codex ^[4] on Qwen3-8B, with aggregate KV sizes from 1.2 to 9.4 GB, GPUWeaver reduces KV-transfer flow time by 19.4 to 34.4% and mean time to first token (TTFT), the delay before the first output token, by 2.4 to 6.1%. The TTFT gain is smaller because the transfer is only part of the TTFT path.

Limitations

This is a vision paper with a preliminary prototype. The policy is simple and the control action is coarse (offload delay injection). In the case study, GPUWeaver sweeps candidate PCIe allocations and selects the best one using bandwidth and TTFT measured in earlier runs, so the policy is tuned per request shape and does not yet adapt online. The setup emulates disaggregation with two logical servers of one GPU each, and the results hold for that case study.

The scheduler does not yet manage the HBM and NVLink contention that the characterization exposes. Extending it there, adding per-flow models, and allocating bandwidth across latency-critical and background traffic are future work.

Citation

@inproceedings{chen2026gpuweaver,
  title={Unlocking Software-defined GPU Fabric Scheduling in the LLM Era},
  author={Danyang Chen and Yufeng Gu and Yibo Huang and Chengxuan Pei and Peichun Hua and Yang Zhou and Yunming Xiao},
  booktitle={ACM SIGOPS Asia-Pacific Workshop on Systems (APSys)},
  year={2026},
  isbn = {9798400729126},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3838177.3841731},
  numpages = {8},
  keywords = {GPU fabrics, datacenter systems, heterogeneous accelerators, machine learning systems, communication scheduling, resource contention, PCIe, NVLink, RDMA, LLM serving},
  location = {Bangkok Marriott Marquis Queen's Park, Bangkok, Thailand},
  series = {APSys '26}
  pages={118--125},
  doi={10.1145/3838177.3841731},
}

References

1. Woosuk Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP, 2023. arxiv.org/abs/2309.06180
2. Ruoyu Qin et al. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. FAST, 2025. arxiv.org/abs/2407.00079
3. Yinmin Zhong et al. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI, 2024. arxiv.org/abs/2401.09670
4. Inferact. codex_swebenchpro_traces: SWE-bench Pro agentic workload traces collected from Codex agent runs. Hugging Face dataset, 2026. huggingface.co/datasets/Inferact/codex_swebenchpro_traces