The problem
An edge server with one GPU serves many users at once. Each running request keeps a "KV cache" in GPU memory that grows with the answer length. First-come-first-served wastes memory and makes short requests wait behind long ones; serving systems such as vLLM introduced paging and continuous batching, but their policies assume a large data centre, not a small edge GPU with deadlines.
Who needs it
Telecom operators and companies that host their own models (privacy, cost) on a few GPUs need predictable response times.
Idea of the approach
- Install vLLM (or a simulator built from its published behaviour) with a small model on one GPU.
- Generate request traces with different prompt/answer lengths and deadlines.
- Implement 2–3 scheduling policies: shortest-predicted-job-first (predict answer length from the prompt), deadline-aware admission, and memory-aware batching.
- Measure latency percentiles, deadline misses and throughput.
What you will build and deliver
- Scheduling policies implemented as a plug-in or simulator.
- Request traces and evaluation scripts.
- Paper-ready figures.
Skills needed
Python, operating-system scheduling concepts (CPS 3250), GPU access (lab or cloud credits).
Research contribution & where to publish
Contribution: deadline-aware scheduling for small edge GPUs, connecting classic list-scheduling ideas to LLM serving. Venues: IEEE/ACM IWQoS, ICPP, IEEE TPDS, Euro-Par.
Key references — start reading here
- W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, I. Stoica. Efficient Memory Management for Large Language Model Serving with PagedAttention. ACM SOSP, 2023.
- G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, B.-G. Chun. Orca: A Distributed Serving System for Transformer-Based Generative Models. USENIX OSDI, 2022.
- A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, R. Ramjee. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. USENIX OSDI, 2024.
- H. Djigal, L. Liu, J. Luo, J. Xu. BUDA: Budget and Deadline Aware Scheduling Algorithm for Task Graphs in Heterogeneous Systems. IEEE/ACM IWQoS, 2022.
Interested?
Send the topic title and the courses you have taken.
Contact Dr. Hamza DjigalOther topics in LLMs at the edge
Splitting a small LLM between a smartphone and an edge server: latency- and energy-aware partitioning
An LLM request router: on-device, edge or cloud model, chosen by cost, latency and answer quality
An offline campus assistant on a quantized small language model running on a Raspberry Pi / Jetson
Prompt compression for LLM offloading over low-bandwidth networks