← All capstone topics
Research project LLMs at the edge

Scheduling LLM inference requests on a shared edge GPU server with KV-cache awareness

Decide in which order and in which batches to serve many users of one GPU so that nobody waits too long and the GPU memory (the KV cache) is not wasted.

Level: Undergraduate Format: Individual project Duration: 2 semesters Supervisor: Dr. Hamza Djigal

The problem

An edge server with one GPU serves many users at once. Each running request keeps a "KV cache" in GPU memory that grows with the answer length. First-come-first-served wastes memory and makes short requests wait behind long ones; serving systems such as vLLM introduced paging and continuous batching, but their policies assume a large data centre, not a small edge GPU with deadlines.

Who needs it

Telecom operators and companies that host their own models (privacy, cost) on a few GPUs need predictable response times.

Idea of the approach

  • Install vLLM (or a simulator built from its published behaviour) with a small model on one GPU.
  • Generate request traces with different prompt/answer lengths and deadlines.
  • Implement 2–3 scheduling policies: shortest-predicted-job-first (predict answer length from the prompt), deadline-aware admission, and memory-aware batching.
  • Measure latency percentiles, deadline misses and throughput.

What you will build and deliver

  • Scheduling policies implemented as a plug-in or simulator.
  • Request traces and evaluation scripts.
  • Paper-ready figures.

Skills needed

Python, operating-system scheduling concepts (CPS 3250), GPU access (lab or cloud credits).

Research contribution & where to publish

Contribution: deadline-aware scheduling for small edge GPUs, connecting classic list-scheduling ideas to LLM serving. Venues: IEEE/ACM IWQoS, ICPP, IEEE TPDS, Euro-Par.

Key references — start reading here

  1. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, I. Stoica. Efficient Memory Management for Large Language Model Serving with PagedAttention. ACM SOSP, 2023.
  2. G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, B.-G. Chun. Orca: A Distributed Serving System for Transformer-Based Generative Models. USENIX OSDI, 2022.
  3. A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, R. Ramjee. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. USENIX OSDI, 2024.
  4. H. Djigal, L. Liu, J. Luo, J. Xu. BUDA: Budget and Deadline Aware Scheduling Algorithm for Task Graphs in Heterogeneous Systems. IEEE/ACM IWQoS, 2022.

Interested?

Send the topic title and the courses you have taken.

Contact Dr. Hamza Djigal

Other topics in LLMs at the edge