The problem
A large language model writes one token at a time, which makes answers slow, especially when every token travels over the network. Speculative decoding lets a small "draft" model propose several tokens that the large model verifies in one step. It was designed for one data-centre GPU; splitting the draft (on the phone) and the verifier (on the edge) adds network delay and changes the best draft length.
Who needs it
Phone assistants, customer-service chat and voice applications, where every second of waiting loses users; telecom operators offering edge AI.
Idea of the approach
- Pair a small and a large model of the same family (e.g. Qwen2.5-0.5B with Qwen2.5-7B).
- Implement speculative decoding locally first, then with the draft on a Raspberry Pi or phone and the verifier on a GPU server.
- Measure acceptance rate, tokens per second and energy for several network delays.
- Propose a rule that adapts the number of drafted tokens to the measured delay and acceptance rate.
What you will build and deliver
- A working device + edge prototype and its measurements.
- The adaptive draft-length rule with its evaluation.
- A paper.
Skills needed
Python, PyTorch, Hugging Face Transformers; basic networking.
Research contribution & where to publish
Contribution: network-aware speculative decoding split across device and edge. Venues: IEEE INFOCOM / ICDCS workshops, MobiCom workshops, IEEE Transactions on Mobile Computing.
Key references — start reading here
- Y. Leviathan, M. Kalman, Y. Matias. Fast Inference from Transformers via Speculative Decoding. ICML, 2023.
- C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, J. Jumper. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv, 2023.
- H. Djigal, J. Xu, L. Liu, Y. Zhang. Machine and Deep Learning for Resource Allocation in Multi-Access Edge Computing: A Survey. IEEE Communications Surveys & Tutorials, 2022.
Interested?
Send the topic title and the courses you have taken.
Contact Dr. Hamza DjigalOther topics in LLMs at the edge
Splitting a small LLM between a smartphone and an edge server: latency- and energy-aware partitioning
An LLM request router: on-device, edge or cloud model, chosen by cost, latency and answer quality
Scheduling LLM inference requests on a shared edge GPU server with KV-cache awareness
An offline campus assistant on a quantized small language model running on a Raspberry Pi / Jetson