← All capstone topics
Research project LLMs at the edge

Speculative decoding between a small on-device model and an edge LLM

A tiny model on the phone guesses the next words quickly; a larger model on the edge server only checks them — answers keep the large model's quality but arrive much faster.

Level: Undergraduate Format: Individual project Duration: 2 semesters Supervisor: Dr. Hamza Djigal

The problem

A large language model writes one token at a time, which makes answers slow, especially when every token travels over the network. Speculative decoding lets a small "draft" model propose several tokens that the large model verifies in one step. It was designed for one data-centre GPU; splitting the draft (on the phone) and the verifier (on the edge) adds network delay and changes the best draft length.

Who needs it

Phone assistants, customer-service chat and voice applications, where every second of waiting loses users; telecom operators offering edge AI.

Idea of the approach

  • Pair a small and a large model of the same family (e.g. Qwen2.5-0.5B with Qwen2.5-7B).
  • Implement speculative decoding locally first, then with the draft on a Raspberry Pi or phone and the verifier on a GPU server.
  • Measure acceptance rate, tokens per second and energy for several network delays.
  • Propose a rule that adapts the number of drafted tokens to the measured delay and acceptance rate.

What you will build and deliver

  • A working device + edge prototype and its measurements.
  • The adaptive draft-length rule with its evaluation.
  • A paper.

Skills needed

Python, PyTorch, Hugging Face Transformers; basic networking.

Research contribution & where to publish

Contribution: network-aware speculative decoding split across device and edge. Venues: IEEE INFOCOM / ICDCS workshops, MobiCom workshops, IEEE Transactions on Mobile Computing.

Key references — start reading here

  1. Y. Leviathan, M. Kalman, Y. Matias. Fast Inference from Transformers via Speculative Decoding. ICML, 2023.
  2. C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, J. Jumper. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv, 2023.
  3. H. Djigal, J. Xu, L. Liu, Y. Zhang. Machine and Deep Learning for Resource Allocation in Multi-Access Edge Computing: A Survey. IEEE Communications Surveys & Tutorials, 2022.

Interested?

Send the topic title and the courses you have taken.

Contact Dr. Hamza Djigal

Other topics in LLMs at the edge