The problem
Phones are too weak to run a language model fully and fast, while sending every request to the cloud is slow, costly and exposes user data. A model is a stack of layers: the first layers can run on the phone and the intermediate result is sent to an edge server that finishes the work. The best split point changes with the phone's battery, the network speed and the server load.
Who needs it
Phone makers and app companies (assistants, translation, keyboard prediction) want on-device AI that stays fast and private; telecom operators sell edge servers in their 5G networks.
Idea of the approach
- Choose a small open model (e.g. Qwen2.5-0.5B or TinyLlama) and measure the time and output size of every layer on a phone (or a Raspberry Pi as a stand-in) and on a laptop acting as the edge server.
- Build a cost model: phone compute time + transfer time (size / bandwidth) + edge compute time, and energy on the phone.
- Write a decision policy that picks the split point from the measured bandwidth (start with an exhaustive search, then a light learned predictor).
- Compare with three baselines: all on the phone, all on the edge, fixed split.
What you will build and deliver
- A working phone + edge prototype that answers prompts through the split model.
- A measurement dataset (layer times, output sizes, bandwidth traces).
- The split policy and an evaluation under 3G / 4G / Wi-Fi conditions.
- A short paper and a demo video.
Skills needed
Python, PyTorch, basic networking (sockets or gRPC); Android or Raspberry Pi is a plus.
Research contribution & where to publish
Contribution: an adaptive split policy for decoder-only LLMs, where the intermediate state grows with every generated token (unlike classic CNN splitting). Target venues: IEEE INFOCOM / IEEE ICDCS workshops, IEEE Internet of Things Journal, Computer Networks.
Key references — start reading here
- Y. Kang, J. Hauswald, C. Gao, A. Rovinski, T. Mudge, J. Mars, L. Tang. Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge. ASPLOS, 2017.
- S. Teerapittayanon, B. McDanel, H. T. Kung. Distributed Deep Neural Networks Over the Cloud, the Edge and End Devices. IEEE ICDCS, 2017.
- H. Djigal, J. Xu, L. Liu, Y. Zhang. Machine and Deep Learning for Resource Allocation in Multi-Access Edge Computing: A Survey. IEEE Communications Surveys & Tutorials, 2022.
- J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, S. Han. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. MLSys, 2024.
Interested?
Send the topic title and the courses you have taken.
Contact Dr. Hamza DjigalOther topics in LLMs at the edge
An LLM request router: on-device, edge or cloud model, chosen by cost, latency and answer quality
Scheduling LLM inference requests on a shared edge GPU server with KV-cache awareness
An offline campus assistant on a quantized small language model running on a Raspberry Pi / Jetson
Prompt compression for LLM offloading over low-bandwidth networks