The problem
Long prompts (documents, chat history) are slow to upload on weak networks and expensive to process. Prompt-compression methods drop unimportant tokens, but they are judged on quality alone; on a mobile link the real question is the end-to-end time and cost.
Who needs it
Mobile apps that summarize or answer about documents, field workers on weak networks, any company paying per token.
Idea of the approach
- Use two compression methods (e.g. LLMLingua-2 and a simple keyword/TF-IDF baseline).
- Measure compression time on the device, upload time under several bandwidths, and answer quality on public QA/summarization sets.
- Propose a rule that chooses the compression rate from the measured bandwidth.
- Relate to soft-prompt compression for relation extraction (GAPrompt).
What you will build and deliver
- Benchmark scripts and results.
- The adaptive compression rule.
- A paper.
Skills needed
Python, NLP basics, Hugging Face Transformers.
Research contribution & where to publish
Contribution: bandwidth-adaptive compression with an end-to-end latency model. Venues: Findings of EMNLP/ACL, IEEE ICC, Neurocomputing, Knowledge-Based Systems.
Key references — start reading here
- H. Jiang, Q. Wu, C.-Y. Lin, Y. Yang, L. Qiu. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. EMNLP, 2023.
- Z. Pan, Q. Wu, H. Jiang, M. Xia, X. Luo, J. Zhang, et al.. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression. Findings of ACL, 2024.
- GAPrompt authors (H. Djigal et al.). GAPrompt: A soft prompt compression model for few-shot relation extraction. Knowledge-Based Systems, 2025.
- H. Djigal, J. Xu, L. Liu, Y. Zhang. Machine and Deep Learning for Resource Allocation in Multi-Access Edge Computing: A Survey. IEEE Communications Surveys & Tutorials, 2022.
Interested?
Send the topic title and the courses you have taken.
Contact Dr. Hamza DjigalOther topics in LLMs at the edge
Splitting a small LLM between a smartphone and an edge server: latency- and energy-aware partitioning
An LLM request router: on-device, edge or cloud model, chosen by cost, latency and answer quality
Scheduling LLM inference requests on a shared edge GPU server with KV-cache awareness
An offline campus assistant on a quantized small language model running on a Raspberry Pi / Jetson