The problem
Students ask the same practical questions all the time. A cloud chatbot raises privacy and cost issues and fails without internet. Quantized small models can now run on cheap hardware, but answer quality, speed and energy on such devices are not well documented for real use.
Who needs it
Universities, hospitals, factories and kiosks that need an assistant on-site and offline (privacy rules, poor connectivity).
Idea of the approach
- Build a small document collection (public university handbook pages).
- Retrieval-augmented generation: embed the documents, retrieve the relevant passages, answer with a 4-bit quantized model (llama.cpp / Ollama).
- Compare 2–3 models and quantization levels on accuracy, speed (tokens/s) and energy (USB power meter).
- Simple web or kiosk interface.
What you will build and deliver
- The running assistant on the device, with its interface.
- A test set of questions with expected answers and the evaluation results.
- A deployment guide.
Skills needed
Python, Linux, basic web development; no GPU needed.
Research contribution & where to publish
Contribution: an energy-and-accuracy benchmark of quantized RAG on low-cost edge hardware. Venues: IEEE EDGE, ACM SEC (workshops), IEEE Access, Sensors.
Key references — start reading here
- P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, et al.. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS, 2020.
- E. Frantar, S. Ashkboos, T. Hoefler, D. Alistarh. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR, 2023.
- J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, S. Han. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. MLSys, 2024.
- T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS, 2023.
Interested?
Send the topic title and the courses you have taken.
Contact Dr. Hamza DjigalOther topics in LLMs at the edge
Splitting a small LLM between a smartphone and an edge server: latency- and energy-aware partitioning
An LLM request router: on-device, edge or cloud model, chosen by cost, latency and answer quality
Scheduling LLM inference requests on a shared edge GPU server with KV-cache awareness
Prompt compression for LLM offloading over low-bandwidth networks