← All capstone topics
Industry project LLMs at the edge

An offline campus assistant on a quantized small language model running on a Raspberry Pi / Jetson

A question-answering assistant about university rules, courses and services that runs entirely on a small device, without internet and without sending data out.

Level: Undergraduate Format: Individual project Duration: 1–2 semesters Supervisor: Dr. Hamza Djigal

The problem

Students ask the same practical questions all the time. A cloud chatbot raises privacy and cost issues and fails without internet. Quantized small models can now run on cheap hardware, but answer quality, speed and energy on such devices are not well documented for real use.

Who needs it

Universities, hospitals, factories and kiosks that need an assistant on-site and offline (privacy rules, poor connectivity).

Idea of the approach

  • Build a small document collection (public university handbook pages).
  • Retrieval-augmented generation: embed the documents, retrieve the relevant passages, answer with a 4-bit quantized model (llama.cpp / Ollama).
  • Compare 2–3 models and quantization levels on accuracy, speed (tokens/s) and energy (USB power meter).
  • Simple web or kiosk interface.

What you will build and deliver

  • The running assistant on the device, with its interface.
  • A test set of questions with expected answers and the evaluation results.
  • A deployment guide.

Skills needed

Python, Linux, basic web development; no GPU needed.

Research contribution & where to publish

Contribution: an energy-and-accuracy benchmark of quantized RAG on low-cost edge hardware. Venues: IEEE EDGE, ACM SEC (workshops), IEEE Access, Sensors.

Key references — start reading here

  1. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, et al.. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS, 2020.
  2. E. Frantar, S. Ashkboos, T. Hoefler, D. Alistarh. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR, 2023.
  3. J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, S. Han. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. MLSys, 2024.
  4. T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS, 2023.

Interested?

Send the topic title and the courses you have taken.

Contact Dr. Hamza Djigal

Other topics in LLMs at the edge