← All capstone topics
Industry project LLMs at the edge

An LLM request router: on-device, edge or cloud model, chosen by cost, latency and answer quality

Send each user question to the cheapest model that can answer it well — a tiny local model for easy questions, a large cloud model only when needed.

Level: Undergraduate Format: Individual project Duration: 1–2 semesters Supervisor: Dr. Hamza Djigal

The problem

Companies that put a chatbot in front of customers pay for every call to a large cloud model, although many questions ("opening hours?", "reset my password") could be answered by a small free model. Routing every question to the right model saves money and time, but a wrong route gives a bad answer.

Who needs it

Customer-service chatbots, e-commerce assistants, banks and telecom help desks: LLM cost is now one of their main IT expenses.

Idea of the approach

  • Collect a set of real-looking customer questions (public FAQ datasets) with reference answers.
  • Answer them with 3 models of different size (local small, local medium, a cloud API) and score the answers automatically (and by hand for a sample).
  • Train a light classifier (e.g. a small BERT or logistic regression on embeddings) that predicts which model is enough for a question.
  • Measure cost saved vs quality lost, and add a confidence threshold that sends doubtful questions to the larger model.

What you will build and deliver

  • A router service with an API and a small web chat demo.
  • A labelled dataset of questions with the quality of each model's answer.
  • A cost–quality curve showing the savings.
  • Paper and demo.

Skills needed

Python, REST APIs, basic machine learning (scikit-learn), prompt engineering.

Research contribution & where to publish

Contribution: routing that also accounts for latency and on-device execution (three tiers instead of two), evaluated on a domain-specific workload. Venues: EMNLP / ACL industry tracks, IEEE CLOUD, Future Generation Computer Systems.

Key references — start reading here

  1. L. Chen, M. Zaharia, J. Zou. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv, 2023.
  2. I. Ong, A. Almahairi, V. Wu, W.-L. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous, I. Stoica. RouteLLM: Learning to Route LLMs with Preference Data. arXiv, 2024.
  3. D. Ding, A. Mallick, C. Wang, R. Sim, S. Mukherjee, V. Ruhle, L. V. S. Lakshmanan, A. H. Awadallah. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. ICLR, 2024.

Interested?

Send the topic title and the courses you have taken.

Contact Dr. Hamza Djigal

Other topics in LLMs at the edge