The problem
Companies that put a chatbot in front of customers pay for every call to a large cloud model, although many questions ("opening hours?", "reset my password") could be answered by a small free model. Routing every question to the right model saves money and time, but a wrong route gives a bad answer.
Who needs it
Customer-service chatbots, e-commerce assistants, banks and telecom help desks: LLM cost is now one of their main IT expenses.
Idea of the approach
- Collect a set of real-looking customer questions (public FAQ datasets) with reference answers.
- Answer them with 3 models of different size (local small, local medium, a cloud API) and score the answers automatically (and by hand for a sample).
- Train a light classifier (e.g. a small BERT or logistic regression on embeddings) that predicts which model is enough for a question.
- Measure cost saved vs quality lost, and add a confidence threshold that sends doubtful questions to the larger model.
What you will build and deliver
- A router service with an API and a small web chat demo.
- A labelled dataset of questions with the quality of each model's answer.
- A cost–quality curve showing the savings.
- Paper and demo.
Skills needed
Python, REST APIs, basic machine learning (scikit-learn), prompt engineering.
Research contribution & where to publish
Contribution: routing that also accounts for latency and on-device execution (three tiers instead of two), evaluated on a domain-specific workload. Venues: EMNLP / ACL industry tracks, IEEE CLOUD, Future Generation Computer Systems.
Key references — start reading here
- L. Chen, M. Zaharia, J. Zou. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv, 2023.
- I. Ong, A. Almahairi, V. Wu, W.-L. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous, I. Stoica. RouteLLM: Learning to Route LLMs with Preference Data. arXiv, 2024.
- D. Ding, A. Mallick, C. Wang, R. Sim, S. Mukherjee, V. Ruhle, L. V. S. Lakshmanan, A. H. Awadallah. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. ICLR, 2024.
Interested?
Send the topic title and the courses you have taken.
Contact Dr. Hamza DjigalOther topics in LLMs at the edge
Splitting a small LLM between a smartphone and an edge server: latency- and energy-aware partitioning
Scheduling LLM inference requests on a shared edge GPU server with KV-cache awareness
An offline campus assistant on a quantized small language model running on a Raspberry Pi / Jetson
Prompt compression for LLM offloading over low-bandwidth networks