The problem
Vendors publish peak numbers that do not reflect real applications. Developers choosing a device for a product need comparable measurements of latency, accuracy after compression, power and cost on realistic tasks (detection, speech, small LLM).
Who needs it
IoT product companies, system integrators, schools building AI labs.
Idea of the approach
- Select 5–6 workloads (image classification, detection, keyword spotting, small LLM, embedding search).
- Run them on 3–4 devices with identical scripts; measure latency, throughput, power (USB meter), accuracy after quantization.
- Publish an open repository with scripts, results and a web page of charts.
What you will build and deliver
- Benchmark suite and public results page.
- Paper (benchmark track).
Skills needed
Python, Linux, careful experimentation.
Research contribution & where to publish
Contribution: a reproducible, energy-aware edge benchmark including small LLMs. Venues: IEEE IISWC, ACM SEC, IEEE Embedded Systems Letters, Sensors.
Key references — start reading here
- V. J. Reddi, C. Cheng, D. Kanter, P. Mattson, G. Schmuelling, C.-J. Wu, et al.. MLPerf Inference Benchmark. ACM/IEEE ISCA, 2020.
- A. Ignatov, R. Timofte, W. Chou, K. Wang, M. Wu, T. Hartley, L. Van Gool. AI Benchmark: Running Deep Neural Networks on Android Smartphones. ECCV Workshops, 2018.
- M. Satyanarayanan. The Emergence of Edge Computing. IEEE Computer, 2017.
Interested?
Send the topic title and the courses you have taken.
Contact Dr. Hamza Djigal