← All capstone topics
Research project Benchmark

A reproducible benchmark of federated learning frameworks on real edge devices

Run the same federated training tasks with popular frameworks (Flower, FedML) on real phones and Raspberry Pis, and publish how they compare in time, energy and accuracy.

Level: Undergraduate Format: Individual project Duration: 1–2 semesters Supervisor: Dr. Hamza Djigal

The problem

Federated learning papers are mostly evaluated in simulation on one server, so real-device costs (memory, energy, dropped connections) are unknown. Frameworks differ in how easy they are to deploy and how much they cost on small devices; there is no simple, reproducible comparison.

Who needs it

Companies choosing a federated learning framework for products; researchers who need realistic baselines.

Idea of the approach

  • Pick 3 tasks from the LEAF benchmark (image, text, sensor).
  • Run them with Flower and FedML on 5–10 Raspberry Pis / phones and in simulation.
  • Measure training time, energy, memory, communication and accuracy; inject device drop-outs.
  • Publish scripts and results as an open repository.

What you will build and deliver

  • Benchmark scripts, results and a public results page.
  • A paper (benchmark track).

Skills needed

Python, Linux, PyTorch; careful experimentation.

Research contribution & where to publish

Contribution: real-device, energy-aware comparison of FL frameworks. Venues: IEEE IISWC, MLSys workshops, ACM SEC, Future Generation Computer Systems.

Key references — start reading here

  1. S. Caldas, S. M. K. Duddu, P. Wu, T. Li, J. Konečný, H. B. McMahan, V. Smith, A. Talwalkar. LEAF: A Benchmark for Federated Settings. arXiv, 2018.
  2. D. J. Beutel, T. Topal, A. Mathur, X. Qiu, J. Fernandez-Marques, Y. Gao, et al.. Flower: A Friendly Federated Learning Research Framework. arXiv, 2020.
  3. C. He, S. Li, J. So, X. Zeng, M. Zhang, H. Wang, et al.. FedML: A Research Library and Benchmark for Federated Machine Learning. arXiv, 2020.
  4. B. McMahan, E. Moore, D. Ramage, S. Hampson, B. Agüera y Arcas. Communication-Efficient Learning of Deep Networks from Decentralized Data. AISTATS, 2017.

Interested?

Send the topic title and the courses you have taken.

Contact Dr. Hamza Djigal

Other topics in Benchmark