← All capstone topics
Industry + research Federated learning and IoT

Federated LoRA fine-tuning of small LLMs across organisations without sharing data

Several organisations (hospitals, schools, company branches) adapt the same small language model to their domain together, exchanging only small LoRA adapters, never their documents.

Level: Undergraduate Format: Individual project Duration: 2 semesters Supervisor: Dr. Hamza Djigal

The problem

Fine-tuning a language model on private documents gives much better answers, but no organisation wants to send its data to another. LoRA makes fine-tuning cheap by training small adapters; how to merge adapters from different organisations with different data is still open.

Who needs it

Healthcare, legal and finance groups, multi-campus universities; data-protection laws forbid pooling the data.

Idea of the approach

  • Small model (e.g. Qwen2.5-0.5B/1.5B) with LoRA, 3–5 simulated organisations with different instruction datasets.
  • Implement federated averaging of adapters; compare with local-only and centralized fine-tuning.
  • Study heterogeneous ranks and personalization (keep part of the adapter local).
  • Measure quality, communication and GPU time.

What you will build and deliver

  • Training code and experiments.
  • Paper.

Skills needed

Python, PyTorch, Hugging Face PEFT; GPU access needed.

Research contribution & where to publish

Contribution: personalized and rank-heterogeneous federated LoRA. Venues: FL workshops at NeurIPS/ICML, ICASSP, IEEE Transactions on Mobile Computing.

Key references — start reading here

  1. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 2022.
  2. J. Zhang, S. Vahidian, M. Kuo, C. Li, R. Zhang, T. Yu, G. Wang, Y. Chen. Towards Building the FederatedGPT: Federated Instruction Tuning. IEEE ICASSP, 2024.
  3. H. Djigal et al.. ProLoRA: Resource-Efficient Personalized Federated Learning for Sensor Based Human Activity Recognition. , 2024.
  4. T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS, 2023.

Interested?

Send the topic title and the courses you have taken.

Contact Dr. Hamza Djigal

Other topics in Federated learning and IoT