← All capstone topics
Research project NLP and data quality

Quality control of crowdsourced and LLM-generated labels: aggregating unreliable annotators

Combine labels from several imperfect sources — human crowd workers and LLMs — into one reliable label, by learning how good each source is.

Level: Undergraduate Format: Individual project Duration: 1–2 semesters Supervisor: Dr. Hamza Djigal

The problem

Companies label training data with crowds and, more and more, with LLMs. Both make mistakes, and LLMs make systematic ones. Classic aggregation (majority vote, Dawid–Skene) assumes independent human workers; mixing humans and LLMs is new.

Who needs it

Data-labelling companies, any team building training datasets.

Idea of the approach

  • Public crowdsourcing datasets with gold labels; add labels produced by 2–3 LLMs.
  • Implement majority vote, Dawid–Skene and a worker-ability model (as in the IPM 2024 paper).
  • Extend to treat LLMs as annotators with correlated errors; measure accuracy and cost per correct label.

What you will build and deliver

  • Aggregation library and benchmark.
  • Paper.

Skills needed

Python, probability/statistics, machine learning.

Research contribution & where to publish

Contribution: hybrid human–LLM label aggregation with correlated LLM errors. Venues: HCOMP, CIKM, Information Processing & Management.

Key references — start reading here

  1. A. P. Dawid, A. M. Skene. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Journal of the Royal Statistical Society, Series C (Applied Statistics), 1979.
  2. H. Djigal et al.. Robust Annotation Aggregation in Crowdsourcing via Enhanced Worker Ability Modeling. Information Processing & Management, 2024.
  3. F. Gilardi, M. Alizadeh, M. Kubli. ChatGPT outperforms crowd workers for text-annotation tasks. PNAS, 2023.

Interested?

Send the topic title and the courses you have taken.

Contact Dr. Hamza Djigal

Other topics in NLP and data quality