← All capstone topics
Industry + research NLP and data quality

Automatic, trustworthy feedback on student programming assignments with LLMs

Give students instant hints on their code — what is wrong and where to look — without giving away the solution, and check that the hints are correct.

Level: Undergraduate Format: Individual project Duration: 2 semesters Supervisor: Dr. Hamza Djigal

The problem

Large classes make individual feedback on code slow. LLMs can explain errors, but they sometimes give wrong advice or simply hand out the full solution. Combining automatic tests with an LLM that only explains failing tests, and checking its hints, could make feedback fast, correct and pedagogical.

Who needs it

Education technology companies, online course platforms, universities.

Idea of the approach

  • Collect anonymised (or synthetic) student submissions for a set of Java/Python exercises with unit tests.
  • Generate hints with an open model from the failing tests and the code; forbid full solutions in the prompt and check it automatically.
  • Evaluate hint correctness with instructors and whether students fix their code faster (small user study).
  • Integrate the hints in a learning platform (e.g. the Learning Hub labs).

What you will build and deliver

  • A hint-generation service connected to an auto-grader.
  • Annotated evaluation set and user-study results.
  • A paper.

Skills needed

Python or Java, web development, interest in teaching.

Research contribution & where to publish

Contribution: test-grounded, solution-free LLM hints with measured correctness and learning effect. Venues: ACM SIGCSE, ITiCSE, Computers & Education.

Key references — start reading here

  1. E. Kasneci, K. Sessler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, et al.. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 2023.
  2. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, et al.. Evaluating Large Language Models Trained on Code. arXiv, 2021.
  3. Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, et al.. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 2023.

Interested?

Send the topic title and the courses you have taken.

Contact Dr. Hamza Djigal

Other topics in NLP and data quality