The problem
Large classes make individual feedback on code slow. LLMs can explain errors, but they sometimes give wrong advice or simply hand out the full solution. Combining automatic tests with an LLM that only explains failing tests, and checking its hints, could make feedback fast, correct and pedagogical.
Who needs it
Education technology companies, online course platforms, universities.
Idea of the approach
- Collect anonymised (or synthetic) student submissions for a set of Java/Python exercises with unit tests.
- Generate hints with an open model from the failing tests and the code; forbid full solutions in the prompt and check it automatically.
- Evaluate hint correctness with instructors and whether students fix their code faster (small user study).
- Integrate the hints in a learning platform (e.g. the Learning Hub labs).
What you will build and deliver
- A hint-generation service connected to an auto-grader.
- Annotated evaluation set and user-study results.
- A paper.
Skills needed
Python or Java, web development, interest in teaching.
Research contribution & where to publish
Contribution: test-grounded, solution-free LLM hints with measured correctness and learning effect. Venues: ACM SIGCSE, ITiCSE, Computers & Education.
Key references — start reading here
- E. Kasneci, K. Sessler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, et al.. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 2023.
- M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, et al.. Evaluating Large Language Models Trained on Code. arXiv, 2021.
- Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, et al.. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 2023.
Interested?
Send the topic title and the courses you have taken.
Contact Dr. Hamza DjigalOther topics in NLP and data quality
Few-shot relation extraction to build knowledge graphs from company documents (contracts, maintenance reports)
A retrieval-augmented (RAG) assistant over enterprise technical documents, with hallucination checking
Quality control of crowdsourced and LLM-generated labels: aggregating unreliable annotators