The problem
RAG assistants are being deployed everywhere, but they still invent facts. In technical settings (equipment manuals, safety procedures) a wrong answer is costly. Automatic checking of whether each sentence is supported by the retrieved text is the missing piece.
Who needs it
Manufacturing, energy, IT support, any company with large technical documentation.
Idea of the approach
- Public technical manuals as the corpus; a question set with answers.
- RAG pipeline with an open model; citations to passages.
- Support checking: an NLI model or LLM judge verifies each sentence against the passages; unsupported sentences are flagged.
- Evaluate with RAGAS metrics and human checks.
What you will build and deliver
- Assistant with citations and warnings; evaluation set.
- Paper.
Skills needed
Python, web development, NLP basics.
Research contribution & where to publish
Contribution: sentence-level support checking evaluated on technical documentation. Venues: EMNLP/ACL industry tracks, ECIR, Information Processing & Management.
Key references — start reading here
- P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, et al.. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS, 2020.
- S. Es, J. James, L. Espinosa-Anke, S. Schockaert. RAGAS: Automated Evaluation of Retrieval Augmented Generation. EACL (demo), 2024.
- Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, et al.. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 2023.
Interested?
Send the topic title and the courses you have taken.
Contact Dr. Hamza Djigal