Open Source Projects
Here are some of the projects that may be useful to you:
Awesome-Hallu-Eval
A Comprehensive Collection of Hallucination Evaluation Methods
This is a curated list of evaluators designed to assess model hallucination. Here, you can easily find the right tools you need to evaluate and analyze hallucination behavior in language models.
Key Features:
- Comprehensive Coverage: Includes evaluation methods from both before and after the LLM era
- Categorized Methods: Organized by evaluation perspective (Source-Free vs. With-Fact)
- Detailed Documentation: Each method includes data sources, models used, evaluation metrics, and implementation details
- Active Maintenance: Regularly updated with the latest hallucination detection techniques
Research Areas Covered:
- Text Summarization hallucination detection
- Question Answering factuality evaluation
- Dialogue generation consistency assessment
- Multi-modal hallucination detection
- Cross-lingual hallucination evaluation
Impact:
- Potentially used by the NLP research community
- Serves as a go-to resource for hallucination evaluation
FHSumBench
Evaluating LLMs’ Assessment of Mixed-Context Hallucination Through the Lens of Summarization
This project provides the data and code for our research on evaluating how large language models assess mixed-context hallucination through summarization tasks.
Research Focus:
- Mixed-Context Analysis: Evaluating how LLMs handle conflicting information in source materials
- Self-Assessment Capabilities: Understanding LLMs’ ability to detect their own hallucination patterns
- Summarization Lens: Using summarization as a framework to study hallucination assessment
Key Contributions:
- Novel dataset for mixed-context hallucination evaluation
- Framework for assessing LLM self-evaluation capabilities
- Insights into hallucination detection limitations
Technical Approach:
- Creates scenarios with mixed or conflicting information
- Evaluates LLM performance in detecting inconsistencies
- Analyzes self-assessment accuracy of language models
For more details about any specific project, feel free to contact me at siya.qi@kcl.ac.uk