Left: Examples from HealthVer [1]; Right: Example of a claim that is supported and refuted by different evidence [2]
With the rapid growth of biomedical research and the concurrent rise in misinformation, ensuring the accuracy of claims about treatment effectiveness is increasingly critical. Inaccurate or misleading information can have profound implications, influencing patient care and treatment decisions. Addressing this challenge involves advancing methods for verifying biomedical claims, particularly by determining how well claims align with evidence derived from medical studies or clinical trial reports. Supported by a grant from Accenture, a global leader in consulting, technology, and outsourcing, this research explores a natural language inference (NLI) framework powered by large language models (LLMs). The framework focuses on integrating domain-specific and factual evidence, ensuring that generated responses are accurate and aligned with task-specific requirements.
Example from NLI4CT [3] shows that multi-hop reasoning is required to verify complex statements include multiple pieces of information to be checked.
In this research, we intend to develop the most suitable prompting strategies and evidence-based approach along with the key implementation aspects including textual context analysis, multi-hop reasoning.
- Textual context analysis involves mechanisms of analyzing the context provided in the input prompt or conversation to identify relevant information and determine the appropriate evidence for retrieval.
- Multi-hop reasoning strategies enable the LLM-based agent to reason over multiple pieces of information and make inferences based on complex terms and their relationships.
References
[1] Mourad Sarrouti, Asma Ben Abacha, Yassine M’rabet, and Dina Demner-Fushman. 2021. Evidence-based fact-checking of health-related claims. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3499–3512.
[2] Mourad Sarrouti, Asma Ben Abacha, Yassine M’rabet, and Dina Demner-Fushman. 2021. Evidence-based fact-checking of health-related claims. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3499–3512.
[3] Maël Jullien, Marco Valentino, Hannah Frost, Paul O’regan, Donal Landers, and André Freitas. 2023. SemEval-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Data. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2216–2226, Toronto, Canada. Association for Computational Linguistics.
