Fact Checking Solution

This project process claims and verify them against provided documents using a fact-checking agent.

Project Structure

• main.py: Fact check solution
• structures.py: Defines data models and types.
• graph_builder.py: Builds the fact-checking workflow.
• data_processor.py: Handles document loading and response structuring.
• prompts.py: Contains the prompt templates for the LLM.

Setup

1. Place PDF documents in the Clinical Files directory.
2. Add claims in Flublok_Claims.json.

Run main.py and results will be saved in fact_checked_claims.json. It takes about a minute to run, with majority of the time spent adding chunks to the vector store.

High level design decisions

1. Asynchronous LLM Processing: Each claim is processed independently to ensure precise and focused fact-checking. Relevant document chunks are retrieved for each claim, and the LLM identifies the facts within these chunks that best support the claim. This asynchronous approach scales well with a large number of claims, enabling efficient processing even for datasets with thousands of claims. In contrast, combining all claims and retrievals into a single prompt is less scalable and may compromise accuracy for larger datasets.
2. Hallucination Checks: LLMs are prone to generating hallucinated content. To address this, a fuzzy matching technique is used to verify if the LLM's output aligns with the retrieved context. A match threshold of 75% is deemed sufficient for validation, taking into account special characters introduced by pdf parsing. Further details can be found in the "Response Checker" section.
3. Structured Output: The LLM returns a response in a pydantic structure, which is then used to generate a JSON schema that matches the desired output format.

Models

1. Embedding Model:

   • Model Used: text-embedding-3-small
     This model is selected for embedding document chunks, as chunks are not more than 500 characters.
   • Alternatives Considered: Larger embedding models were deemed unnecessary due to the relatively small size of each chunk. Using a smaller model also reduces cost when chunking thousands (if not millions) of pages.

2. Fact-Checking Agent:

   • Model Used: GPT-4o
     This model is chosen for its robust responses and ability to parse responses into a structured JSON format using a native API to structure outputs.
   • Alternatives Considered:
     • GPT-4o mini: While smaller and cost efficient, it was excluded due to the high cost of missing a fact. GPT-4o is more robust.
     • o1 and o3: These models are strong in reasoning tasks but lack the ability to produce structured outputs (using native APIs), making them unsuitable. They also have a longer processing time due to intermediate tokens generated.

Data Processing

Loading PDFs

The project uses the PyPDFLoader from LangChain, which leverages the pypdf library to parse PDF documents.

Potential Improvement: The Unstructured API offers significantly better performance for extracting content from documents, including tables and complex layouts. I went with pypdf given that Unstructured API is not open source, and that the priority was to develop a quick and cheap end-to-end script.

Chunking

• Chunk Overlap: 150 characters
  Overlapping chunks by 150 characters (approximately 20 words) ensures that no single fact, typically no longer than 20 words, is split across chunks. This preserves the continuity of facts, ensuring that no single fact will be split into distinct chunks.

• Chunk Size: 500 characters
  This size is chosen to ensure that each chunk contains sufficient context for semantic relevance, which is critical for extracting meaningful vector representations for accurate fact-checking.

Output Structure

The LLM response for a single claim is structured as a pydantic model with the following fields:

• Reasoning: Like with CoT prompting, this field is to generate intermediate tokens that can inform the LLM on specific facts that are representative of the claim.
• Details: This is a list of matching texts and document names that support the claim.

This structured output is used to generate a JSON schema.

Fact Checking Agent

LangGraph was used to build the fact checker to seamlessly add potential future capabilities to the agent, with each node serving a unique purpose.

• Retrieve: This node performs the following tasks:

  1. Retrieves the 30 most similar chunks based on embedding similarity using the similarity_search method. This ensures that the retrieved chunks are highly relevant to the claim being processed. Retrieving too many chunks can introduce noise, while retrieving too few may result in missing critical information.
  2. Combines the content of the retrieved chunks into a single string, including the document source and page content for each chunk. This is then passed to the generate node.

• Generate: This node generates the fact-checking response using the LLM. It performs the following tasks:

  1. Constructs a prompt using the claim and the retrieved context using predefined prompt template.
  2. Invokes the LLM to generate a response, which includes reasoning and supporting details.
  3. Validates the response using response_checker to ensure that the facts provided by the LLM match the retrieved context.
  4. If mismatches are found, the LLM is prompted again with feedback on the mistakes. This process is repeated up to two times to refine the response. Else, the mismatches are removed.
  5. Returns the final validated response, ensuring that only accurate and contextually supported facts are included.

Response Checker

To avoid hallucination, the response_checker function ensures that the facts generated by the LLM align with the retrieved context. This is achieved through fuzzy matching technique that compares each fact against the context using a similarity threshold of 75%. The function performs the following steps:

1. Preprocessing: Both the context and the fact text are normalized by removing extra whitespace for consistency.

2. Sliding Window Matching: For each fact, the function iterates through the context using a sliding window. This ensures that all possible substrings of the context are checked for similarity with the fact text. Although this is computationally expensive, it is necessary to ensure all matching texts are reliable.

3. Similarity Check: SequenceMatcher from Python's difflib library is used to compute the similarity ratio between the fact text and the context substring. If the ratio meets or exceeds the 75% threshold, the fact is considered valid. A 75% similarity threshold was chosen to account for minor discrepancies introduced by PDF parsing, such as special characters or formatting issues.

4. Mismatch Handling: If a fact does not meet the threshold, it is flagged as a mismatch. The function returns a detailed error message along with the indices of the mismatched facts. This message also includes feedback for the LLM, prompting it to refine its response by ensuring that the facts match the context verbatim. This error message is appended to the prompt template.

Future improvements

• Removing hard coded values: Replace hard coded values, such as model names, thresholds, and chunking details, with a configurable YAML file. This would allow users to easily adjust parameters without modifying the codebase.
• Using robust PDF parsing: Improve document parsing by integrating advanced tools like the Unstructured API or other libraries that handle complex layouts, tables, and images effectively.
• Constructing a knowledge graph and/or internal knowledge base: Build a structured knowledge graph to represent relationships between entities and facts, enabling advanced retrieval capabilities. A limitation of the current approach is that each chunk is treated independently.
• Dynamic chunking strategy: Implement an approach that adjusts chunk size and overlap based on document content, such as increasing overlap for dense or technical sections.
• Re ranking: Use a re-ranking model to improve retrieval precision by evaluating the top n documents retrieved by the retriever and selecting the most relevant k. This approach combines the speed of the retriever with the accuracy of the re-ranker, ensuring scalability and relevance for complex claims.