Skip to content

README.md

Latest commit

 

History

History
80 lines (54 loc) · 4.02 KB

README.md

File metadata and controls

80 lines (54 loc) · 4.02 KB
Raw

AI-Powered Document Management API

This project is a sophisticated, AI-powered Flask API for managing and understanding documents. It provides a complete, end-to-end solution for uploading documents, processing them asynchronously, extracting key information, and conversing with them through a chat interface.

Built on a modern, scalable architecture, it leverages MongoDB Atlas for data persistence and Redis for message queuing.

Key Features

  • Document Upload: Securely upload documents (PDF, DOCX, TXT, etc.) via a REST API.
  • Asynchronous AI Processing: Offloads heavy AI tasks to a Celery worker using Redis, ensuring the API remains responsive and fault-tolerant.
  • MongoDB Atlas: The single source of truth for all data, including file storage (GridFS), metadata, and vector embeddings.
  • Conversational AI Chat: A powerful chat endpoint uses MongoDB Atlas Vector Search to find relevant documents and a Large Language Model (LLM) to answer questions.
  • Failure Recovery: An endpoint to re-trigger the AI processing for documents that may have failed.
  • Human-in-the-Loop: Endpoints allow users to validate or correct the AI's extracted data.

Getting Started

Prerequisites

  1. MongoDB Atlas Account: A MongoDB Atlas cluster with Vector Search enabled is required.
  2. Redis: A running Redis server. You can run this locally using Docker or use a managed cloud service.
  3. Project Dependencies: The required Python packages are listed in requirements.txt.

1. Configure Environment Variables

  1. Copy the example file: cp .env.example .env
  2. Edit .env and set the MONGO_URI and CELERY_BROKER_URL to your service addresses.

2. Activate the Virtual Environment

source .venv/bin/activate

3. Run the Application

You need to run the services in separate terminals:

  • Terminal 1: Flask Server: ./devserver.sh
  • Terminal 2: Celery Worker: celery -A main.celery worker --loglevel=info
  • Terminal 3 (if running locally): Redis Server: Ensure your Redis server is running.

API Endpoints

Here is a summary of the available API endpoints.

Documents

  • POST /documents

    • Description: Uploads a new document. The file should be sent as multipart/form-data in the file field.
    • On Success: Returns 202 Accepted with a JSON object of the created document. The document is queued for AI processing.

  • GET /documents

    • Description: Retrieves a list of all documents and their current status.

  • GET /documents/<doc_id>

    • Description: Retrieves the full details for a single document, including its status, text, and extracted KVPs.

  • GET /documents/search

    • Description: Searches for documents by filename based on a query string.
    • Query Parameter: q=<search_term>
    • On Success: Returns a list of matching documents.

  • POST /documents/<doc_id>/reprocess

    • Description: (Failure Recovery) Re-triggers the asynchronous AI processing for a document. This is useful if a document has a Processing Failed status.
    • On Success: Returns 202 Accepted and re-queues the document for processing.

  • PUT /documents/<doc_id>/kvp

    • Description: (Human-in-the-Loop) Updates the Key-Value Pairs for a document after manual review. The request body should be a JSON object of the new KVPs.
    • On Success: The document's status is updated to Validated.

  • PUT /documents/<doc_id>/recategorize

    • Description: (Human-in-the-Loop) Manually changes the category of a document. The request body should be a JSON object with a new_category key and an optional explanation key.
    • On Success: The document's status is updated to Re-categorized.

Chat

  • POST /chat
    • Description: Asks a question about the processed documents. The request body should be a JSON object with a query key.
    • On Success: Returns an answer generated by the LLM, along with the source documents used to create the answer.