🧹 Housekeeping:
- Removed src/functionapp/ directory containing old Azure Functions code
- Removed Function App configuration files (.funcignore, host.json)
- Cleaned up legacy Docker and configuration files from Functions implementation

Since we've successfully migrated to Azure Container Apps and validated the
complete end-to-end workflow, the old Functions code is no longer needed.
This cleanup reduces repository size and eliminates confusion between
the old and new architectures.

🎯 Key Improvements:
- Replace problematic streamlit_pdf_viewer with robust iframe fallback
- Add comprehensive caching (@st.cache_data) for 60s-5min TTL
- Upgrade all dependencies to latest performant versions
- Increase container resources (1.0 CPU, 2Gi memory, 5 replicas)
- Add cache clear button in sidebar for better UX

🐛 Fixes:
- Resolve PDF viewer component errors causing system instability
- Fix file loading and display issues in Explore Data tab
- Improve error handling with graceful fallbacks
- Clean up debug output and unused imports

⚡ Performance:
- Streamlit 1.40.2, pandas 2.2.3, plotly 5.24.1
- Multi-tiered caching strategy for documents, blobs, and details
- Optimized backend client with connection pooling
- Smart file size handling for different PDF sizes

🔧 Infrastructure:
- Updated Bicep templates with enhanced container app scaling
- Improved Azure authentication and fallback mechanisms
- Dockerization improvements for both frontend and backend
- Better resource allocation and concurrent request handling

✅ Successfully deployed and tested on Azure Container Apps

- Add delete_document and reprocess_document endpoints to backend API
- Add corresponding methods to frontend backend client
- Remove unused clear_cache function from frontend
- Improve error handling in backend client methods
- Ensure complete API coverage for document management operations

✅ Complete implementation of configuration-driven dataset management
✅ Enhanced frontend UX with clean interface
✅ Robust backend API with full CRUD operations
✅ Successfully deployed and tested on Azure

Key improvements:
- Removed unnecessary UI elements (cache button, success messages)
- Implemented single configuration structure for all datasets
- Added delete and reprocess API endpoints
- Ensured consistent dataset processing flow
- Cleaned up temporary development files

System is now production-ready with improved user experience
and reliable configuration management.

- Add Event Grid system topic and subscription for blob storage events
- Implement Logic App for async file processing triggered by blob uploads
- Support subdirectory uploads (datasets/dataset_name/file) with dataset name extraction
- Update frontend to upload directly to blob storage instead of backend API
- Add /api/process-file endpoint for Logic App to trigger backend processing
- Enable true async processing: UI upload → blob → Event Grid → Logic App → backend
- Improve upload performance and reliability with event-driven architecture

Changes:
- infra/main.bicep: Add Event Grid, Logic App, and blob trigger configuration
- frontend/process_files.py: Direct blob upload using Azure SDK and managed identity
- src/containerapp/main.py: Add /api/process-file endpoint for async processing
- Support for files uploaded to datasets/dataset_name/ structure

Validated end-to-end: file upload → event trigger → async processing → success

- Reduced cache TTL from 30s to 15s for faster document list updates
- Reduced document details cache from 5min to 1min for better responsiveness
- Enhanced comments to clarify that UI bypasses backend when Cosmos DB available
- Changed error messages to warnings for better UX when Cosmos fallback occurs
- UI now prioritizes direct Cosmos DB access for all document operations

This ensures the UI remains responsive even when backend is busy processing files.

- Fix Streamlit serialization error in Processing Details tab
- Ensure all values in DataFrame 'Value' column are strings
- Add explicit str() conversions for mixed data types
- Add df_details['Value'].astype(str) to prevent type conflicts
- Resolves 'Expected bytes, got int object' error
- Maintains proper formatting for numeric values

- Add UI controls in frontend for setting max_pages_per_chunk per dataset
- Add number input fields for both editing existing datasets and creating new ones
- Update backend to fetch and use configurable max_pages_per_chunk value
- Extend Cosmos DB configuration schema to store max_pages_per_chunk
- Modify PDF splitting logic to use dynamic value instead of hardcoded 10
- Maintain backward compatibility with default value of 10 for existing datasets
- Update initialize_document function to accept and store max_pages_per_chunk
- Enhanced fetch_model_prompt_and_schema to return max_pages_per_chunk

Tested and deployed successfully:
- Frontend UI allows configuration per dataset
- Backend processing respects the configured values
- All changes are backward compatible
- Feature is live and operational in Azure Container Apps

- Fixed serialization issue where background tasks were blocking on future.result()
- Removed timeout blocking that was preventing parallel file processing
- Added enhanced error handling with proper status tracking
- Files can now be processed truly in parallel instead of serially
- Logic Apps can now trigger simultaneous processing without waiting

Technical changes:
- Modified process_blob_event() to fire-and-forget pattern
- Added process_blob_with_error_handling() for better error tracking
- Enhanced status tracking in Cosmos DB during processing
- Maintains backward compatibility with existing processing pipeline

This enables the system to process multiple files simultaneously as intended,
dramatically improving throughput for batch document processing scenarios.

🔧 CORE FIXES:
- Replace ThreadPoolExecutor context manager with global persistent pool
- Initialize 10-worker ThreadPoolExecutor at application startup
- Eliminate singleton Document Intelligence client bottleneck
- Create per-request Azure service clients for true concurrency

🚀 PERFORMANCE IMPROVEMENTS:
- Multiple files now process simultaneously instead of serially
- Logic Apps can trigger concurrent processing without blocking
- Enhanced thread-level logging for monitoring parallel execution
- Proper resource cleanup on application shutdown

🛠️ TECHNICAL CHANGES:
- Global executor: ThreadPoolExecutor(max_workers=10) in lifespan
- Per-request clients: get_document_intelligence_client()
- Enhanced logging: Thread ID tracking in OCR and processing
- Fire-and-forget pattern: No blocking on future.result()

TESTED: Confirmed working in Azure deployment with thread ID logs
showing parallel execution across multiple worker threads.

This resolves the serialization issue where files were processed
one at a time despite parallel infrastructure setup.

- Remove redundant 'Current Settings' section since input field is prepopulated
- Fix navigation icons for Concurrency (🚀) and Instructions (📋) tabs
- Auto-load current settings without requiring manual refresh
- Prepopulate concurrency input field with current value
- Add comprehensive help sections with technical details and troubleshooting
- Implement cleaner, centered form layout
- Add contextual guidance based on selected concurrency value
- Improve user experience with smart feedback and status indicators
- Maintain all powerful concurrency control functionality
- Deploy updated frontend with streamlined interface

- Fixed timing and state management in chunked processing
- Only mark steps as completed after ALL chunks are processed
- Added robust evaluation result merging with confidence scores
- Improved error handling for incomplete processing steps
- Added proper resource cleanup for temporary files and directories
- Enhanced logging for each processing step and chunk operations
- Aggregated timing metrics for accurate total processing time
- Updated merge_evaluation_results and _merge_values_with_confidence functions
- Refactored process.py functions to support delayed state updates

- Updated get_config() to ignore cosmos_config_container parameter
- Modified all OpenAI-related function calls to pass None instead of conf_container
- Updated API endpoints to reflect read-only environment variable usage
- Removed Cosmos DB dependency for OpenAI configuration
- Ensures configuration is sourced exclusively from environment variables

- Update backend PUT /api/openai-settings to accept environment variable updates
- Add frontend form for runtime environment variable updates
- Organize frontend with tabs for runtime vs persistent updates
- Maintain backward compatibility with legacy database config
- Add clear warnings about temporary nature of runtime updates

- Correct AZURE_OPENAI_API_KEY to AZURE_OPENAI_KEY
- Correct AZURE_OPENAI_DEPLOYMENT_NAME to AZURE_OPENAI_MODEL_DEPLOYMENT_NAME
- Ensure environment variable updates use correct names

- Remove 'Processing Options' title for cleaner interface
- Change 'Max Pages per Chunk' to 'Document Chunk Size (pages)' for clarity
- Enhance help text for all processing options with detailed explanations:
  - OCR: Explain what it captures and when it's essential
  - Images: Detail visual processing capabilities and use cases
  - Summary: Describe content and benefits for organization
  - Evaluation: Explain quality checks and validation features
  - Chunk size: Provide guidance on trade-offs and recommendations
- Make tooltips more user-friendly and actionable
- Improve user decision-making with comprehensive descriptions

- Fix backend configuration loading to use correct schema and prompt files
- Add configuration refresh endpoint for runtime updates
- Implement truncation detection by checking incomplete JSON and OpenAI finish_reason
- Provide actionable error messages advising users to reduce chunk size
- Update test script to validate truncation error detection and reporting
- Ensure robust error handling for max completion token limits

- Replace manual text input/button with st.chat_input for modern UX
- Use st.chat_message for displaying chat history
- Remove all emoji/icons for clean professional look
- Remove predefined question suggestions for streamlined interface
- Fix backend unpacking errors in Cosmos DB connection
- Fix document fetching with cross-partition queries
- Fix session state modification errors with proper input handling
- Deploy all changes to Azure for live testing

The chat interface now uses official Streamlit chat components
and provides a modern, robust chat experience.

- Replace document chunk merging with structured page range JSON
- Create page range structure instead of merging extracted data
- Apply same approach to evaluation results
- Add helper functions for page range structure creation
- Keep backward compatibility with existing deep merge functions
- Deploy all changes to Azure

…ture

- Replace merging logic with page range structure (pages_1-10, pages_11-20, etc.)
- Create create_page_range_structure() function to organize chunks by page ranges
- Apply page range structure to both extraction and evaluation data
- Preserve individual chunk data instead of merging for better traceability
- Update GPT extraction and evaluation sections to use new structure
- Deploy changes to Azure backend for testing

This allows users to see exactly which data came from which page ranges
of the document, providing better granular access to chunked results.

Instead of merging chunked document results, now creates structured JSON
with page range keys (e.g., 'pages_1-10', 'pages_11-20') for better
organization and access to individual chunk data.

Changes:
- Add create_page_range_structure() function to generate page-based JSON
- Add create_page_range_evaluations() for evaluation page ranges
- Update GPT extraction to use page range structure instead of merging
- Update GPT evaluation to maintain individual evaluations per page range
- Remove dependency on merge_extracted_data() for chunked documents
- Single documents continue to work as before
- Deployed backend with new structure to Azure for testing

This allows frontend to access specific page ranges individually
and maintains separation of data from different document sections.

- Removed max_tokens=4000 from all OpenAI chat completion calls
- Updated error handling to remove references to max_tokens_configured
- Cleaned up technical details in error responses
- Allow models to use their natural token limits without artificial constraints
- This enables more complete responses and reduces truncation issues

The OpenAI models will now use their full context window capacity
without being artificially limited by hardcoded max_tokens values.

- Remove test files (9 files): test_*.py files
- Remove debug files (3 files): debug_*.py files
- Remove test documents (5 files): test-*.txt and test config files
- Remove implementation docs (4 files): implementation summary markdown files
- Remove utility scripts (6 files): one-off configuration scripts
- Remove frontend backup files (3 files): app_backup.py, etc.
- Remove infrastructure files (3 files): unused logic app configs and generated ARM template
- Remove docker directory: not used with Container Apps deployment
- Remove src/containerapi: unused API implementation

Total cleanup: 30+ files removed, significantly reduced workspace clutter
All core functionality preserved: containerapp, frontend, infrastructure, docs

- Fix Logic App name mismatch in Bicep template (missing 'v2' in env var)
- Add enhanced error diagnostics to concurrency settings frontend
- Add diagnostic endpoint /api/concurrency/diagnostics for troubleshooting
- Improve error reporting with specific guidance for 503 errors

The Logic App was named 'logic-argus-v2-{token}' but env var was set to
'logic-argus-{token}', causing LogicAppManager to fail finding the resource.

- Split 1675-line main.py into 6 focused modules:
  - main.py: FastAPI app entry point and routing (139 lines)
  - models.py: Data models and classes (40 lines)
  - dependencies.py: Azure client management (112 lines)
  - logic_app_manager.py: Logic App concurrency management (217 lines)
  - blob_processing.py: Document processing pipeline (407 lines)
  - api_routes.py: All API endpoint handlers (635 lines)

- Benefits:
  - Improved maintainability and code organization
  - Better testability with isolated modules
  - Easier debugging and development
  - Clear separation of concerns

- Updated Dockerfile to use modular structure
- Added comprehensive test suite for validation
- Backed up original main.py as main_old.py
- Updated README with backend architecture documentation

- All 20 API endpoints preserved and functional
- No breaking changes to the API interface

- Split main.py into focused modules:
  - models.py: Data models and Pydantic schemas
  - logic_app_manager.py: Logic App management functionality
  - blob_processing.py: Document/blob processing with OCR integration
  - api_routes.py: FastAPI route handlers
  - dependencies.py: Azure clients and global state management
  - main.py: New modular FastAPI application entrypoint

- Fixed all import dependencies and relative import issues
- Updated Dockerfile for modular structure
- Added missing functions to blob_processing.py (create_page_range_structure, create_page_range_evaluations)
- Maintained backward compatibility with existing API endpoints
- Verified deployment and functionality on Azure Container Apps
- All endpoints tested and working correctly

Resolves backend modularity and maintainability issues

- Removed main_old.py (old monolithic backup)
- Removed test_modular_structure.py (temporary test file)
- Removed test_startup_simulation.py (temporary test file)
- Cleaned up all __pycache__ directories

…cumentation

- Add comprehensive architecture diagrams with Mermaid
- Include detailed API reference with real examples
- Add production deployment guide and scaling recommendations
- Include security, compliance, and enterprise features documentation
- Add advanced use cases for healthcare, finance, and legal industries
- Include comprehensive troubleshooting and monitoring guides
- Add professional team showcase and contribution guidelines
- Include performance evaluation and quality assurance sections
- Add visual elements, badges, and professional formatting
- Transform from basic documentation to world-class project showcase

- Remove 🔐 Security & Infrastructure section from mermaid diagram
- Add fancy colorful styling to mermaid diagram boxes
- Remove step 3 for Azure OpenAI credentials configuration (not needed)
- Update dataset creation section with clearer explanation of what datasets are and when to create them
- Rewrite usage examples to prioritize frontend UI upload method over blob storage
- Remove 🧪 Testing & Quality Assurance section
- Simplify team section by removing job titles/tasks, keeping just names

…ions

- Expand backend architecture to full project structure
- Add accurate descriptions for all directories and key files based on code analysis
- Include infrastructure, frontend, evaluation, demo, and documentation sections
- Use meaningful emojis to categorize different types of files and functionality
- Provide clear hierarchy showing the modular architecture of ARGUS

- Updated Logic App Manager default concurrency value from 1 to 5
- Updated global processing semaphore initialization to 5 permits
- Updated frontend components to use 5 as default fallback value
- Changes apply to new Logic Apps and when no specific concurrency is configured
- Maintains consistency across backend and frontend layers

…edundant page headers

- Updated app title back to original with brain emoji
- Added concise information box above dataset selector explaining what datasets are
- Removed page headers from Explore Data and Settings tabs to save space
- Enhanced page configuration with proper title and icon