Feature/phase2 observability testing - #241
Open
Abhilash Jaiswal (abhilashjaiswal0110) wants to merge 16 commits into
Open
Feature/phase2 observability testing#241Abhilash Jaiswal (abhilashjaiswal0110) wants to merge 16 commits into
Abhilash Jaiswal (abhilashjaiswal0110) wants to merge 16 commits into
Conversation
* chore: update .gitignore to exclude progress reports and documentation drafts - Add patterns to exclude *_PROGRESS.md, *_QUICKSTART.md files - Add patterns to exclude *_PLAN.md, *_REPORT.md, *_SUMMARY.md files - Add DOCUMENTATION_INDEX.md to exclusions - Prevents temporary documentation from being committed * docs: consolidate documentation and remove outdated references - Remove references to deleted progress/report files - Remove Production Ready and Security Certified badges - Clean up documentation links in Additional Documentation section - Keep essential links: Knowledge.md, Agents.md, CONTRIBUTING.md - Streamline overview section * fix: update API version requirement for o4-mini model support - Change OPENAI_API_VERSION to 2024-12-01-preview - Add comment noting o4-mini requires 2024-12-01-preview or later - Enable AZURE_OPENAI_API_VERSION for compatibility - Update documentation to reflect model-specific requirements Breaking change: o4-mini model requires API version 2024-12-01-preview * feat(samples): add working example for o4-mini model with Azure OpenAI - Add example_usage.py demonstrating correct o4-mini configuration - Update .env.example with API version 2024-12-01-preview requirement - Document temperature=1.0 requirement for o4-mini model - Include token usage tracking example Configuration notes: - o4-mini requires API version 2024-12-01-preview or later - o4-mini only supports temperature=1.0 (default) - Direct Azure OpenAI endpoint recommended over AI Foundry for this model
- Enterprise architecture with Bicep templates for Hub/Project - Managed Identity and RBAC configuration (Azure AI User role) - OpenTelemetry tracing integration with Application Insights - Agent proxy pattern for hybrid approach (existing agents + AI Foundry) - Feature flags for gradual rollout (USE_AZURE_FOUNDRY) - Copilot Studio integration guide - Production deployment with Container Apps and CI/CD - Cost estimation and troubleshooting guides Based on official Microsoft docs and Azure Verified Modules.
- Enterprise architecture with Bicep templates for Hub/Project - Managed Identity and RBAC configuration (Azure AI User role) - OpenTelemetry tracing integration with Application Insights - Agent proxy pattern for hybrid approach (existing agents + AI Foundry) - Feature flags for gradual rollout (USE_AZURE_FOUNDRY) - Copilot Studio integration guide - Production deployment with Container Apps and CI/CD - Cost estimation and troubleshooting guides Based on official Microsoft docs and Azure Verified Modules.
…#3) * docs: add Phase 1 Azure AI Foundry enterprise integration plan - Enterprise architecture with Bicep templates for Hub/Project - Managed Identity and RBAC configuration (Azure AI User role) - OpenTelemetry tracing integration with Application Insights - Agent proxy pattern for hybrid approach (existing agents + AI Foundry) - Feature flags for gradual rollout (USE_AZURE_FOUNDRY) - Copilot Studio integration guide - Production deployment with Container Apps and CI/CD - Cost estimation and troubleshooting guides Based on official Microsoft docs and Azure Verified Modules. * feat: Phase 1 Azure AI Foundry agent wrappers implementation Implemented wrapper classes for all 3 agent types: IT Agent Wrappers: - ITAgentWrapper, ITHelpdeskWrapper, ServiceNowWrapper, HITLSupportWrapper - Support for helpdesk, servicenow, and hitl subtypes - Session-based chat with memory Enterprise Agent Wrappers: - EnterpriseAgentWrapper with analyze() method - ResearchAgentWrapper, ContentAgentWrapper, DataAnalystWrapper - DocumentAgentWrapper, CodeAssistantWrapper, RAGAgentWrapper - DocumentIntelligenceWrapper for Azure Document Intelligence DeepAgent Wrappers (Multi-Agent Systems): - DeepAgentWrapper with multi-agent orchestration - ITOperationsWrapper, SalesIntelligenceWrapper, RecruitmentWrapper - SubAgentConfig for defining sub-agents - execute_workflow() for complex multi-step tasks FastAPI Server: - Chat UI at /chat and /chatui endpoints - REST endpoints: /api/it, /api/enterprise, /api/deepagent - Agent registry for managing all wrapped agents - Health check and agent listing endpoints Configuration Module: - AppConfig with AzureOpenAIConfig, AzureAIFoundryConfig - ObservabilityConfig for App Insights and LangSmith - AuthConfig with managed identity support - Environment-based configuration Also includes: - Example usage file at libs/azure-ai/examples/ - .env.example with all configuration options - Updated package __init__.py with documentation Addresses Phase 1 of enterprise integration plan. * feat: Enhanced UI with full agent support and streaming - Load all 13 agents at startup (3 IT, 7 Enterprise, 3 DeepAgents) - Added professional chat UI matching langchain-agents design - Implemented SSE streaming endpoints for Deep Agents - Added Deep Agent side panels (Task Progress, Context Files, Subagents) - Added collapsible thinking blocks (Claude Code-style) - Added tool activity visualization with animations - Added file upload support for Enterprise and Deep Agents - Added session management endpoints - Added Quick Actions for each agent type - All agents tested and verified working via Playwright Tested agents: ✅ IT: Helpdesk, ServiceNow, HITL Support ✅ Enterprise: Research, Content, Data Analyst, Document, Code Assistant, RAG, Document Intelligence ✅ DeepAgents: IT Operations, Sales Intelligence, Recruitment UI Features: - Agent categorization by type with color coding - Connection status indicator - Message timestamps - Thinking process visibility - Real-time SSE event streaming - File upload with type validation - Subagent activity monitoring * Update libs/azure-ai/langchain_azure_ai/wrappers/base.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update libs/azure-ai/langchain_azure_ai/wrappers/it_agents.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update libs/azure-ai/langchain_azure_ai/server/__init__.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
## Azure Monitor OpenTelemetry Integration - New observability module: langchain_azure_ai/observability/ - TelemetryConfig for Azure Monitor configuration - AgentTelemetry class with track_execution() context manager - ExecutionMetrics for tracking duration, tokens, success - setup_azure_monitor() function for initialization - trace_agent decorator for automatic method tracing - Middleware: RequestLoggingMiddleware, TracingMiddleware, MetricsMiddleware - Metrics: duration histogram, token counter, request counter, error counter - Graceful fallback when APPLICATIONINSIGHTS_CONNECTION_STRING not set ## Comprehensive Unit Tests - tests/unit_tests/test_wrappers.py with tests for: - WrapperConfig and FoundryAgentWrapper - IT, Enterprise, and DeepAgent wrappers - Observability module and middleware - Server endpoints ## Integration Tests - tests/integration_tests/test_agents.py with: - Azure OpenAI connection tests - Wrapper integration tests - Multi-turn conversation tests - Streaming response tests - Server endpoint integration tests - Azure AI Foundry specific tests - tests/conftest.py with shared fixtures ## API Documentation - Enhanced FastAPI with OpenAPI tags and descriptions - Swagger UI at /docs, ReDoc at /redoc - Comprehensive endpoint documentation ## Documentation Updates - Knowledge.md: Added Phase 2 observability details - Agents.md: Added observability and testing sections - README.md: Added v2.0.0 changelog entry
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Initial plan * Fix trace_agent decorator example to match actual signature Co-authored-by: abhilashjaiswal0110 <114993276+abhilashjaiswal0110@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: abhilashjaiswal0110 <114993276+abhilashjaiswal0110@users.noreply.github.com>
* Initial plan * Remove misleading log_response_body parameter from RequestLoggingMiddleware Co-authored-by: abhilashjaiswal0110 <114993276+abhilashjaiswal0110@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: abhilashjaiswal0110 <114993276+abhilashjaiswal0110@users.noreply.github.com>
#8) * Initial plan * Replace unsafe eval() with safe AST-based expression evaluator Co-authored-by: abhilashjaiswal0110 <114993276+abhilashjaiswal0110@users.noreply.github.com> * Add type validation for constants and remove deprecated ast.Num Co-authored-by: abhilashjaiswal0110 <114993276+abhilashjaiswal0110@users.noreply.github.com> * Add bounds checking for power operator to prevent DoS Co-authored-by: abhilashjaiswal0110 <114993276+abhilashjaiswal0110@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: abhilashjaiswal0110 <114993276+abhilashjaiswal0110@users.noreply.github.com>
* Initial plan * Fix unreachable statements in test_agent_telemetry_error_tracking Co-authored-by: abhilashjaiswal0110 <114993276+abhilashjaiswal0110@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: abhilashjaiswal0110 <114993276+abhilashjaiswal0110@users.noreply.github.com>
Collaborator
|
Abhilash Jaiswal (@abhilashjaiswal0110) are you still working on this PR? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.