docs: restructure observability documentation for clarity and neutrality

[vishu-bh]

Summary

Restructures the observability documentation to improve clarity, eliminate duplication, and use neutral language that describes capabilities rather than prescribing choices.

Changes

New Files

• **** - Comprehensive Prometheus metrics guide (extracted from landing page)
• **** - Span attributes reference (moved from repository root)

Renamed Files

• → (clearer naming)
• → (shorter, clearer)

Updated Files

• **** - Rewritten landing page with neutral language and comparison matrix
• **** - Updated navigation structure
• Cross-references - Fixed all broken links in langfuse.md, phoenix.md, and architecture docs

Key Improvements

1. Eliminated Duplication - Prometheus content extracted to dedicated file
2. Neutral Language - Removed prescriptive terms like "Perfect for", "Ideal for", "Essential for"
3. Factual Descriptions - Changed from "Use When" to "Characteristics" and "Common Use Cases"
4. Better Organization - Clear separation between user guides (manage/) and technical docs (architecture/)
5. Complete Cross-References - All documentation properly linked

Documentation Structure

docs/docs/
├── manage/
│   ├── observability.md (landing page)
│   └── observability/
│       ├── opentelemetry.md (external backends)
│       ├── internal.md (built-in tracing)
│       ├── prometheus.md (NEW - metrics)
│       ├── langfuse.md (LLM observability)
│       └── phoenix.md (AI/LLM observability)
└── architecture/
    ├── observability-otel.md (OTEL architecture)
    └── otel-span-attributes.md (NEW - span reference)

Testing

• Documentation builds successfully with make: Nothing to be done for `build'.
• All cross-references verified
• Navigation structure updated
• No broken links

Closes #4245

[ja8zyjits]

Hi @vishu-bh
Thanks for a comprehensive documentation for Obeservability.
Cursory reading reveals this is really good.
Please fix the PR description.

Regards,
Jitesh