Support Custom Annotations on Metrics Services #2343

Is your feature request related to a problem? Please describe.

The ArgoCD Operator provides no way to add custom annotations to metrics Services (argocd-server-metrics, argocd-metrics, etc.). This prevents service-based autodiscovery for monitoring tools like Datadog that rely on service annotations.

Current workaround requires manual patching:

oc annotate svc argocd-server-metrics \
  ad.datadoghq.com/service.check_names='["openmetrics"]' \
  ad.datadoghq.com/service.init_configs='[{}]' \
  ad.datadoghq.com/service.instances='[{"openmetrics_endpoint":"http://%%host%%:%%port%%/metrics","namespace":"argocd_server","metrics":[".*"]}]'

This is not GitOps-friendly, not persistent across deployments, and cannot be managed declaratively.

Describe the solution you'd like

Add optional annotations field for metrics services in the ArgoCD CR:

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
  name: argocd
spec:
  server:
    metrics:
      enabled: true
      annotations:
        ad.datadoghq.com/service.check_names: '["openmetrics"]'
        ad.datadoghq.com/service.init_configs: '[{}]'
        ad.datadoghq.com/service.instances: '[{"openmetrics_endpoint":"http://%%host%%:%%port%%/metrics","namespace":"argocd_server","metrics":[".*"]}]'
  
  controller:
    metrics:
      enabled: true
      annotations:
        ad.datadoghq.com/service.check_names: '["openmetrics"]'
        ad.datadoghq.com/service.init_configs: '[{}]'
        ad.datadoghq.com/service.instances: '[{"openmetrics_endpoint":"http://%%host%%:%%port%%/metrics","namespace":"argocd_controller","metrics":[".*"]}]'

This would enable:

• Datadog OpenMetrics autodiscovery (primary use case)
• Annotation-based Prometheus scraping (alternative to ServiceMonitor)
• Service mesh integration for metrics endpoints

Describe alternatives you've considered

1. Manual annotation - Current workaround, not persistent or GitOps-friendly
2. Pod annotations - Available since v0.13.0 via spec.server.annotations, but Datadog service-based autodiscovery requires service annotations, not pod annotations
3. ServiceMonitor only - Works for Prometheus Operator but doesn't help with Datadog or other monitoring tools
4. CR metadata.annotations - Doesn't work; operator doesn't propagate CR annotations to child Services

Additional context

• Related: Issue Add service annotations #651 requested service annotations but was closed by PR feat: Edit annotations for the argocd-server service #2095, which was never merged and only targeted the main argocd-server service (port 443), not metrics services
• Tested: Manual service annotations confirmed working for Datadog OpenMetrics collection
• Environment: ArgoCD v3.4.5 (OpenShift GitOps 1.21.3), v1beta1 API, OpenShift 4.15

I'm willing to collaborate on a PR if this design is accepted.

Note: This issue was drafted with assistance from AI tools, reviewed for technical accuracy.