Create multiple alerts for one alertsconfig has error (first time)

[lilwan]

Is this a BUG REPORT or FEATURE REQUEST?:
bug

What happened:
I tried to create two alerts for one alertsconfig

apiVersion: alertmanager.keikoproj.io/v1alpha1
kind: AlertsConfig
metadata:
  name: cluster-test-1
spec:
  # Add fields here
    globalGVK:
      group: alertmanager.keikoproj.io
      version: v1alpha1
      kind: WavefrontAlert
    alerts:
      - alertName: wavefrontalert-pod-restart-sample2
        params:
          env: preprod
          count: "10"
          severity: warn
      - alertName: wavefrontalert-pod-restart-sample3
        params:
          env: preprod
          count: "20"
          severity: warn

wavefrontalerts are installed

MTVL16092f2af:Downloads lwan3$ k get wavefrontalerts
NAME                                 AGE
wavefrontalert-pod-restart-sample2   132m
wavefrontalert-pod-restart-sample3   132m

The first alert was created successfully but the second was failed with the following error

2021-08-20T20:50:59.598Z	ERROR	controllers.alertsconfig_controller.PatchIndividualAlertsConfigError	error occured in alerts config for alert name	{"request_id": "e8708100-6f3e-4f5e-a324-a0eff6020973", "alertsConfig_cr": "cluster-test-1", "namespace": "alert-manager-system", "alertName": "wavefrontalert-pod-restart-sample3", "error": "server returned 400 Bad Request\n{\"status\":{\"result\":\"ERROR\",\"message\":\"Alert query has taken too long to execute. Please refine your query for faster execution time, for example by including fewer series.\",\"code\":400}}\n"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
github.com/keikoproj/alert-manager/controllers.(*AlertsConfigReconciler).PatchIndividualAlertsConfigError
	/workspace/controllers/alertsconfig_controller.go:239
github.com/keikoproj/alert-manager/controllers.(*AlertsConfigReconciler).Reconcile
	/workspace/controllers/alertsconfig_controller.go:177
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.2/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.2/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.2/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
	/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
	/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:99
I0820 20:50:59.598445       1 event.go:282] Event(v1.ObjectReference{Kind:"AlertsConfig", Namespace:"alert-manager-system", Name:"cluster-test-1", UID:"d0a9308d-5546-4ac0-9ba4-0b2989a55e48", APIVersion:"alertmanager.keikoproj.io/v1alpha1", ResourceVersion:"8723375", FieldPath:""}): type: 'Warning' reason: 'server returned 400 Bad Request
{"status":{"result":"ERROR","message":"Alert query has taken too long to execute. Please refine your query for faster execution time, for example by including fewer series.","code":400}}
' unable to create the alert

Alertconfig status

status:
  alertStatus:
    wavefrontalert-pod-restart-sample2:
      alertName: pod-restart-2
      associatedAlert:
        CR: wavefrontalert-pod-restart-sample2
      id: "1629492595288"
      lastChangeChecksum: 9b6bf561cdaee1ae709a076250e51285
      link: https://intuit.wavefront.com/alerts/1629492595288
      state: Ready
    wavefrontalert-pod-restart-sample3:
      alertName: ""
      associatedAlert: {}
      id: ""
      state: Error
  alertsCount: 0
  retryCount: 1
  state: Error

Retry worked

status:
  alertStatus:
    wavefrontalert-pod-restart-sample2:
      alertName: pod-restart-2
      associatedAlert:
        CR: wavefrontalert-pod-restart-sample2
      id: "1629492595288"
      lastChangeChecksum: 9b6bf561cdaee1ae709a076250e51285
      link: https://intuit.wavefront.com/alerts/1629492595288
      state: Ready
    wavefrontalert-pod-restart-sample3:
      alertName: pod-restart-3
      associatedAlert:
        CR: wavefrontalert-pod-restart-sample3
      id: "1629492786383"
      lastChangeChecksum: ecf7c719852a172da42509e448fee7b7
      link: https://intuit.wavefront.com/alerts/1629492786383
      state: Ready
  alertsCount: 0
  retryCount: 0
  state: Ready

What you expected to happen:
The alerts should be created successfully for the first time

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• alert-manager version
• Kubernetes version :

$ kubectl version -o yaml

Other debugging information (if applicable):

- controller logs:

$ kubectl logs