[Failing Test]: TestAPMConfig – no such index [traces-apm-default]

[ycombinator]

Failing test case

TestAPMConfig

Error message

apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}     

Build

https://buildkite.com/elastic/elastic-agent-extended-testing/builds/4075#0192dd03-00ea-41c0-a56a-a5edb99318eb
https://buildkite.com/elastic/elastic-agent-extended-testing/builds/4077#0192dd14-e4c3-4901-b0d9-f50065d445b5
https://buildkite.com/elastic/elastic-agent-extended-testing/builds/4086#0192dda7-21c4-4d6a-9ce7-2c9a9ac4a3e3

OS

Linux, Windows

Stacktrace and notes

=== RUN   TestAPMConfig
    fetcher.go:93: Using existing artifact elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz
    fixture.go:288: Extracting artifact elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestAPMConfig4001669197
    fixture.go:306: Completed extraction of artifact elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestAPMConfig4001669197
    fixture.go:963: All component specifications where removed
    fixture.go:1004: Placed component specifications: fake
    apm_propagation_test.go:72: Rendered agent config:
        
        outputs:
          default:
            type: fake-output
        inputs:
          - id: fake-apm
            type: fake-apm
        agent.monitoring:
          traces: true
          apm:
            hosts:
              - https://9c26ecf86b6347b1b8363bce0edebab6.apm.us-west2.gcp.elastic-cloud.com:443
            environment: nsi3gya7ccprbvh2xzveb5ytfy0beb0csqxcuftfqu
            secret_token: AeKIQKeIi5tKM2Wt9s
            global_labels:
              test_name: TestAPMConfig
              test_type: Agent integration test
            tls:
              skip_verify: true
    fixture.go:1234: 2024-10-30T14:19:57.114200275Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent.sock: connect: no such file or directory" retrying: 77.874125ms
    fixture.go:1234: 2024-10-30T14:19:57.192670596Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent.sock: connect: no such file or directory" retrying: 134.287776ms
    log.go:85: Flag --disable-encrypted-store has been deprecated, agent on Mac OS can be run/installed without root privileges, see elastic-agent install --help
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:57.323Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":195},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"process.pid":21903,"agent.version":"9.0.0","agent.unprivileged":true,"ecs.version":"1.6.0"}
    fixture.go:1234: 2024-10-30T14:19:57.327931667Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent.sock: connect: no such file or directory" retrying: 136.304281ms
    fixture.go:1234: 2024-10-30T14:19:57.465385895Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent.sock: connect: no such file or directory" retrying: 278.736473ms
    fixture.go:1234: 2024-10-30T14:19:57.745135101Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent.sock: connect: no such file or directory" retrying: 526.499465ms
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.026Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.InvokeWatcher","file.name":"upgrade/rollback.go","file.line":149},"message":"agent is not upgradable, not starting watcher","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.029Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":284},"message":"APM instrumentation disabled","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.030Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":62},"message":"Gathered system information","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.031Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":68},"message":"Detected available inputs and outputs","log":{"source":"elastic-agent"},"inputs":["fake","fake-apm","fake-isolated-units"],"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.032Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/capabilities.LoadFile","file.name":"capabilities/capabilities.go","file.line":48},"message":"Capabilities file not found in /tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/capabilities.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.032Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":74},"message":"Determined allowed capabilities","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    fixture.go:1234: 2024-10-30T14:19:58.272176018Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent.sock: connect: no such file or directory" retrying: 1.087157926s
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.478Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.NewManager","file.name":"runtime/manager.go","file.line":175},"message":"GRPC comms socket listening at localhost:6789","log":{"source":"elastic-agent"},"address":"localhost:6789","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.478Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":130},"message":"Elastic Agent has been started in testing mode and is managed through the control protocol","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.480Z","log.logger":"control","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/control/v2/server.(*Server).Start","file.name":"server/server.go","file.line":88},"message":"GRPC control socket listening at unix:///tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent.sock","log":{"source":"elastic-agent"},"address":"unix:///tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent.sock","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.481Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).Run","file.name":"runtime/manager.go","file.line":238},"message":"Starting grpc control protocol listener on port 6789 with max_message_size 104857600","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.482Z","log.logger":"composable.providers.docker","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable/providers/docker.(*dynamicProvider).Run","file.name":"docker/docker.go","file.line":44},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    fixture.go:1234: 2024-10-30T14:19:59.359729146Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent.sock: connect: no such file or directory" retrying: 1.588421624s
    fixture.go:1242: 2024-10-30T14:20:00.948864551Z: StateWatch started
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:00.951Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.(*testingModeConfigManager).SetConfig","file.name":"application/testing_mode.go","file.line":53},"message":"Testing mode received new configuration from protocol, passing to coordinator","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:00.953Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*Upgrader).Reload","file.name":"upgrade/upgrade.go","file.line":123},"message":"Source URI changed from \"https://artifacts.elastic.co/downloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:00.954Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Start","file.name":"reload/reload.go","file.line":54},"message":"Starting monitoring server with cfg &config.MonitoringConfig{Enabled:true, MonitorLogs:true, MonitorMetrics:true, MetricsPeriod:\"\", LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc0022b4360), Namespace:\"default\", Pprof:(*config.PprofConfig)(nil), MonitorTraces:true, APM:config.APMConfig{Environment:\"nsi3gya7ccprbvh2xzveb5ytfy0beb0csqxcuftfqu\", APIKey:\"\", SecretToken:\"AeKIQKeIi5tKM2Wt9s\", Hosts:[]string{\"https://9c26ecf86b6347b1b8363bce0edebab6.apm.us-west2.gcp.elastic-cloud.com:443\"}, GlobalLabels:map[string]string{\"test_name\":\"TestAPMConfig\", \"test_type\":\"Agent integration test\"}, TLS:config.APMTLS{SkipVerify:true, ServerCertificate:\"\", ServerCA:\"\"}, SamplingRate:(*float32)(nil)}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.Limit{Interval:60000000000, Burst:1}}}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:00.954Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":96},"message":"creating monitoring API with cfg api.Config{Enabled:true, Host:\"http://localhost:6791\", Port:6791, User:\"\", SecurityDescriptor:\"\", Timeout:5000000000}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:00.954Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start","file.name":"api/server.go","file.line":85},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:00.955Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":87},"message":"Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:00.971Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1270},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.227Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":625},"message":"Spawned new component fake-apm-default: Starting: spawned pid '21913'","log":{"source":"elastic-agent"},"component":{"id":"fake-apm-default","state":"STARTING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.227Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":632},"message":"Spawned new unit fake-apm-default-fake-apm: Starting: spawned pid '21913'","log":{"source":"elastic-agent"},"component":{"id":"fake-apm-default","state":"STARTING"},"unit":{"id":"fake-apm-default-fake-apm","type":"input","state":"STARTING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.227Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":632},"message":"Spawned new unit fake-apm-default: Starting: spawned pid '21913'","log":{"source":"elastic-agent"},"component":{"id":"fake-apm-default","state":"STARTING"},"unit":{"id":"fake-apm-default","type":"output","state":"STARTING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.248Z","log.logger":"component.runtime.fake-apm-default","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).CheckinV2","file.name":"runtime/manager.go","file.line":704},"message":"control checkin v2 protocol has chunking enabled","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.248Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":645},"message":"Component state changed fake-apm-default (STARTING->HEALTHY): Healthy: communicating with pid '21913'","log":{"source":"elastic-agent"},"component":{"id":"fake-apm-default","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.251Z","message":"unit change received","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"info","state":"STARTING","expectedState":"HEALTHY","time":"2024-10-30T14:20:01Z","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.257Z","message":"Starting fake APM traces sender with config elastic:{tls:{skip_verify:true}  environment:\"nsi3gya7ccprbvh2xzveb5ytfy0beb0csqxcuftfqu\"  secret_token:\"AeKIQKeIi5tKM2Wt9s\"  hosts:\"https://9c26ecf86b6347b1b8363bce0edebab6.apm.us-west2.gcp.elastic-cloud.com:443\"  global_labels:\"test_name=TestAPMConfig,test_type=Agent integration test\"}","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"info","time":"2024-10-30T14:20:01Z","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.257Z","message":"unit change received","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"info","state":"STARTING","expectedState":"HEALTHY","time":"2024-10-30T14:20:01Z","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.258Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed fake-apm-default-fake-apm (STARTING->HEALTHY): Fake APM traces sender has started","log":{"source":"elastic-agent"},"component":{"id":"fake-apm-default","state":"HEALTHY"},"unit":{"id":"fake-apm-default-fake-apm","type":"input","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.258Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed fake-apm-default (STARTING->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"fake-apm-default","state":"HEALTHY"},"unit":{"id":"fake-apm-default","type":"output","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
    apm_propagation_test.go:78: agent state: &{Info:{ID:ed947dca-69ee-484f-ab24-ec1b22214be7 Version:9.0.0 Commit:e6f281aa4de18d5ba1aa10aa36da9796c3b22a47 BuildTime:2024-10-30 13:48:12 +0000 UTC Snapshot:true PID:21903 Unprivileged:true} State:HEALTHY Message:Running Components:[{ID:fake-apm-default Name:fake-apm State:HEALTHY Message:Healthy: communicating with pid '21913' Units:[{UnitID:fake-apm-default UnitType:OUTPUT State:HEALTHY Message:Healthy Payload:map[]} {UnitID:fake-apm-default-fake-apm UnitType:INPUT State:HEALTHY Message:Fake APM traces sender has started Payload:map[]}] VersionInfo:{Name:fake Meta:map[input:fake]}}] FleetState:STOPPED FleetMessage:Not enrolled into Fleet UpgradeDetails:}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.425Z","message":"config request failed: request failed with 503 Service Unavailable: {\"ok\":false,\"message\":\"The requested resource is currently unavailable.\"}","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"error","time":"2024-10-30T14:20:01Z","ecs.version":"1.6.0"}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:12.419Z","message":"request failed: request failed with 502 Bad Gateway: {\"ok\":false,\"message\":\"The instance rejected the connection.\"} (next request in ~0s)","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"debug","time":"2024-10-30T14:20:12Z","ecs.version":"1.6.0"}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:23.410Z","message":"request failed: request failed with 502 Bad Gateway: {\"ok\":false,\"message\":\"The instance rejected the connection.\"} (next request in ~1s)","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"debug","time":"2024-10-30T14:20:23Z","ecs.version":"1.6.0"}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:31.261Z","message":"gathering metrics","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"debug","time":"2024-10-30T14:20:31Z","ecs.version":"1.6.0"}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:34.410Z","message":"request failed: request failed with 502 Bad Gateway: {\"ok\":false,\"message\":\"The instance rejected the connection.\"} (next request in ~4s)","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"debug","time":"2024-10-30T14:20:34Z","ecs.version":"1.6.0"}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:45.410Z","message":"request failed: request failed with 502 Bad Gateway: {\"ok\":false,\"message\":\"The instance rejected the connection.\"} (next request in ~9s)","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"debug","time":"2024-10-30T14:20:45Z","ecs.version":"1.6.0"}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:56.410Z","message":"request failed: request failed with 502 Bad Gateway: {\"ok\":false,\"message\":\"The instance rejected the connection.\"} (next request in ~16s)","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"debug","time":"2024-10-30T14:20:56Z","ecs.version":"1.6.0"}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:81: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/apm_propagation_test.go:81
        	            				/home/ubuntu/agent/pkg/testing/machine.go:230
        	            				/home/ubuntu/agent/pkg/testing/fixture.go:608
        	            				/home/ubuntu/agent/pkg/testing/fixture.go:646
        	            				/home/ubuntu/agent/testing/integration/apm_propagation_test.go:110
        	Error:      	Condition never satisfied
        	Test:       	TestAPMConfig
    fixture.go:1282: Temporary directory "/tmp/TestAPMConfig4001669197" preserved for investigation/debugging
--- FAIL: TestAPMConfig (74.56s)

This test is failing on main from 053b320 onwards. Note that main is pointing to 9.0.0 now. It is not failing on the 8.x branch.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[pchila]

Most likely it's not a flaky test, rather APM server is unhealthy on the ESS deployment, as shown by the log from the tracer below

 log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:23.410Z","message":"request failed: request failed with 502 Bad Gateway: {\"ok\":false,\"message\":\"The instance rejected the connection.\"} (next request in ~1s)","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"debug","time":"2024-10-30T14:20:23Z","ecs.version":"1.6.0"}

[ycombinator]

Thanks, I agree. It's failed 3 builds in a row, so it's unlikely to be flaky. I'll try to spin up an ESS deployment on 9.0.0 and check the health of APM server.

[ycombinator]

Indeed, APM Server on 9.0.0 in ESS CFT is unhealthy right now:

@simitt Is this something the APM Server team is already aware of?

[kruskall]

Hey @ycombinator 👋

This might be related to the beats bump, let me quickly check

[kruskall]

According to the logs in cloud the error is

Error: error loading config file: stat apm-server.yml: no such file or directory

Downloading the docker image shows the apm-server config and binary owned by 1000:1000. Did anything change in the recent version ? What user is used to run apm-server ?

[ycombinator]

Did anything change in the recent version?

Something must've changed very recently because, as of a5de320 (created October 30 00:46 UTC), this test was passing.

[ycombinator]

According to the logs in cloud the error is

Error: error loading config file: stat apm-server.yml: no such file or directory

Downloading the docker image shows the apm-server config and binary owned by 1000:1000. Did anything change in the recent version ? What user is used to run apm-server ?

BTW, this looks mighty similar to #5615.

[1pkg]

Seems like it's neither file owner nor permissions this time, the output is identical from both
docker run elastic-agent-cloud:9.0.0-SNAPSHOT
and
docker run elastic-agent-cloud:8.16.0-SNAPSHOT

ls -la apm*
-rwxr-xr-x    1 elastic-agent elastic-agent  53936280 Oct 30 17:08 apm-server
-rw-r--r--    1 elastic-agent elastic-agent       542 Oct 30 17:08 apm-server.spec.yml
-rw-r--r--    1 elastic-agent elastic-agent     39322 Oct 30 17:08 apm-server.yml