[beats receivers] switch monitoring runtime from process to otel deletes run directories

[leehinman]

If you switch the monitoring "_runtime_experimental" from "process" to "otel" the run directories are all deleted.

If you use inotifywatch you can observe this

inotifywait -mr /opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ MODIFY log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ MODIFY log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ MODIFY log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ CLOSE_WRITE,CLOSE log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ OPEN,ISDIR beat
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/ OPEN,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/ OPEN,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/metrics-monitoring/ OPEN,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/ ACCESS,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/metrics-monitoring/ ACCESS,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/ ACCESS,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/metrics-monitoring/ ACCESS,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/metrics-monitoring/ DELETE meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/ CLOSE_NOWRITE,CLOSE,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/metrics-monitoring/ CLOSE_NOWRITE,CLOSE,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/metrics-monitoring/ DELETE_SELF
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/ DELETE,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ CLOSE_NOWRITE,CLOSE,ISDIR beat
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/ CLOSE_NOWRITE,CLOSE,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ OPEN,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ OPEN,ISDIR filestream-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ OPEN,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ ACCESS,ISDIR filestream-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ ACCESS,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ ACCESS,ISDIR filestream-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ ACCESS,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ OPEN,ISDIR registry
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ OPEN,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ ACCESS,ISDIR registry
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ ACCESS,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ ACCESS,ISDIR registry
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ ACCESS,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ OPEN,ISDIR filebeat
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ OPEN,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ ACCESS,ISDIR filebeat
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ ACCESS,ISDIR filebeat
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ DELETE log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ DELETE meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ CLOSE_NOWRITE,CLOSE,ISDIR filebeat
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ CLOSE_NOWRITE,CLOSE,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ DELETE_SELF
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ DELETE,ISDIR filebeat
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ CLOSE_NOWRITE,CLOSE,ISDIR registry
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ CLOSE_NOWRITE,CLOSE,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ DELETE_SELF
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ DELETE,ISDIR registry
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ DELETE meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ CLOSE_NOWRITE,CLOSE,ISDIR filestream-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ CLOSE_NOWRITE,CLOSE,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ DELETE_SELF
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ DELETE,ISDIR filestream-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ CLOSE_NOWRITE,CLOSE,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ OPEN,ISDIR http
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/ OPEN,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/ OPEN,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ OPEN,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/ ACCESS,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ ACCESS,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/ ACCESS,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ ACCESS,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ DELETE meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/ CLOSE_NOWRITE,CLOSE,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ CLOSE_NOWRITE,CLOSE,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ DELETE_SELF
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/ DELETE,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ CLOSE_NOWRITE,CLOSE,ISDIR http
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/ CLOSE_NOWRITE,CLOSE,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/prometheus/metrics-monitoring/ OPEN meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/prometheus/metrics-monitoring/ ACCESS meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/prometheus/metrics-monitoring/ CLOSE_NOWRITE,CLOSE meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/ CREATE,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/ OPEN,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/ ACCESS,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/ CLOSE_NOWRITE,CLOSE,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ CREATE meta.json.new4122996731
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ OPEN meta.json.new4122996731
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ MODIFY meta.json.new4122996731
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ CLOSE_WRITE,CLOSE meta.json.new4122996731
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ MOVED_FROM meta.json.new4122996731
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ MOVED_TO meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/ OPEN,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ OPEN,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/ CLOSE_NOWRITE,CLOSE,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ CLOSE_NOWRITE,CLOSE,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/ CREATE,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/ OPEN,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/ ACCESS,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/ CLOSE_NOWRITE,CLOSE,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/metrics-monitoring/ MODIFY meta.json.new2975256364
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/metrics-monitoring/ CLOSE_WRITE,CLOSE meta.json.new2975256364
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/metrics-monitoring/ MOVED_FROM meta.json.new2975256364
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/metrics-monitoring/ MOVED_TO meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/ OPEN,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/metrics-monitoring/ OPEN,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/ CLOSE_NOWRITE,CLOSE,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/metrics-monitoring/ CLOSE_NOWRITE,CLOSE,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ CREATE,ISDIR filestream-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ OPEN,ISDIR filestream-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ ACCESS,ISDIR filestream-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ CLOSE_NOWRITE,CLOSE,ISDIR filestream-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ CREATE meta.json.new2874223691
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ OPEN meta.json.new2874223691
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ MODIFY meta.json.new2874223691
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ CLOSE_WRITE,CLOSE meta.json.new2874223691
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ MOVED_FROM meta.json.new2874223691
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ MOVED_TO meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ OPEN,ISDIR filestream-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ OPEN,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ CLOSE_NOWRITE,CLOSE,ISDIR filestream-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ CLOSE_NOWRITE,CLOSE,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ CREATE,ISDIR registry
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ OPEN,ISDIR registry
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ ACCESS,ISDIR registry
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ CLOSE_NOWRITE,CLOSE,ISDIR registry
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ CLOSE_WRITE,CLOSE meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ OPEN,ISDIR filebeat
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ OPEN,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ CLOSE_NOWRITE,CLOSE,ISDIR filebeat
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ CLOSE_NOWRITE,CLOSE,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ OPEN meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ CLOSE_NOWRITE,CLOSE meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ OPEN,ISDIR filebeat
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ OPEN,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ ACCESS,ISDIR filebeat
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ ACCESS,ISDIR filebeat
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ CLOSE_NOWRITE,CLOSE,ISDIR filebeat
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ CLOSE_NOWRITE,CLOSE,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ CREATE log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ OPEN log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ MODIFY log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ MODIFY log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ MODIFY log.json

If you go the other way "otel" -> "process" the correct behavior is observed where the run directories are preserved.

/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ MODIFY log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ MODIFY log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ CLOSE_WRITE,CLOSE log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ ATTRIB,ISDIR filestream-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ ATTRIB,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/ ATTRIB,ISDIR filestream-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ ATTRIB,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ OPEN meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ ACCESS meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/ CLOSE_NOWRITE,CLOSE meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ OPEN meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ CLOSE_NOWRITE,CLOSE meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ OPEN meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ CLOSE_WRITE,CLOSE meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ OPEN meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ CLOSE_NOWRITE,CLOSE meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ OPEN,ISDIR filebeat
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ OPEN,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ ACCESS,ISDIR filebeat
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ ACCESS,ISDIR filebeat
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/ CLOSE_NOWRITE,CLOSE,ISDIR filebeat
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ CLOSE_NOWRITE,CLOSE,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ OPEN log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ CLOSE_WRITE,CLOSE log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ OPEN log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ ACCESS log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ CLOSE_NOWRITE,CLOSE log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ OPEN log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/ ATTRIB,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/metrics-monitoring/ ATTRIB,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/ ATTRIB,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/metrics-monitoring/ ATTRIB,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/metrics-monitoring/ OPEN meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/metrics-monitoring/ ACCESS meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/beat/metrics-monitoring/ CLOSE_NOWRITE,CLOSE meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/ ATTRIB,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ ATTRIB,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/ ATTRIB,ISDIR metrics-monitoring
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ ATTRIB,ISDIR
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ OPEN meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ ACCESS meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/http/metrics-monitoring/ CLOSE_NOWRITE,CLOSE meta.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ MODIFY log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ MODIFY log.json
/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-3c3de2/run/filestream-monitoring/registry/filebeat/ MODIFY log.json

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

I think this happens because the command runtime deletes the run directory when it tears down the component, because ownership of the component is transferring to the otel runtime. We don't have a way to distinguish between the component being removed from the policy and the component being transferred to another runtime but staying in the policy.

elastic-agent/pkg/component/runtime/command.go

Lines 474 to 482 in c26dee6

	case actionStop, actionTeardown:
	// stopping (should have exited)
	if c.actionState == actionTeardown {
	// teardown so the entire component has been removed (cleanup work directory)
	_ = os.RemoveAll(c.workDirPath())
	}
	stopMsg := fmt.Sprintf("Stopped: pid '%d' exited with code '%d'", state.Pid(), state.ExitCode())
	c.forceCompState(client.UnitStateStopped, stopMsg)
	}

[cmacknz]

elastic-agent/pkg/component/runtime/manager.go

Lines 770 to 786 in c26dee6

	var stop []*componentRuntimeState
	m.currentMx.RLock()
	for id, existing := range m.current {
	// skip if already touched (meaning it still existing)
	if _, done := touched[id]; done {
	continue
	}
	// component was removed (time to clean it up)
	stop = append(stop, existing)
	}
	m.currentMx.RUnlock()
	var stoppedWg sync.WaitGroup
	stoppedWg.Add(len(stop))
	for _, existing := range stop {
	m.logger.Debugf("Stopping component %q", existing.id)
	_ = existing.stop(teardown, model.Signed)

That's where teardown gets triggered from.

We will know when a component is moved in the block below and can hopefully have a way to indicate not to teardown from here.

elastic-agent/internal/pkg/agent/application/coordinator/coordinator.go

Lines 1775 to 1791 in c26dee6

	// updateManagersWithConfig updates runtime managers with the component model and config.
	// Components may be sent to different runtimes depending on various criteria.
	func (c *Coordinator) updateManagersWithConfig(model *component.Model) {
	runtimeModel, otelModel := c.splitModelBetweenManagers(model)
	c.logger.With("components", runtimeModel.Components).Debug("Updating runtime manager model")
	c.runtimeMgr.Update(*runtimeModel)
	c.logger.With("components", otelModel.Components).Debug("Updating otel manager model")
	if len(otelModel.Components) > 0 {
	componentIDs := make([]string, 0, len(otelModel.Components))
	for _, comp := range otelModel.Components {
	componentIDs = append(componentIDs, comp.ID)
	}
	c.logger.With("component_ids", componentIDs).Info("Using OpenTelemetry collector runtime.")
	}
	c.otelMgr.Update(c.otelCfg, otelModel.Components)
	}

[swiatekm]

Yeah, we need to detect this situation in the coordinator and inform the runtime manager that it doesn't need to clean up. I guess one simple way would be to give the components which are being switched to both runtimes and have the runtime itself do the right thing.

Incidentally, the otel runtime doesn't do this teardown at all, so factoring it out and making it runtime-independent could be a good idea as well.

[cmacknz]

Yeah factoring teardown out into the coordination would be a reasonable way to do this and fix it in both places, instead of communicating the need to teardown and implementing it twice.

[cmacknz]

The tricky part of having teardown in the coordinator or outside the runtimes themselves, is you can't do it until the components are actually stopped, which will add in some additional synchronization overhead.

[swiatekm]

I don't think it's a problem for runtimes to be responsible for teardown in general, my point was more that the logic should live in its own package. Right now the logic for determining paths for a component is duplicated between the runtime manager and otel manager, for example.

[leehinman]

I know we need a solution for the general case of active components moving from one runtime to another.

But I do wonder if the monitoring component should ever get actionTeardown. If you disable monitoring and then re-enable it do we want to re-ingest all the log files? That is consistent with how other components work, but as a user it seems surprising.

[blakerouse]

The simplest thing to do here is just not remove the work directory in teardown. The code is here:

https://github.com/elastic/elastic-agent/blob/main/pkg/component/runtime/command.go#L478

With @leehinman comment about monitoring not expecting that turning it off and back on would cause re-ingest, we might not ever want to delete it. Allow integrations to be stopped and re-started in the policy without it always starting from the beginning.

[cmacknz]

Allow integrations to be stopped and re-started in the policy without it always starting from the beginning.

The trick with this is people do sometimes want to start from the beginning and we'd be removing one way to do that. We ideally probably want a separate "reset state" action or configuration we could respect if we did this. Having a way to reset filestream/log inputs in agent is a pretty common ask.

[leehinman]

Just for confirmation, switching to

		if c.actionState == actionTeardown && c.current.ID != "filestream-monitoring" {

changes the behavior so we don't delete the run directory and we don't re-ingest the log files. 0 duplicates when switching from process -> otel -> process.