INFO: Collecting Logs on Bottlerocket AMI

[PettitWesley]

Bottlerocket log collection

No host logs on Bottlerocket

UPDATE: please see new workaround from community: #136 (comment)

The Bottlerocket AMIs are meant to be a very stripped down container OS. Consequently, there are fewer log types to collect.

I've reached out the bottlerocket team and they said that all logs should go to journald on Bottlerocket.

On Bottlerocket, the /aws/containerinsights/Cluster_Name/host will not be populated because /var/log/dmesg, /var/log/secure, and /var/log/messages files do not exist on bottlerocket.

dmesg logs can be obtained with journalctl -k or journalctl --dmesg:

bash-5.1# journalctl -k
May 19 19:21:55 localhost kernel: Linux version 5.15.108 (builder@buildkitsandbox) (x86_64-bottlerocket-linux-gnu-gcc (Buildroot 2022.11.1) 11.3.0, GNU ld (GNU Binutils) 2.38) #1 SMP Tue May 9 23:54:36 UTC 2023
May 19 19:21:55 localhost kernel: Command line: BOOT_IMAGE=(hd0,gpt3)/vmlinuz console=tty0 console=ttyS0,115200n8 net.ifnames=0 netdog.default-interface=eth0:dhcp4,dhcp6? quiet bootconfig root=/dev/dm-0 rootwait ro raid=noautodetect random.trust_cpu=on selinux=1 enforcing=1 "dm-mod.create=root,,,ro,0 1884160 verity 1 PARTUUID=9b48037e-3bab-4072-8daf-082d8fae9f5e/PARTNROFF=1 PARTUUID=9b48037e-3bab-4072-8daf-082d8fae9f5e/PARTNROFF=2 4096 4096 235520 1 sha256 f12f74a243a23635effccde072228bc55a9f06c4c6e587001619bff01f6f8a16 80248401eae3104a1d94e19d04da0bc2b90289d599845063bdf6b330c422e713 2 restart_on_corruption ignore_zero_blocks" -- systemd.log_target=journal-or-kmsg systemd.log_color=0 systemd.show_status=true
May 19 19:21:55 localhost kernel: KASLR enabled
May 19 19:21:55 localhost kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
May 19 19:21:55 localhost kernel: x86/fpu: Supporting XSAVE

However, Fluent Bit appears to only be able to collect systemd unit journald logs. The docs say that it can filter by systemd unit file only: https://docs.fluentbit.io/manual/pipeline/inputs/systemd

And when I tried an input with no filters, no logs were collected:

[INPUT]
        Name                systemd
        Tag                 systemd.all.*
        DB                  /var/fluent-bit/state/systemd.db
        Read_From_Tail      Off

Here's the contents of /var/log on my node:

[ec2-user@admin]$ sudo sheltie
bash-5.1# ls /var/log
aws-routed-eni    containers  dmesg  journal  kdump  pods  private  support

The logs in aws-routed-eni might be interesting to some users and could be collected with a Tail input: https://docs.fluentbit.io/manual/pipeline/inputs/tail

bash-5.1# cd aws-routed-eni/
bash-5.1# ls
egress-v4-plugin.log  ipamd.log  plugin.log

Notes:

• There is a dmesg directory on my node in /var/log, but its empty. I've reached out the bottlerocket team and they said that all logs should go to journald on Bottlerocket. As noted above, you can use journalctl to obtain them.
• Make sure you use the admin container if you choose to poke around the filesystem. I found that it seemed some directories were not visible without admin.

Pod Logs will still be collected

Pod logs can still be collected without any change in experience.

Dataplane logs will still be collected

Kubelet and Containerd logs can still be collected without any change in experience.

[INPUT]
        Name                systemd
        Tag                 dataplane.systemd.*
        Systemd_Filter      _SYSTEMD_UNIT=kubelet.service
        Systemd_Filter      _SYSTEMD_UNIT=containerd.service
        DB                  /var/fluent-bit/state/systemd.db
        Path                /var/log/journal
        Read_From_Tail      ${READ_FROM_TAIL}

[arianvp]

FYI Systemd_Filter can take any journald filter! not just _SYSTEMD_UNIT

If you look at the manpage of journalctl you'll see:

       -k, --dmesg
           Show only kernel messages. This implies -b and adds the match
           "_TRANSPORT=kernel".

           Added in version 205.

AKA you can use the following to get dmesg logs:

[INPUT]
        Name                systemd
        Tag                 host.dmesg.*
        Systemd_Filter      _TRANSPORT=kernel

I hope this helps solve your problem

[joebowbeer]

Thanks for the info!

The docs need to include this:

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html

https://docs.aws.amazon.com/prescriptive-guidance/latest/implementing-logging-monitoring-cloudwatch/kubernetes-eks-logging.html

[mkrisko-pdi]

We are facing a small issue on the EC2 nodes which are using Bottlerocket AMI.

Can you also give a peace of advice or alternative solution how to achieve the result while using the Bottlerocket AMI? As the Support sent me over here :)

We do have couple applications running in the EKS cluster which are statefulset types and are using csi provider to create EBS volumes.

On those volumes we have also /data/logs folder and we are storing some more detailed logs there, which we split into multiple files based on cateogry. We need to retain this for security & audit purposes.

Usually on non Bottlerocket OS this would be reachable on path like:
var/lib/kubelet/pods//volumes/*csi/....

This path doesn't exist in Bottlerocket directly but if i go
sudo sheltie
that path is there.

As this our implementation are causing as to be AZ locked because of the EBS volume and i would like to get rid of this approach which was set up years ago i came to a conclusion:

Does bottleRocket support a way how to externalize the access to this?

1. Should we rather switch to EFS volume so that way we would not be AZ locked and still could access it?
2. Use a side-car container to get the logs?
3. Set the application to send the logs including the splitting file logs to 3rd party vendor?

Doing option 3 i want to avoid as what if the vendor change and it would require a code level change which the developers may not like.
Doing option 2 is managable on small number of application's, but if we would reach hundreds this would be un-maintanable.

[mkrisko-pdi]

To add one more piece of information:

sudo sheltie
/var/lib/kubelet/pods/bc7dd655-aee3-4942-993e-bc3f53b6256b/volumes/kubernetes.io~csi/pvc-4553ea4f-92a6-4729-8ea4-685267ca85e8/mount/
-> Shows all the files i want to see.
Outside of selfie the .bottlerocket is permission denied unless i am not doing command under sudo

/.bottlerocket/rootfs/var/lib/kubelet/pods/bc7dd655-aee3-4942-993e-bc3f53b6256b/volumes/kubernetes.io~csi/pvc-4553ea4f-92a6-4729-8ea4-685267ca85e8/mount/

Tihs is empty

But on non bottlerocketOS i would see the same stuff as inside of sheltie and would be able to access what i need with fluenbit easily