Skip to content

KB-4-Logging And Monitoring.md

Latest commit

 

History

History
150 lines (111 loc) · 6.65 KB

KB-4-Logging And Monitoring.md

File metadata and controls

150 lines (111 loc) · 6.65 KB
Raw

Logging

most linux distros currently host 2 systems that take care of logging
- systemd-journald - the systemd integrated logger service
- rsyslog - the legacy linux logging system
some services take care of their own

systemd-journald overview

  • systemd-journald is part od systemd and such, captures all messages generated by systemd's units
  • systemctl show <unit> - check messages that were generated by a specific unit
  • journalctl - check the journal which contains all messages
  • the systemd journal is not persistent by default
  • to make the journal persistent, create the following folder /var/log/journal

rsyslog overview

  • rsyslog is compatable with the legacy syslogd service
  • this service uses faciliy: severity destination to approproate location
  • typically, messages are written to files in /var/log, but output modules can also be used

logrotate

  • Logrotate is a mechanism that ensures that log files don't grow to big
  • Logrotate is started as a cronjob, and works with the /etc/logrotate.conf and /etc/logrotate.d/* configuration files

common journald options

journalctl -f - watch mode while new messages are added
journalctl -p err - print events with error priority or higher
journalctl -since "yyyy-MM-dd hh:mm:ss" -until ".." - datetime filter
journalctl -o verbose - verbose output of the logs
journalctl -u <name>.service - prints events for a specific unit
man systemd.journal-fields - for more information

managing journal persistency

  • by default journal is not persistant
  • when in /etc/systemd/journal.conf the option Storage=auto is set, a persistent journal will be created after creating a dir /var/log/journal
  • might need to use systemctl daemon-reload to trigger systemd to reload its configuration after the mentioned setup

logger command

  • logger - used to write messages to systemd-journald and rsyslog
  • very useful command is scripts, where information needs to be written to the logs from the script
  • logger -p <priority> to send logs with a specific priority

Managing and Monitoring Performance

ps - report a snapshot of the current processes
top - display Linux processes
htop - like top but better

top command overview

CPU

%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
  • us - user space
  • sy - system space
  • ni - niced processes (procecess with adjusted priority)
  • id - idle loop (the higher the idle the bettwe)
  • wa - waitiing for I/O (the higher the worst), idicating slow disc throughput
  • hi - hardware interputs - indicated wheter processes are handling SIGs
  • si - software interupts - indicated wheter processes are handling SIGs
  • st - stolen time, common with VMs
PID USER      PR  NI    VIRT    RES    SHR  S
  • PID - Process ID
  • USER - User the process belongs to
  • PR/ NI - Pririty values
  • VIRT - virtual memory - memory calimed my the process
  • RES - resident memory - what the proc is actually using
  • SHR - shared memory with other processes
  • S - status (i.e. S - Sleeping, R - Running )

Memory terminology

Cache - temporary storage used for frequently accessed data for rapid access
Buffer - temporary storage in memory for data while it is moved
Annonyous - memory that is allocated by applications
Swap - emulated RAM on disk which can be used for annonymous memory
Virtual - address space that is determined by the CPU architecture and hass nothing to do with swap
Active - memory that has been used recently
Inactive - memory that had not been used recently
Inactive file - memory that is eligible to be dropped in case of memory shortage
Inactive Annonyous - memory that is eligible to be moved to swap in case of memory storage

monitoring memory

free -m - good start and generic memory usage overview
vmstat - gives more details on memory usage
cat /proc/meminfo - read the information the kernel stores about memory

Cgroups

  • Cgroups are used for resource allocation
  • because of Cgroups, processes can be limited for the maximum amount of resources they can use
  • manged via systemd
  • slices are parts of the OS used for resource allocation
    • system is for all privileged processes
    • machine is for virual machines and containers
    • user is for non-privileged processes
  • within a slice, resources are further divided
  • relative resource weight counts within a slice

resources allocation with Cgroups

  • CPUWeight defines the relative weight of a process regarding CPU load as compared to other processes
    • if p1 has a CPUWeight of 1024 and p2 has 512, then p1 gets twice the amount of CPU cycles
    • CPUWeight is determined between peer processes running in the same slice
  • MemoryMin and MemoryMax se limits to the amount of memory available for processes
    • the process will get an OOM (Out Of Memory) error when trying to allocate more memory

managing process priority and Cgroups

  • if no Cgroups resource allocations have been used, all processes are equal and nice and renice can be used to change process priority
  • when Cgroups are used, nice valies apply to processes with the same relative CPU weight
  • best practice: use Cgroups to define resource use in general. where nice can be used for some final tuning of user processes

kernel tuning

  • the kernel has many settings that can be tuned through the /proc/sys interface
  • man proc for detailed information
  • to change a setting, the new setting can be echo'd into the /proc file
    • echo 30 > /proc/sys/vm/sawppiness
  • to make pernament changes use sysctl
    • sysctl -a - shows all settings available for tuning
  • the parameters in the output are filenames relative to /proc/sys where the / as a seperator between directories and files has been replaces with a dot (.)
  • to make settings persistent, write them to /etc/sysctl.conf or to a drop-in file in /etc/sysctl.d/*.conf
    • echo "vm.swappiness = 60" > /etc/sysctl.d/swappiness.conf
  • actovate changes by rebooting or re-reding the config file
    • sysctl -p /etc/sysctl.d/swappiness.conf

troubleshooting

/proc/cpuinfo - detailed information about CPUs
uptime - show current uptime and load stats
sar - used to gather long time usage statistics
sysctl - interfaces parameters in /proc to tune different CPU related settings