salt project for installing percona chained, redis, memcached, mcrouter, elasticsearch, and kibana.

kernel tunning

https://medium.com/@pawilon/tuning-your-linux-kernel-and-haproxy-instance-for-high-loads-1a2105ea553e

• net.core.netdev_max_backlog if using 1G network ok to be default value of 1000 in ubuntu but if network port is 10G is better to increase it.

redis

• disabling daemonize and using supervised systemd

• huge pages doc https://www.pingcap.com/blog/transparent-huge-pages-why-we-disable-it-for-databases/ https://docs.oracle.com/en/database/oracle/oracle-database/12.2/ladbi/restrictions-for-hugepages-and-transparent-hugepages-configurations.html#GUID-D8178896-D00F-4F02-82A7-A44F89D8F103

• vm.overcommit https://access.redhat.com/discussions/4021161 https://redis.io/docs/getting-started/faq/#background-saving-fails-with-a-fork-error-on-linux

• max net.core.somaxconn and tcp_max_syn_backlog needed for increasing tcp_backlog size https://access.redhat.com/solutions/6439141 https://stackoverflow.com/questions/62641621/what-is-the-difference-between-tcp-max-syn-backlog-and-somaxconn

• unix sockets are faster but only allowing access from local connections, could be beneficial in environments like docker or kubernetes https://medium.com/@jonbaldie/how-to-connect-to-redis-with-unix-sockets-in-docker-9e94e01b7acd#:~:text=Connecting%20to%20Redis%20via%20Unix,only%20local%20connections%20are%20possible.

• It is possible to completely kill a command by renaming it into an empty string

• oom-score-adj https://www.kernel.org/doc/Documentation/filesystems/proc.txt

• for better latency and performance rdb has been disabled and aof is enabled

• response time increases a little after removing advanced configuration parts like encoding and hashing

percona

• replication with extrabackup https://docs.percona.com/percona-xtrabackup/2.4/howtos/setting_up_replication.html

• port installation https://docs.percona.com/percona-server/8.0/installation/post-installation.html

• gtid https://dev.mysql.com/doc/refman/8.0/en/replication-mode-change-online-concepts.html

• relay log https://dev.mysql.com/doc/refman/8.0/en/replica-logs-relaylog.html

• file /mysql_get_backup is for checking if a backup is needed. 0 is for the first time. 1 is backup in progress. 2 is not first time but backup needed.

• copy backup https://unix.stackexchange.com/questions/106480/how-to-copy-files-from-one-machine-to-another-using-ssh

• sftp vs scp https://www.cerberusftp.com/comparing-scp-vs-sftp-which-is-better/

• nfs vs sshfs https://blog.ja-ke.tech/2019/08/27/nas-performance-sshfs-nfs-smb.html

• rsync vs scp https://stackoverflow.com/questions/20244585/how-does-scp-differ-from-rsync

• rsync and scp options https://blog.knoldus.com/what-is-the-difference-between-rsync-and-scp/#:~:text=Conclusion%20is%2C%20rsync%20is%20good,or%20to%20the%20remote%20nodes.

• query performance with parallel execution https://www.percona.com/blog/2014/01/07/increasing-slow-query-performance-with-parallel-query-execution/

• namespace in mysql https://dev.mysql.com/doc/refman/8.0/en/network-namespace-support.html

• mysql options https://dev.mysql.com/doc/refman/8.0/en/server-options.html https://dev.mysql.com/doc/refman/8.0/en/connection-interfaces.html https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html

• logfiles https://dev.mysql.com/doc/refman/8.0/en/log-destinations.html

• threadpool sadly it's only available on enterprise edition https://dev.mysql.com/doc/refman/8.0/en/thread-pool.html

• benchmarking https://dev.mysql.com/doc/refman/8.0/en/optimize-benchmarking.html

• server thread information https://dev.mysql.com/doc/refman/8.0/en/thread-information.html

• hashed password https://groups.google.com/g/salt-users/c/pS18paNvwpo

• Log file ./ib_logfile1 is of different size 50331648 bytes than other log files 8388608 bytes! https://www.percona.com/blog/2016/05/31/what-is-a-big-innodb_log_file_size/ https://support.plesk.com/hc/en-us/articles/115001738733-How-to-change-the-innodb-log-file-size-value-in-MySQL-MariaDB#:~:text=Under%20the%20%5Bmysqld%5D%20section%2C,under%20the%20%5Bmysqld%5D%20section. https://support.plesk.com/hc/en-us/articles/213401729

• some replication errors https://www.percona.com/blog/2014/10/08/mysql-replication-got-fatal-error-1236-causes-and-cures/#:~:text=This%20error%20occurs%20when%20the,fails%20with%20the%20above%20error.

• mysql ports https://dev.mysql.com/doc/mysql-port-reference/en/mysql-ports-reference-tables.html

• mysqlx https://www.percona.com/blog/2019/01/07/understanding-mysql-x-all-flavors/

• sysbench https://www.percona.com/blog/2011/04/29/sysbench-with-support-of-multi-tables-workload/ https://techcommunity.microsoft.com/t5/azure-database-for-mysql-blog/benchmarking-azure-database-for-mysql-flexible-server-using/ba-p/3108799#:~:text=You%20can%20use%20Sysbench%20to,database%20under%20an%20intensive%20load.

• lowering cpu usage https://dba.stackexchange.com/questions/44896/mysql-using-too-much-cpu

• finding table is MYISAM or innodb SHOW TABLE STATUS

• optimize innodb disk i/o https://dev.mysql.com/doc/refman/8.0/en/optimizing-innodb-diskio.html https://dev.mysql.com/doc/refman/8.0/en/optimizing-innodb-configuration-variables.html For durability and consistency in a replication setup that uses InnoDB with transactions: If binary logging is enabled, set sync_binlog=1.Always set innodb_flush_log_at_trx_commit=1.

• optimization tips https://www.devart.com/dbforge/mysql/studio/mysql-performance-tips.html

• query cache https://dev.mysql.com/blog-archive/mysql-8-0-retiring-support-for-the-query-cache/

• fsync https://man7.org/linux/man-pages/man2/fsync.2.html

• best practice aws https://aws.amazon.com/blogs/database/best-practices-for-configuring-parameters-for-amazon-rds-for-mysql-part-2-parameters-related-to-replication/#:~:text=The%20sync_binlog%20option%20controls%20how,value%20of%20sync_binlog%20is%201.

memcache

• tutorial https://www.tutorialspoint.com/memcached/memcached_quick_guide.htm

• list all keys https://bobcares.com/blog/memcached-list-all-keys/

• memcache replacements which can persist data https://stackoverflow.com/questions/1316852/alternative-to-memcached-that-can-persist-to-disk

mcrouter

https://github.com/facebook/mcrouter/wiki https://engineering.fb.com/2014/09/15/web/introducing-mcrouter-a-memcached-protocol-router-for-scaling-memcached-deployments/

• route handles https://github.com/facebook/mcrouter/wiki/List-of-Route-Handles#poolroute

• prefix selector route https://github.com/facebook/mcrouter/wiki/Config-Files

• combining OperationSelectorRoute and PrefixSelectorRoute facebook/mcrouter#65

elasticsearch

• user authorization https://www.elastic.co/guide/en/elasticsearch/reference/current/authorization.html

• define role https://www.elastic.co/guide/en/elasticsearch/reference/current/defining-roles.html

• certutil https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html

• keystore vs truststore https://www.baeldung.com/java-keystore-truststore-difference

• xpack security settings https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-settings.html

• security realms https://www.elastic.co/guide/en/elasticsearch/reference/7.17/realms.html

• security privileges https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html

• common cluster issues https://www.elastic.co/guide/en/elasticsearch/reference/7.17/fix-common-cluster-issues.html

• lifecycle management errors https://www.elastic.co/guide/en/elasticsearch/reference/7.17/index-lifecycle-error-handling.html

• automate rollover https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html

• skip rollover https://www.elastic.co/guide/en/elasticsearch/reference/current/skipping-rollover.html

• create or update index template https://www.elastic.co/guide/en/elasticsearch/reference/7.17/indices-put-template.html https://www.elastic.co/guide/en/elasticsearch/reference/7.17/indices-component-template.html

• memory lock https://www.elastic.co/guide/en/elasticsearch/reference/7.17/_memory_lock_check.html

kibana

• define role https://www.elastic.co/guide/en/kibana/7.17/role-management-api-put.html

• xpack.encryptedSavedObjects.encryptionKey https://www.elastic.co/guide/en/kibana/7.9/alert-action-settings-kb.html#general-alert-action-settings

securing password in saltstack

• gpg https://docs.saltproject.io/en/latest/ref/renderers/all/salt.renderers.gpg.html https://fabianlee.org/2016/10/18/saltstack-keeping-salt-pillar-data-encrypted-using-gpg/ https://notesfromthelifeboat.com/post/salt-secrets/ https://sgoel.dev/posts/encrypting-sensitive-data-in-saltstack/

• sdb https://docs.saltproject.io/en/latest/topics/sdb/

• how I did it (all commands were executed on salt master), my key name is saltstack sudo mkdir -p /etc/salt/gpgkeys sudo chmod 0700 /etc/salt/gpgkeys sudo gpg --gen-key --homedir /etc/salt/gpgkeys (without passphrase) sudo gpg --homedir /etc/salt/gpgkeys --export-secret-keys --armor > /etc/salt/gpgkeys/exported_private.key sudo gpg --homedir /etc/salt/gpgkeys --armor --export > /etc/salt/gpgkeys/exported_pubkey.gpg sudo gpg --import /etc/salt/gpgkeys/exported_pubkey.gpg echo -n [password] | gpg --armor --batch --trust-model always --encrypt -r saltstack

write module in salt

• write module https://docs.saltproject.io/en/latest/ref/modules/index.html#virtual-function https://www.linode.com/docs/guides/create-a-salt-execution-module/

• request library https://www.geeksforgeeks.org/get-post-requests-using-python/?ref=lbp

metricbeat

• install https://www.elastic.co/guide/en/beats/metricbeat/7.17/metricbeat-installation-configuration.html

• privileges https://www.elastic.co/guide/en/beats/metricbeat/7.17/privileges-to-setup-beats.html

• elasticsearch https://www.elastic.co/guide/en/beats/metricbeat/7.17/metricbeat-module-elasticsearch.html https://www.elastic.co/guide/en/beats/metricbeat/7.17/configuration-ssl.html https://www.elastic.co/guide/en/beats/metricbeat/7.17/securing-communication-elasticsearch.html

• kibana https://www.elastic.co/guide/en/beats/metricbeat/7.17/metricbeat-module-kibana.html

• mysql https://www.elastic.co/guide/en/beats/metricbeat/7.17/metricbeat-module-mysql.html

• redis https://www.elastic.co/guide/en/beats/metricbeat/7.17/metricbeat-module-redis.html

• memcached https://www.elastic.co/guide/en/beats/metricbeat/7.17/metricbeat-module-memcached.html

• processors: include_fields https://www.elastic.co/guide/en/beats/metricbeat/7.17/include-fields.html

• haproxy https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-haproxy.html

• monitoring would monitor an outside cluster add it to standalone cluster, xpack.monitoring would monitor itself elastic/beats#13182

• change index name https://www.elastic.co/guide/en/beats/metricbeat/7.17/change-index-name.html

filebeat

• install https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-installation-configuration.html

• privileges https://www.elastic.co/guide/en/beats/filebeat/7.17/privileges-to-setup-beats.html

• elasticsearch https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-module-elasticsearch.html

• kibana https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-module-kibana.html

• mysql https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-module-mysql.html

• redis https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-module-redis.html

• journalctl https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-input-journald.html

• syslog https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-input-syslog.html

• filestream https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html

• haproxy https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-haproxy.html

• monitoring would monitor an outside cluster add it to standalone cluster, xpack.monitoring would monitor itself elastic/beats#13182

• load kibana dashboards https://www.elastic.co/guide/en/beats/filebeat/7.17/load-kibana-dashboards.html

• ilm https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html

• #to be read https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files

logstash

• install https://www.elastic.co/guide/en/logstash/7.17/installing-logstash.html

• pipeline https://www.elastic.co/guide/en/logstash/7.17/advanced-pipeline.html

• security https://www.elastic.co/guide/en/logstash/current/ls-security.html

• centralized pipeline management https://www.elastic.co/guide/en/logstash/current/logstash-centralized-pipeline-management.html

• distribute pipelines https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html#distributor-pattern https://discuss.elastic.co/t/distinguishing-between-log-files-when-sending-multiple-log-files-from-filebeat-to-logstash/266043

• failed to read pipeline https://discuss.elastic.co/t/pipelines-yml-failed-to-read-with-pipeline-to-pipeline-model/281659

• filters https://logz.io/blog/5-logstash-filter-plugins/ https://www.elastic.co/guide/en/logstash/current/filter-plugins.html

• monitoring error https://discuss.elastic.co/t/issues-sending-monitoring-data-from-logstash-to-elasticsearch/215103/8

• why use filebeat instead of logstash file input https://stackoverflow.com/questions/58585855/difference-between-using-filebeat-and-logstash-to-push-log-file-to-elasticsearch#:~:text=Another%20advantage%20of%20using%20filebeat,lose%20events%20in%20some%20cases.

• grok patterns https://logz.io/blog/logstash-grok/#:~:text=Put%20simply%2C%20grok%20is%20a,%2C%20Linux%2DSyslog%20and%20more. /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.3.4/patterns/ecs-v1/

• differenciate between modules in pipeline https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html#use-ingest-pipelines https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html

• multiple grok patterns for a single pipeline https://stackoverflow.com/questions/50349637/what-is-the-correct-way-to-have-multiple-grok-patterns-in-a-logstash-filter https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-match

• grok online tester https://grokconstructor.appspot.com/do/match#result https://grokdebugger.com/

• redis log format https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Log_Delivery.html

• mysql log format https://dev.mysql.com/doc/refman/8.0/en/error-log-format.html

• add new field with another field value https://discuss.elastic.co/t/how-to-create-a-new-field-using-value-from-another-field/175885

• grok for elasticsearch https://gist.github.com/radu-gheorghe/5404512ca75029ead9b9

• remove ilm policy from index https://www.elastic.co/guide/en/elasticsearch/reference/7.17/ilm-remove-policy.html

elastic integration

Elastic Agent is a single, unified way to add monitoring for logs, metrics, and other types of data to a host. It can also protect hosts from security threats, query data from operating systems, forward data from remote services or hardware, and more

https://docs.elastic.co/integrations/all_integrations

sadly most features need kibana and elastic version 8 or newer

haproxy

• stats https://www.haproxy.com/blog/exploring-the-haproxy-stats-page/

rsyslog

• config https://fabianlee.org/2017/05/24/ubuntu-enabling-syslog-on-ubuntu-and-custom-templates/

salt

https://docs.saltproject.io/salt/install-guide/en/latest/topics/accept-keys.html

• add new node salt-key --gen-keys [name_of_node] # run this on master node and mv it to /vagrant/keys
• list all keys salt-key -L
• accept key salt-key -a [key-name] # accept a single key salt-key -A # accept all keys
• delete key salt-key -d [key-name] # delete a single key salt-key -D # delete all keys