add production monitoring best practices

dir.yaml

@@ -845,6 +845,7 @@
       path: observability/overview
       collapsed: true
       children:
+        - observability/monitoring-best-practices
         - observability/metrics-and-stats
         - observability/alarms
         - observability/log

en_US/observability/log.md

@@ -204,3 +204,40 @@ If any events are throttled within a time window, a summary warning message will
 ```
 
 As you can see, the first "authorization_permission_denied" event is fully logged. The next 4 similar events are dropped but their number is recorded in "log_events_throttled_during_last_period" statistics.
+
+## Centralize Logs in Production
+
+In production, send logs from every EMQX node to a central system outside the EMQX cluster. Logs kept only on the broker host may become unavailable when the node or its storage fails. Central collection also makes it possible to correlate events across Core and Replicant nodes and to alert on conditions that are not exposed as metrics or built-in alarms.
+
+### Choose a Collection Method
+
+Use one of the following collection patterns:
+
+- In a containerized deployment, such as Kubernetes, write JSON logs to the console and use the platform's logging agent to collect the container output.
+- For file logging, use a log agent that collects `emqx.log.N` files, handles rotation without duplicating records, and preserves the structured fields.
+- Use the [OpenTelemetry log handler](./opentelemetry/logs.md) to export logs to an OpenTelemetry Collector and a compatible backend.
+
+### Add Context and Protect Logs
+
+Add deployment metadata such as cluster, node, node role, EMQX version, and availability zone in the collection pipeline.
+
+Protect centralized logs as operational data. Log fields can contain client IDs, usernames, topics, peer addresses, and error details.
+
+### Monitor the Collection Pipeline
+
+Monitor the collection path by using collector and transport health metrics or an explicit heartbeat that does not depend on application log volume. Configure alerts for the following conditions:
+
+- The collector or transport is unhealthy.
+- The collector or transport rejects or drops records.
+- The central backend approaches its storage or retention limits.
+
+Do not alert merely because a reachable EMQX node produces no logs. An idle or healthy node may have nothing to report at the configured severity.
+
+### Define a Log Alerting Policy
+
+Create log-based alerts selectively and match stable structured fields such as `level` and `msg`.
+
+- **Warning events:** These events are often useful as early-warning signals, but some can be caused by expected client behavior. Use a rate or deviation from the normal baseline where individual events do not require action.
+- **Error or critical events:** Events that indicate loss of replication, configuration synchronization, listener startup, or durable storage should normally alert immediately.
+
+For a recommended set of metric- and log-based alerts, including Mria replication signals, see [Production Monitoring Best Practices](./monitoring-best-practices.md#centralize-logs-and-alert-selectively).

en_US/observability/overview.md

@@ -1,8 +1,14 @@
 # Logs and Observability
 
-EMQX provides a series of observability-related features to help with system monitoring, management, and diagnosing. All these features can be accessed and configured on the Dashboard under the following menu items:
+EMQX provides built-in observability features, external integrations, and production monitoring guidance to help you monitor, manage, and diagnose a deployment. The following pages cover production monitoring, observability data and integrations, and diagnostic tools.
 
-**Monitoring**:
+**Production Monitoring**:
+
+- [Production Monitoring Best Practices](./monitoring-best-practices.md)
+
+  Design production monitoring and alerts to identify availability, capacity, dependency, and message-delivery risks and support timely maintenance.
+
+**Observability and Integrations**:
 
 - [Metrics](./metrics-and-stats.md)
 
@@ -12,8 +18,6 @@ EMQX provides a series of observability-related features to help with system mon
 
   EMQX has offered a built-in monitoring and alarm functionality for monitoring the CPU occupancy, system and process memory occupancy, number of processes, rule engine resource status, cluster partition and healing, and will raise an alarm in case of system malfunctions.
 
-**Management**:
-
 - [Logs](./log.md)
 
   Logs provide a reliable source of information for troubleshooting and system performance optimization. You can find the record about the access, operating, or network issues from EMQX logs.
@@ -26,7 +30,7 @@ EMQX provides a series of observability-related features to help with system mon
 
   [Datadog](https://www.datadoghq.com/) is an observability platform that provides unified, real-time observability and security solutions for applications. EMQX supports the integration of Datadog to help you understand the EMQX operating status, monitor and troubleshoot system performance issues, and view EMQX metrics on the Datadog console.
 
-**Diagnose**:
+**Diagnostics**:
 
 - [Topic Metrics](./topic-metrics.md)
 
@@ -39,6 +43,3 @@ EMQX provides a series of observability-related features to help with system mon
 - [Log Trace](./tracer.md)
 
   EMQX 5.x has added the Log Trace feature, allowing users only to enable debug-level logs output for specific client IDs, topics or IPs in real-time.
-
-
-