Add documentation for default logging configuration in Airflow #61203

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open

sanchalitorpe-source wants to merge 2 commits into apache:main from sanchalitorpe-source:add-logging-docs

+92 −2

airflow-core/src/airflow/airflow/docs/apache-airflow/logging-configuration.rst

-Original file line number
+Diff line change
@@ -0,0 +1,70 @@
+    Default Logging in Apache Airflow
+    =================================
+    Apache Airflow has multiple loggers for different components, which can be confusing for new users.
+    This section explains the default loggers, their purposes, and how to modify their behavior.
+    Default Loggers
+    ---------------
+    +------------------------+----------------+----------------------------+--------------------------------+
+    | Logger Name            | Component      | Output                     | Notes                          |
+    +========================+================+============================+================================+
+    | root                   | Webserver      | stdout / webserver log     | Default root logger used by webserver. |
+    +------------------------+----------------+----------------------------+--------------------------------+
+    | airflow.task           | Scheduler/Worker | logs/<dag_id>/<task_id>/<execution_date>/<try_number>.log | A new log file is created per task instance and try. Shown in the Web UI. |
+    +------------------------+----------------+----------------------------+--------------------------------+
+    | airflow.processor      | Scheduler/Worker | logs/<dag_file_name>.log   | Logs DAG parsing for scheduler and workers. |
+    +------------------------+----------------+----------------------------+--------------------------------+
+    | airflow.processor_manager | Scheduler    | logs/<dag_file_name>.log   | Logs task instance execution control. |
+    +------------------------+----------------+----------------------------+--------------------------------+
+    | flask_appbuilder       | Webserver      | filters verbose FAB logs  | Typically used for filtering; no config needed by most users. |
+    +------------------------+----------------+----------------------------+--------------------------------+
+    Logging by Airflow Component
+    ----------------------------
+    - **Webserver**: Uses the root logger. Logs to stdout and webserver log file.
+    - **Worker**: Uses `airflow.task` and `airflow.processor`. Task logs are stored per task instance. DAG parsing logs are stored per DAG file.
+    - **Scheduler**: Uses `airflow.processor`, `airflow.processor_manager`, and the root logger.
+    Customizing Logging
+    -------------------
+    You can influence the logging configuration using the following methods:
+. **Configuration via airflow.cfg**
+       - `[logging]` section allows changing:
+         - Base log folder (`base_log_folder`)
+         - Remote logging settings
+         - Logging format
+. **Custom Python logging configuration**
+       - Airflow uses `airflow.utils.log.logging_config.py`
+       - You can override `LOGGING_CONFIG` in `airflow_local_settings.py`
+       - Example:
+       .. code-block:: python
+           from airflow.utils.log.logging_config import DEFAULT_LOGGING_CONFIG
+           LOGGING_CONFIG = DEFAULT_LOGGING_CONFIG.copy()
+           LOGGING_CONFIG['handlers']['console']['level'] = 'INFO'
+. **Environment Variables**
+       - Some logging options can be set via environment variables, e.g.:
+         - `AIRFLOW__LOGGING__BASE_LOG_FOLDER`
+         - `AIRFLOW__LOGGING__REMOTE_LOGGING`
+    Recommendations
+    ---------------
+    - Use `airflow.task` logs to debug task failures.
+    - Use `airflow.processor` to debug DAG parsing issues.
+    - For production, consider remote logging (S3, GCS, Elasticsearch) for scalability.
+    - Do **not** modify `flask_appbuilder` logger unless needed.
+    References
+    ----------
+    - :doc:`/configuration/logging`
+    - :ref:`task-logs`

airflow-core/src/airflow/ui/src/components/DataTable/DataTable.tsx

-Original file line number
+Diff line change
@@ Expand Up / @@ -38,7 +38,14 @@ @@
     import { createSkeletonMock } from "src/components/DataTable/skeleton";
     import type { CardDef, MetaColumn, TableState } from "src/components/DataTable/types";
     import { ProgressBar, Pagination, Toaster } from "src/components/ui";
+    const loadFromStorage = <T,>(key: string, fallback: T): T => {
+      try {
+        const raw = localStorage.getItem(key);
+        return raw ? JSON.parse(raw) : fallback;
+      } catch {
+        return fallback;
+      }
+    };
     type DataTableProps<TData> = {
       readonly allowFiltering?: boolean;
       readonly cardDef?: CardDef<TData>;
@@ Expand Down Expand Up / @@ -83,6 +90,7 @@ @@
       skeletonCount = 10,
       total = 0,
     }: DataTableProps<TData>) => {
+      const storageKey = `airflow.datatable.${modelName}`;
       "use no memo"; // remove if https://github.com/TanStack/table/issues/5567 is resolved
       const { t: translate } = useTranslation(["common"]);
@@ Expand All / @@ -109,9 +117,21 @@ @@
         [onStateChange],
       );
-      const [columnVisibility, setColumnVisibility] = useState<VisibilityState>(
+      const [columnVisibility, setColumnVisibility] = useState<VisibilityState>(() =>
+      loadFromStorage(
+        `${storageKey}.columnVisibility`,
         initialState?.columnVisibility ?? {},
+      ),
+    );
+      useEffect(() => {
+      localStorage.setItem(
+        `${storageKey}.columnVisibility`,
+        JSON.stringify(columnVisibility),
       );
+      }, [columnVisibility, storageKey]);
       const rest = Boolean(isLoading) ? createSkeletonMock(displayMode, skeletonCount, columns) : {};
@@ Expand Down @@

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add documentation for default logging configuration in Airflow #61203

Diff view

Diff view

There are no files selected for viewing

GitHub Actions / Basic tests / React UI tests

GitHub Actions / Basic tests / React UI tests

GitHub Actions / Basic tests / React UI tests

GitHub Actions / Basic tests / React UI tests

GitHub Actions / Basic tests / React UI tests

GitHub Actions / Basic tests / React UI tests

GitHub Actions / Basic tests / React UI tests

GitHub Actions / Basic tests / React UI tests

GitHub Actions / Basic tests / React UI tests

GitHub Actions / Basic tests / React UI tests

Uh oh!

Add documentation for default logging configuration in Airflow #61203

Are you sure you want to change the base?

Add documentation for default logging configuration in Airflow #61203

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

GitHub Actions / Basic tests / React UI tests

GitHub Actions / Basic tests / React UI tests

GitHub Actions / Basic tests / React UI tests

GitHub Actions / Basic tests / React UI tests

GitHub Actions / Basic tests / React UI tests

GitHub Actions / Basic tests / React UI tests

GitHub Actions / Basic tests / React UI tests

GitHub Actions / Basic tests / React UI tests

GitHub Actions / Basic tests / React UI tests

GitHub Actions / Basic tests / React UI tests

Uh oh!