Merge pull request #16 from f5devcentral/module4-ladt-intro

Module4 ladt intro

Author: tholemu

docs/class1/module4/images/api-detection-screenshot.png

@@ -0,0 +1,150 @@
+.. _LADT - Overview:
+
+F5 LADT Overview
+================
+
+In this lab, we'll run through an overview of the **F5 Local API Discovery Tool**. At a high level, the **F5 LADT** is:
+
+*The AST Local API Discovery Tool (LADT) is a fork of the Application Study Tool focused on analyzing data about customer APIs based on data from classic F5 BIG-IP logs.*
+
+*The LADT receives per-http-request data from F5 BIG-IP devices and stores relevant fields for periodic batch analysis. Output of the analysis is a list of likely API endpoints, whether or not they are authenticated, and the number of requests processed by each endpoint. API detection logic is based on a subset of official F5 product offerings, with some advanced functionality (e.g. schema discovery and Machine Learning analysis) excluded for performance reasons.*
+
+Example API Detection Output
+----------------------------
+
+.. image:: images/api-detection-screenshot.png
+    :width: 800
+
+System Overview
+---------------
+
+The F5 LADT relies on a specially-crafted iRule to export per-request metadata to the analysis system, running on-prem. The analysis system consists of:
+
+- Opentelemetry Collector: For receiving incoming logs, processing, and forwarding them to storage system
+- Clickhouse Database: For storing log data and running analysis queries
+- Analysis Job: Runs periodically to generate results
+- Grafana: For visualizing results, and system component states
+
+.. image:: images/system-overview.png
+    :width: 800
+
+System Prerequisites and Recommendations
+----------------------------------------
+
+- Linux system with docker (or compatible) - Installation Instructions
+- F5 BIG-IP with logging iRule (contact your F5 Team for access)
+- Network connectivity (default TCP port 6514) between the BigIP and instance running LADT
+
+Minimal system specs for running the Local API Discovery Tool will depend heavily on expected traffic load.
+
+The following performance characteristics were observed with an Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz running localized testing
+(single TCP+TLS connection meant to simulate a stream of data from a single BigIP Device). Actual message throughput
+under real world network conditions would likely be quite a bit lower.
+
+.. list-table:: Performance Characteristics
+   :widths: 15 25 25 25 25
+   :header-rows: 1
+
+   * - Msg/Sec
+     - Collector CPU Util
+     - Clickhouse CPU Util
+     - Collector Mem Util
+     - Clickhouse Mem Util
+   * - 0
+     - 0.5%
+     - 0.8%
+     - 40MiB
+     - 546MiB
+   * - 1
+     - 0.7%
+     - 1.3%
+     - 41.5MiB
+     - 629MiB
+   * - 10
+     - 0.8%
+     - 1.6%
+     - 58MiB
+     - 629MiB
+   * - 100
+     - 1.7%
+     - 1.8%
+     - 66MiB
+     - 646MiB
+   * - 1000
+     - 8%
+     - 2.8%
+     - 115MiB
+     - 650MiB
+   * - 10000
+     - 59.3%
+     - 40%
+     - 176MiB
+     - 698MiB
+   * - 20000
+     - 133%
+     - 44%
+     - 216MiB
+     - 750MiB
+   * - 30000
+     - 180%
+     - 47%
+     - 216MiB
+     - 798MiB
+   * - 40000
+     - 235%
+     - 70%
+     - 225MiB
+     - 948MiB
+   * - 54000
+     - 298%
+     - 106%
+     - 245MiB
+     - 1000MiB
+
+The analysis script adds an additional ~10% CPU and 250MiB of load to the clickhouse instance while performing analysis.
+
+.. list-table:: Analysis Characteristics
+   :widths: 25 25
+   :header-rows: 1
+
+   * - Analyzed Messages
+     - Analysis Duration Sec
+   * - 1000
+     - 113ms
+   * - 10000
+     - 0.9s
+   * - 10000
+     - 6.67s
+   * - 20000
+     - 14.7s
+   * - 1.04M
+     - 1.12min
+   * - 2.2M
+     - 2.51min
+   * - 10M
+     - 4.06min
+
+Disk space usage likewise depends on volume, but clickhouse provides a good amount of compression (here 758 million entries are stored in ~210MiB).
+
+.. list-table:: Disk Space Usage
+   :widths: 15 15 15 15 15 15
+   :header-rows: 1
+
+   * - Rows
+     - Disk Size
+     - Bytes Size
+     - Compressed Size
+     - Uncompressed Size
+     - Ratio
+   * - 758799480
+     - 210.24MiB
+     - 220452423
+     - 209.25 MiB
+     - 21.16 GiB
+     - 0.00965486284484136
+
+.. note:: As previously mentioned, we won't get digging into the installation details or process within this lab module.
+    
+    If you're interested in going down that path and learning more, please visit the `F5 LADT GitHub Repo's Installation section <https://github.com/f5devcentral/ast-api-discovery/blob/main/README.md#installation>`_
+
+Select **Next** below for to learn how to access the F5 LADT Grafana instance in the UDF lab environment.

docs/class1/module4/images/api_detection_dashboard.png

@@ -0,0 +1,76 @@
+.. _LADT - Grafana Instance Access:
+
+Accessing the F5 LADT Instance Dashboards
+=========================================
+
+We've pre-provisioned an instance of F5 LADT in the UDF lab environment. It's ready for you to dive in and take a look. Just follow the instructions below!
+
+Accessing F5 LADT Grafana Instance
+----------------------------------
+
+#. From within the UDF course deployment's **LADT** System, locate and select **ACCESS**, then **Grafana**.
+
+    .. image:: images/udf_ladt_grafana_access.png
+        :width: 800
+
+#. Once the new browser tab has loaded, you will be presented with the self-installed F5 LADT Grafana login. Enter the following credentials and select **Log in**.
+
+    Username
+
+    .. code-block:: console
+
+        admin
+
+    Password
+
+    .. code-block:: console
+
+        11F5Networks11
+
+    .. image:: ../module2/images/grafana_login.png
+        :width: 800
+
+#. You'll be presented with the Grafana homepage. From here, select the hamburger menu next to **Home** in the upper left corner, then **Dashboards**
+
+    .. image:: images/ladt_grafana_home.png
+        :width: 800
+
+#. Once the Dashboards page has loaded, you can either expand the list of dashboards by selecting the **>** symbol next to **Local API Discovery** or selecting the **Local API Discovery** folder itself.
+
+    .. image:: images/ladt_dashboards.png
+        :width: 800
+
+API Detection Dashboard
+-----------------------
+
+#. The **API Detection Dashboard** has three sections, as outlined below. Contract the *System Overview* and *System State Detail* sections by selecting the **v** next to each section title, as we're more interested in the *Detection Results*.
+
+    * System Overview
+     * Clickhouse Service Status
+     * OTel Service Status
+     * Log Analyzer Status
+     * Last Analysis Time
+     * Logs Analyzed Last Time
+     * New Logs Since Last Analysis
+    - System State Detail
+     - Analysis Settings
+     - Logs Last Processed by Host
+     - Logs Received
+     - Logs Processed at Last Analysis
+     - Last Analysis Duration
+     - CPU Utilization
+     - Memory Usage
+    - Detection Results
+     - Application Name
+     - HTTP Method
+     - Path
+     - Authentication Detected
+     - Sensitive Headers Present
+     - Sensitive Body Present
+     - Sensitive Data Types
+     - # of Requests
+
+    .. image:: images/api_detection_dashboard.png
+        :width: 800
+    
+    As detailed above, **Detection Results** is where all of the API endpoints and their detected properties will be listed. Straight away, you will be able to get a view of the number of requests an API resource has seen for a given method and whether or not sensitive data or headers have been detected.

docs/class1/module4/images/ladt_dashboards.png

@@ -0,0 +1,20 @@
+.. _Exploring the F5 LADT:
+
+Module 4: Exploring the F5 Local API Discovery Tool
+===================================================
+
+Here we'll provide a sneak peak at an extension of the **F5 AST** called the **F5 Local API Discovery Tool (LADT)**. The **F5 LADT** is a new development which students can explore in the UDF lab environment, and even get their hands on from the official `F5 DevCentral LADT GitHub Repo! <https://github.com/f5devcentral/ast-api-discovery>`_
+
+.. note:: At the time of this lab module's publication, the UDF lab environment is not geared toward students performing installation and configuration.
+    
+    We will be amending the lab in order to mimick that of the F5 AST portions; allowing students to either interact with a pre-installed copy and/or run through the process of installing it themselves.
+
+
+
+Please select **Next** and proceed to the :ref:`LADT - Overview`.
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   lab*