Merge pull request #18 from f5devcentral/develop

LADT module inclusion

Author: tholemu

docs/class1/module3/lab3.rst

@@ -143,3 +143,7 @@ Available Metrics include:
 .. note:: One of the *best* ways to ensure you're in the know as it relates to the **F5 AST** is by having a conversation with *your F5 account team* and voicing your interest, thoughts, and opinions.
 
     Unsure who's on your account team? No problem. Head over to `Get F5 <https://www.f5.com/products/get-f5>`_  and submit an inquiry mentioning the **F5 Application Study Tool**.
+
+But wait, THERE'S MORE! We've added a bonus section to the UDF lab environment and this guide: **the F5 Local API Discovery Tool**.
+
+Select **Next** below for a preview of this extension to the **F5 AST!**

docs/class1/module4/images/api-detection-screenshot.png

@@ -0,0 +1,150 @@
+.. _LADT - Overview:
+
+F5 LADT Overview
+================
+
+In this lab, we'll run through an overview of the **F5 Local API Discovery Tool**. At a high level, the **F5 LADT** is:
+
+*The AST Local API Discovery Tool (LADT) is a fork of the Application Study Tool focused on analyzing data about customer APIs based on data from classic F5 BIG-IP logs.*
+
+*The LADT receives per-http-request data from F5 BIG-IP devices and stores relevant fields for periodic batch analysis. Output of the analysis is a list of likely API endpoints, whether or not they are authenticated, and the number of requests processed by each endpoint. API detection logic is based on a subset of official F5 product offerings, with some advanced functionality (e.g. schema discovery and Machine Learning analysis) excluded for performance reasons.*
+
+Example API Detection Output
+----------------------------
+
+.. image:: images/api-detection-screenshot.png
+    :width: 800
+
+System Overview
+---------------
+
+The F5 LADT relies on a specially-crafted iRule to export per-request metadata to the analysis system, running on-prem. The analysis system consists of:
+
+- Opentelemetry Collector: For receiving incoming logs, processing, and forwarding them to storage system
+- ClickHouse Database: For storing log data and running analysis queries
+- Analysis Job: Runs periodically to generate results
+- Grafana: For visualizing results, and system component states
+
+.. image:: images/system-overview.png
+    :width: 800
+
+System Prerequisites and Recommendations
+----------------------------------------
+
+- Linux system with docker (or compatible) - Installation Instructions
+- F5 BIG-IP with logging iRule (contact your F5 Team for access)
+- Network connectivity (default TCP port 6514) between the BigIP and instance running LADT
+
+Minimal system specs for running the Local API Discovery Tool will depend heavily on expected traffic load.
+
+The following performance characteristics were observed with an Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz running localized testing
+(single TCP+TLS connection meant to simulate a stream of data from a single BigIP Device). Actual message throughput
+under real world network conditions would likely be quite a bit lower.
+
+.. list-table:: Performance Characteristics
+   :widths: 15 25 25 25 25
+   :header-rows: 1
+
+   * - Msg/Sec
+     - Collector CPU Util
+     - Clickhouse CPU Util
+     - Collector Mem Util
+     - Clickhouse Mem Util
+   * - 0
+     - 0.5%
+     - 0.8%
+     - 40MiB
+     - 546MiB
+   * - 1
+     - 0.7%
+     - 1.3%
+     - 41.5MiB
+     - 629MiB
+   * - 10
+     - 0.8%
+     - 1.6%
+     - 58MiB
+     - 629MiB
+   * - 100
+     - 1.7%
+     - 1.8%
+     - 66MiB
+     - 646MiB
+   * - 1000
+     - 8%
+     - 2.8%
+     - 115MiB
+     - 650MiB
+   * - 10000
+     - 59.3%
+     - 40%
+     - 176MiB
+     - 698MiB
+   * - 20000
+     - 133%
+     - 44%
+     - 216MiB
+     - 750MiB
+   * - 30000
+     - 180%
+     - 47%
+     - 216MiB
+     - 798MiB
+   * - 40000
+     - 235%
+     - 70%
+     - 225MiB
+     - 948MiB
+   * - 54000
+     - 298%
+     - 106%
+     - 245MiB
+     - 1000MiB
+
+The analysis script adds an additional ~10% CPU and 250MiB of load to the clickhouse instance while performing analysis.
+
+.. list-table:: Analysis Characteristics
+   :widths: 25 25
+   :header-rows: 1
+
+   * - Analyzed Messages
+     - Analysis Duration Sec
+   * - 1000
+     - 113ms
+   * - 10000
+     - 0.9s
+   * - 10000
+     - 6.67s
+   * - 20000
+     - 14.7s
+   * - 1.04M
+     - 1.12min
+   * - 2.2M
+     - 2.51min
+   * - 10M
+     - 4.06min
+
+Disk space usage likewise depends on volume, but clickhouse provides a good amount of compression (here 758 million entries are stored in ~210MiB).
+
+.. list-table:: Disk Space Usage
+   :widths: 15 15 15 15 15 15
+   :header-rows: 1
+
+   * - Rows
+     - Disk Size
+     - Bytes Size
+     - Compressed Size
+     - Uncompressed Size
+     - Ratio
+   * - 758799480
+     - 210.24MiB
+     - 220452423
+     - 209.25 MiB
+     - 21.16 GiB
+     - 0.00965486284484136
+
+.. note:: As previously mentioned, we won't get digging into the installation details or process within this lab module.
+    
+    If you're interested in going down that path and learning more, please visit the `F5 LADT GitHub Repo's Installation section <https://github.com/f5devcentral/ast-api-discovery/blob/main/README.md#installation>`_
+
+Select **Next** below for to learn how to access the F5 LADT Grafana instance in the UDF lab environment.

docs/class1/module4/images/api_detection_dashboard.png

@@ -0,0 +1,181 @@
+.. _LADT - Grafana Instance Access:
+
+Accessing the F5 LADT Instance Dashboards
+=========================================
+
+We've pre-provisioned an instance of F5 LADT in the UDF lab environment. It's ready for you to dive in and take a look. Just follow the instructions below!
+
+Accessing F5 LADT Grafana Instance
+----------------------------------
+
+#. From within the UDF course deployment's **LADT** System, locate and select **ACCESS**, then **Grafana**.
+
+    .. image:: images/udf_ladt_grafana_access.png
+        :width: 800
+
+#. Once the new browser tab has loaded, you will be presented with the self-installed F5 LADT Grafana login. Enter the following credentials and select **Log in**.
+
+    Username
+
+    .. code-block:: console
+
+        admin
+
+    Password
+
+    .. code-block:: console
+
+        11F5Networks11
+
+    .. image:: ../module2/images/grafana_login.png
+        :width: 800
+
+#. You'll be presented with the Grafana homepage. From here, select the hamburger menu next to **Home** in the upper left corner, then **Dashboards**
+
+    .. image:: images/ladt_grafana_home.png
+        :width: 800
+
+#. Once the Dashboards page has loaded, you can either expand the list of dashboards by selecting the **>** symbol next to **Local API Discovery** or selecting the **Local API Discovery** folder itself.
+
+    .. image:: images/ladt_dashboards.png
+        :width: 800
+
+API Detection Dashboard
+-----------------------
+
+The **API Detection Dashboard** has three sections, as outlined below. Contract the *System Overview* and *System State Detail* sections by selecting the **v** next to each section title, as we're more interested in the *Detection Results*.
+
+* System Overview
+
+  * Clickhouse Service Status
+
+  * OTel Service Status
+
+  * Log Analyzer Status
+
+  * Last Analysis Time
+
+  * Logs Analyzed Last Time
+
+  * New Logs Since Last Analysis
+
+* System State Detail
+
+  * Analysis Settings
+
+  * Logs Last Processed by Host
+
+  * Logs Received
+
+  * Logs Processed at Last Analysis
+
+  * Last Analysis Duration
+
+  * CPU Utilization
+
+  * Memory Usage
+
+* Detection Results
+
+  * Application Name
+
+  * HTTP Method
+
+  * Path
+
+  * Authentication Detected
+
+  * Sensitive Headers Present
+
+  * Sensitive Body Present
+
+  * Sensitive Data Types
+
+  * # of Requests
+
+.. image:: images/api_detection_dashboard.png
+    :width: 800
+
+As detailed above, **Detection Results** is where all of the API endpoints and their detected properties will be listed. Straight away, you will be able to get a view of the number of requests an API resource has seen for a given method and whether or not sensitive data or headers have been detected.
+
+ClickHouse Dashboard
+--------------------
+
+**ClickHouse** is the software used to store logs and perform analysis queries. Its dashboard has 12 sections, which are listed below. This is not necessarily an aspect of the tool we expect students and administrators to interact with on a regular basis. However, it's a good idea to understand the scope of which metrics are captured and represented here in the event troubleshooting is required.
+
+* System Metrics
+
+  * Nodes
+
+  * Memory
+
+  * Connections
+
+  * Read backoff
+
+  * Slow reads
+
+* Queries
+
+* Insert
+
+* Select
+
+* IO
+
+* Replicas
+
+* Merge
+
+* Cache
+
+* Parts
+
+* Distributed
+
+* Background pool
+
+* Zookeeper
+
+.. note:: Many (if not all) of the metrics found within this dashboard may be foreign and daunting if you're not already familiar with **ClickHouse** and/or databases in general.
+    
+    If you would like to get a better understanding of their role, contextual information is available via the *information icon* to the right of each metric label. Hover over the icon to view tooltip details. 
+
+.. attention:: References to *clickhouse:9126* indicate the default port upon which the **ClickHouse** runs: *9126*
+
+.. image:: images/clickhouse_dashboard.png
+    :width: 800
+
+OpenTelemetry Collector Dashboard
+---------------------------------
+
+The **OpenTelemetry Collector Dashboard** contains all information pertaining to the *OTel Collector* employed in receiving, processing, and forwarding incoming logs to the storage system (ClickHouse). This dashboard contains the following eight sections:
+
+There's a section dedicated to *documentation*, should you have the desire to learn more about telemetry metrics collection!
+
+* Receivers
+
+* Processors
+
+* Exporters
+
+* Collector
+
+* Signal Flows
+
+* RPC server/client
+
+* Kubernetes
+
+* Documentation
+
+.. note:: As mentioned within the **ClickHouse Dashboards** portion above, you may obtain a better understanding of OTel Collector metric roles via the *information icon* to the right of each metric label. Hover over the icon to view tooltip details. 
+
+.. image:: images/otel_collector_dashboard.png
+    :width: 800
+
+That's a wrap for the **F5 Application Study Tool Introduction** class, folks. Check back often for updates to the guide, as we have exciting plans for future labs!
+
+Our **appreciation** for *your* **time** and **energy** goes *beyond what words can convey*.
+
+.. attention:: We would like to encourage you to provide feedback through your account team and also the GitHub repos, should you encounter issues or have enhancement requests!

docs/class1/module4/images/clickhouse_dashboard.png

@@ -0,0 +1,20 @@
+.. _Exploring the F5 LADT:
+
+Module 4: Exploring the F5 Local API Discovery Tool
+===================================================
+
+Here we'll provide a sneak peak at an extension of the **F5 AST** called the **F5 Local API Discovery Tool (LADT)**. The **F5 LADT** is a new development which students can explore in the UDF lab environment, and even get their hands on from the official `F5 DevCentral LADT GitHub Repo! <https://github.com/f5devcentral/ast-api-discovery>`_
+
+.. note:: At the time of this lab module's publication, the UDF lab environment is not geared toward students performing installation and configuration.
+    
+    We will be amending the lab in order to mimick that of the F5 AST portions; allowing students to either interact with a pre-installed copy and/or run through the process of installing it themselves.
+
+
+
+Please select **Next** and proceed to the :ref:`LADT - Overview`.
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   lab*