diff --git a/survival_guide/1_concepts.adoc b/survival_guide/1_concepts.adoc index b7a976e..417aa76 100644 --- a/survival_guide/1_concepts.adoc +++ b/survival_guide/1_concepts.adoc @@ -72,12 +72,10 @@ The table below lists the Fusion microservices deployed by our Helm chart. Recog |=== |Microservice |Protocol |Deployment or StatefulSet |Node Pool Assignment |Autoscaling Supported |Description -|`admin` |REST/HTTP |Deployment |`system` |Not required. Minimum of 1 but 2 pods are recommended for HA |Exposes endpoints for admin tasks, such as creating applications and running jobs. +|`fusion-admin` |REST/HTTP |Deployment |`system` |Not required. Minimum of 1 but 2 pods are recommended for HA |Exposes endpoints for admin tasks, such as creating applications and running jobs. |`admin-ui` |Web |Deployment |`system` |Not required; only 1 pod should be sufficient for most clusters |Serves static Web assets for the admin UI. -|`argo` |HTTP |Deployment |`system` |Yes (CPU or custom metric) |Orchestrates parallel jobs on Kubernetes. - |`argo-ui` |Web |Deployment |`system` |Not required; only 1 pod should be sufficient for most clusters |Stores logs and prior Argo workflow runs. |`auth-ui` |Web |Deployment |`system` |Not required; only 1 pod should be sufficient for most clusters |Serves static Web assets for the login form. @@ -90,9 +88,7 @@ The table below lists the Fusion microservices deployed by our Helm chart. Recog |`connector-plugin-` | HTTP/TCP |Deployment |`analytics` or `system` |Yes (CPU or custom metric) | Deployment for each connector plugin type. *Note:* There is a base deployment, `connector-plugin` with 0 replicas. This is used as a deployment template for each connector plugin type. It should not be deleted or scaled. -|`devops-ui` |Web |Deployment |`system` |Not required; only 1 pod should be sufficient for most clusters |Serves static Web assets for the DevOps UI. - -|`indexing` |REST/HTTP |Deployment |`search` or `analytics` depending on write-volume |Yes (CPU or custom metric) |Processes indexing requests. +|`fusion-indexing` |REST/HTTP |Deployment |`search` or `analytics` depending on write-volume |Yes (CPU or custom metric) |Processes indexing requests. |`insights` |Web |Deployment |`system` |Not required; only 1 pod should be sufficient for most clusters |Serves the App Insights UI @@ -100,10 +96,6 @@ The table below lists the Fusion microservices deployed by our Helm chart. Recog |`job-rest-server` |REST/HTTP |Deployment |`analytics` |Not required; only 1 pod should be sufficient for most clusters |Performs admin tasks for creating and running Spark jobs. -|`jupyter` |HTTP |Deployment |`analytics` |Not required; only 1 pod should be sufficient for most clusters |Jupyter notebook for ad hoc analytics and visualization. - -|`logstash` |HTTP |StatefulSet |`system` |Not required. Minimum of 1 but 2 pods are recommended for HA |Collects logs from the other microservices and either indexes into system_logs or ships them to an external service like Elastic - |`milvus` |REST/HTTP |Deployment |`analytics` or `system`| Not required; only 1 pod should be sufficient for most clusters |Dense Vector Search Engine for ML models active. |`milvus-mysql` |REST/HTTP |Deployment |`analytics` or `system` |Not required; only 1 pod should be sufficient for most clusters |Handles metadata for Milvus service active. @@ -114,11 +106,7 @@ The table below lists the Fusion microservices deployed by our Helm chart. Recog |`proxy` / `api-gateway` |HTTP |Deployment |`search` |Not required. Minimum of 1 but 2 pods are recommended for HA |Performs authentication, authorization, and traffic routing. -|`pulsar-bookkeeper` |HTTP |StatefulSet | `search` | At least 3 nodes in HA, you need to run 3 or 5 to ensure a quorum | Write Ahead Log (WAL) used for persistent message storage. - -|`pulsar-broker` |HTTP and TCP | Deployment | `search` | At least 3 nodes in HA | Contains REST API for managing administration and dispatcher for handling all message transfers. - -|`query` |REST/HTTP |Deployment |`search` |Yes (CPU or custom metric) |Processes query requests. +|`query-pipeline` |REST/HTTP |Deployment |`search` |Yes (CPU or custom metric) |Processes query requests. |`rules-ui` |Web |Deployment |`system` |Not required; only 1 pod should be sufficient for most clusters |Serves static Web assets for the Rules UI. @@ -134,18 +122,14 @@ The table below lists the Fusion microservices deployed by our Helm chart. Recog |`spark-executor` |n/a |one or more pods launched by the Spark driver for executing job tasks |`analytics` or dedicated Node Pool for Spark jobs |depends on job configuration; controlled by the spark.executor.instances setting |Executes tasks for a Spark job -|`sql-service` |REST/HTTP and JDBC |Deployment |`analytics` |Not required; only 1 pod should be sufficient for most clusters |Performs admin tasks for creating and managing SQL catalog assets. - -Exposes a JDBC endpoint for the SQL service. - |`templating` |Web |Deployment |`system` |Not required; only 1 pod should be sufficient for most clusters |Retrieves and renders Predictive Merchandiser templates. -|`tikaserver` |HTTP |Deployment |`system` |Not required; only 1 pod should be sufficient for most clusters |Is responsible for text extraction using Tika Server Parser - |`webapps` |REST/HTTP |Deployment |`system` |Not required; only 1 pod should be sufficient for most clusters |Serves App Studio-based Web apps. |`zookeeper` |TCP |StatefulSet |`system` |No, you need to run 1,3, or 5 Zookeeper pods to ensure a quorum; HPA should not be used for scaling ZK |Stores centralized configuration and performs distributed coordination tasks. +|`kafka` |TCP |StatefulSet |`system` |Yes, required; only 1 pod should be sufficient for most clusters | Topic are used to hold incomming data for solr + |=== == Ports used by Fusion @@ -160,11 +144,9 @@ Below you will find the list of ports required to access Fusion services. |`auth-ui` | 8080 |`classic-rest-service` | 9000 |`connector-plugin-service` | 9020 -|`devops-ui` | 8080 |`dnsAccess` | 53 |`fusion-admin` | 8765 |`fusion-indexing` | 8765 -|`fusion-jupyter` | 8888 |`insights` | 8080 |`job-launcher` | 8083 |`job-rest-server` | 8081 @@ -174,17 +156,16 @@ Below you will find the list of ports required to access Fusion services. |`ml-model-service milvus` | 3000 |`ml-model-service mysql` | 3306 |`pm-ui` | 8080 -|`pulsar` | 8080, 8443, 6650, 6651, 9527 |`query-pipeline` | 8787 |`rest-service` | 9010 |`rpc-service` | 8771, 8772 |`rules-ui` | 8080 |`solr` | 8983, 9983 |`solr-backup-runner` | 8983 -|`sql-service` | 5250 |`templating` | 5250 |`webapps` | 8780 |`zookeeper` | 2181, 2888, 3888, 2281, 9141 +|`kafka` | 9092, 9093 |===