Documentation improvements (part 1) (#950)

A bunch of changes to Quesma's documentation:
- Remove a few orphan files - old files, not up-to-date, not referenced
in the menu
- replace `eap.quesma.com` mentions to `docs.quesma.com`
- Document recent changes of `target` configuration (and related
`useCommonTable` and `tableName`)
- Document the expected format of connection string
(`clickhouse://host:port`)
- a bunch of smaller changes

---------

Signed-off-by: Piotr Grabowski <[email protected]>
Co-authored-by: Przemyslaw Delewski <[email protected]>
Co-authored-by: Przemysław Hejman <[email protected]>

Author: 3 people

README.md

@@ -15,7 +15,7 @@
 <a href="https://www.linkedin.com/company/quesma">LinkedIn</a> | 
 <a href="https://twitter.com/QuesmaOrg">Twitter</a> | 
 <a href="https://www.youtube.com/@QuesmaOrg">YouTube</a> | 
-<a href="https://eap.quesma.com/eap-docs">Docs</a>
+<a href="https://docs.quesma.com/eap-docs">Docs</a>
 </p>
 
 Quesma is an actively developed database gateway currently in pre-alpha Early Access. Route your queries through a translation layer that seamlessly connects your apps with the modern database platforms of your choice. No more query refactoring during migrations.

docs/public/README.md

@@ -1,6 +1,6 @@
-# Quesma EAP documentation
+# Quesma documentation
 
-This folder contains our EAP documentation available at https://eap.quesma.com.
+This folder contains our documentation available at https://docs.quesma.com (formerly https://eap.quesma.com).
 These docs are just static files generated with [Vitepress](https://vitepress.dev) and published via CloudFlare Pages.
 
 

docs/public/docs/.vitepress/config.mts

@@ -41,8 +41,6 @@ export default defineConfig({
                   { text: 'Query Hydrolix tables as Elasticsearch indices', link: '/example-2-0'}
                 ]
               },
-              //{ text: 'Scenario I', link: '/scenario-1' },
-              //{ text: 'Reference Docker compose configurations', link: '/reference-conf' }
             ],
           },
           { text: 'Advanced configuration',

docs/public/docs/config-primer.md

@@ -70,12 +70,17 @@ frontendConnectors:
       listenPort: 8080
 ```
 
+The supported configuration options for frontend connectors (under `config`):
+* `listenPort` - port number on which the frontend connector will listen for incoming requests
+* `disableAuth` - when set to `true`, disables authentication for incoming requests (optional, defaults to false). If you use Elasticsearch/Kibana without authentication, set it to `true`.
+
+
 #### Backend connectors
 
 Backend connector has to have a `name`, `type` and `config` fields.
 * `name` is a unique identifier for the connector
 * `type` specifies the type of the connector.\
-  At this moment, only three backend connector types are allowed: `elasticsearch`, `clickhouse` (used for ClickHouse Cloud SaaS service, `clickhouse-os` and `hydrolix`.
+  At this moment, only three backend connector types are allowed: `elasticsearch`, `clickhouse` (used for ClickHouse Cloud SaaS service), `clickhouse-os` (self-hosted ClickHouse) and `hydrolix`.
 * `config` is a set of configuration options for the connector.
 ```yaml
 backendConnectors:
@@ -84,16 +89,25 @@ backendConnectors:
     config:
       user: "elastic"
       password: "change-me"
-      url: "http://elasticsearch:9200"
+      url: "http://192.168.0.7:9200"
   - name: my-clickhouse-data-source
     type: clickhouse-os
     config:
       user: "username"
       password: "username-is-password"
       database: "dbname"
-      url: "clickhouse://clickhouse:9000"
+      url: "clickhouse://192.168.0.9:9000"
 ```
-**WARNING:** When connecting to ClickHouse or Hydrolix, only the native protocol connection (`clickhouse://`) is supported.
+
+The supported configuration options for backend connectors (under `config`):
+* `url` - connection string to the backend service in a URL format (`protocol://host:port`):
+  * for Elastic/OpenSearch the expected format is `http://host:port` (Elastic/OpenSearch default port is 9200)
+  * for ClickHouse/Hydrolix the expected format is `clickhouse://host:port` (ClickHouse default port is 9000, ClickHouse/Hydrolix default encrypted port is 9440). Note that Quesma supports only the ClickHouse native protocol  (`clickhouse://`) and does not support the HTTP protocol.
+* `user` - username for authentication
+* `password` - password for authentication 
+* `database` - name of the database to connect to. It is optional for ClickHouse, but strictly required for Hydrolix, where it is also referred as "project".
+* `adminUrl` - URL for administrative operations to render a handy link in Quesma management UI (optional)
+* `disableTLS` - when set to true, disables TLS for the connection (optional)
 
 ### Processors
 
@@ -178,14 +192,33 @@ The configuration for an index consists of the following configuration options:
    will dual write ingest requests to `my_index` to both ElasticSearch and ClickHouse.
    Note that ElasticSearch/OpenSearch is the only supported backend for the `*` entry.
    If no targets are provided (example: `target: []`) in the configuration of an index in the ingest processor, ingest for that index will be disabled and incoming data will be dropped.
-- `override` (optional): override the name of table in Hydrolix/ClickHouse (by default Quesma uses the same table name as the index name)
-- `useCommonTable` (optional): if enabled, Quesma will store data in a single Hydrolix/ClickHouse table named `quesma_common_table`. See [ingest documentation](/ingest.md) for more details.
+   
+   Some backend connectors have additional attributes which may be used. For example the following configuration sets `useCommonTable` for `backend-clickhouse` target:
+   ```yaml
+      my_index:
+        target:
+          - backend-clickhouse:
+              useCommonTable: true
+   ```
+   Currently only the ClickHouse backend connector supports the following attributes:
+     - `useCommonTable` (optional): if enabled, Quesma will store data in a single Hydrolix/ClickHouse table named `quesma_common_table`. See [ingest documentation](/ingest.md) for more details.
+     - `tableName` (optional): override the name of table in Hydrolix/ClickHouse (by default Quesma uses the same table name as the index name)
+- `schemaOverrides` (optional): manual overrides of schema information for an index. Quesma infers schema for an index based on the data ingested and the schema information fetched from ClickHouse. `schemaOverrides` allows you to override this inferred schema with for some fields. For example the following configuration:
+    ```yaml
+      my_index:
+        target: [ backend-clickhouse ]
+        schemaOverrides:
+          "product_name":
+            type: "text"
+    ```
+    changes the type of `product_name` field to `text`. Note: `schemaOverrides` are currently not supported in `*` configuration.
 
 ## Optional configuration options
 
 ### Quesma licensing configuration
 
-In order to be able to use `hydrolix` or `clickhouse` backend connectors, one needs to supply `licenseKey` in the configuration file. Contact us at [email protected] if you need one.
+In order to be able to use `hydrolix` or `clickhouse` backend connectors Quesma requires a commercial license key (supplied in the `licenseKey` field of the configuration file). 
+Contact us at [email protected] if you need one.
 ```yaml
 licenseKey: ZXlKcGJuTjBZV3hz...
 ```  

docs/public/docs/example-1.md

@@ -32,7 +32,7 @@ flowchart LR
       - name: backend-elasticsearch
         type: elasticsearch
         config:
-          url: #PLACE_YOUR_ELASTICSEARCH_URL_HERE
+          url: #PLACE_YOUR_ELASTICSEARCH_URL_HERE, for example: http://192.168.0.7:9200
           user: #PLACE_YOUR_ELASTICSEARCH_USERNAME_HERE
           password: #PLACE_YOUR_ELASTICSEARCH_PASSWORD_HERE
     processors:

docs/public/docs/example-2-0-clickhouse-specific.md

@@ -55,14 +55,14 @@ flowchart LR
       - name: minimal-elasticsearch
         type: elasticsearch
         config:
-          url: #PLACE_YOUR_ELASTICSEARCH_URL_HERE
+          url: #PLACE_YOUR_ELASTICSEARCH_URL_HERE, for example: http://192.168.0.7:9200
           user: #PLACE_YOUR_ELASTICSEARCH_USERNAME_HERE
           password: #PLACE_YOUR_ELASTICSEARCH_PASSWORD_HERE
       - name: clickhouse-instance
         type: clickhouse-os
         #type: clickhouse # use for ClickHouse cloud service only
         config:
-          url: #PLACE_YOUR_CLICKHOUSE_URL_HERE
+          url: #PLACE_YOUR_CLICKHOUSE_URL_HERE, for example: clickhouse://192.168.0.7:9000
           user: #PLACE_YOUR_CLICKHOUSE_USER_HERE
           password: #PLACE_YOUR_CLICKHOUSE_PASSWORD_HERE
           database: #PLACE_YOUR_CLICKHOUSE_DATABASE_NAME_HERE

docs/public/docs/example-2-0.md

@@ -42,13 +42,13 @@ flowchart LR
       - name: minimal-elasticsearch
         type: elasticsearch
         config:
-          url: #PLACE_YOUR_ELASTICSEARCH_URL_HERE
+          url: #PLACE_YOUR_ELASTICSEARCH_URL_HERE, for example: http://192.168.0.7:9200
           user: #PLACE_YOUR_ELASTICSEARCH_USERNAME_HERE
           password: #PLACE_YOUR_ELASTICSEARCH_PASSWORD_HERE
       - name: hydrolix-instance
         type: hydrolix
         config:
-          url: #PLACE_YOUR_HYDROLIX_URL_HERE
+          url: #PLACE_YOUR_HYDROLIX_URL_HERE, for example: clickhouse://companyname.hydrolix.live:9440
           user: #PLACE_YOUR_HYDROLIX_USER_HERE
           password: #PLACE_YOUR_HYDROLIX_PASSWORD_HERE
           database: #PLACE_YOUR_HYDROLIX_DATABASE_NAME_HERE

docs/public/docs/example-2-1-hydro-specific.md

@@ -44,13 +44,13 @@ flowchart LR
       - name: elasticsearch-instance
         type: elasticsearch
         config:
-          url: #PLACE_YOUR_ELASTICSEARCH_URL_HERE
+          url: #PLACE_YOUR_ELASTICSEARCH_URL_HERE, for example: http://192.168.0.7:9200
           user: #PLACE_YOUR_ELASTICSEARCH_USERNAME_HERE
           password: #PLACE_YOUR_ELASTICSEARCH_PASSWORD_HERE
       - name: hydrolix-instance
         type: hydrolix
         config:
-          url: #PLACE_YOUR_HYDROLIX_URL_HERE
+          url: #PLACE_YOUR_HYDROLIX_URL_HERE, for example: clickhouse://companyname.hydrolix.live:9440
           user: #PLACE_YOUR_HYDROLIX_USER_HERE
           password: #PLACE_YOUR_HYDROLIX_PASSWORD_HERE
           database: #PLACE_YOUR_HYDROLIX_DATABASE_NAME_HERE

docs/public/docs/example-2-1.md

@@ -52,14 +52,14 @@ K[Kibana/OSD] --> Q((Quesma))
       - name: elasticsearch-instance
         type: elasticsearch
         config:
-          url: #PLACE_YOUR_ELASTICSEARCH_URL_HERE
+          url: #PLACE_YOUR_ELASTICSEARCH_URL_HERE, for example: http://192.168.0.7:9200
           user: #PLACE_YOUR_ELASTICSEARCH_USERNAME_HERE
           password: #PLACE_YOUR_ELASTICSEARCH_PASSWORD_HERE
       - name: clickhouse-instance
         type: clickhouse-os
         #type: clickhouse # use for ClickHouse cloud service only
         config:
-          url: #PLACE_YOUR_CLICKHOUSE_URL_HERE
+          url: #PLACE_YOUR_CLICKHOUSE_URL_HERE, for example: clickhouse://192.168.0.7:9000
           user: #PLACE_YOUR_CLICKHOUSE_USER_HERE
           password: #PLACE_YOUR_CLICKHOUSE_PASSWORD_HERE
           database: #PLACE_YOUR_CLICKHOUSE_DATABASE_NAME_HERE

docs/public/docs/index.md

@@ -6,9 +6,9 @@ hero:
   image:
     light: /logo/quesma-logo-black-full-svg.svg
     dark: /logo/quesma-logo-white-full-svg.svg
-  name: "Quesma Early Access Preview (EAP)"
+  name: "Quesma Documentation"
 #text: "Quesma EAP docs"
-  tagline: Welcome to the Quesma Early Access Preview! We're excited to have you on board and can't wait to hear your feedback! This website will help you get started.
+  tagline: Welcome to the Quesma Documentation! This website will help you get started.
   actions:
     - theme: brand
       text: Getting started