DOC-5600 Bring Pulsar connector docs to Astra Streaming

antora.yml

@@ -28,10 +28,19 @@ asciidoc:
     pulsar-reg: 'Apache Pulsar(TM)'
     pulsar-version: '3.1' #DO NOT INCLUDE PATCH VERSION <MAJOR>.<MINOR>.<PATCH>
     pulsar-version-patch: '3.1.0'
+
     starlight-kafka: 'Starlight for Kafka'
     starlight-rabbitmq: 'Starlight for RabbitMQ'
     gpt-schema-translator: 'GPT schema translator'
     astra-streaming-examples-repo: 'https://raw.githubusercontent.com/datastax/astra-streaming-examples/master'
+    scb: 'Secure Connect Bundle (SCB)'
+    scb-short: 'SCB'
+    scb-brief: 'Secure Connect Bundle'
+
+    # Required for include::common partials that are shared with Luna Streaming
+    web-ui: '{astra-ui}'
+    debezium-version: '1.7'
+    astra: 'Astra'
 
     # LEGACY ATTRIBUTES - CONSIDER REMOVING
     langstream: 'RAGStack'

modules/ROOT/nav.adoc

@@ -25,7 +25,29 @@
 * xref:developing:starlight.adoc[]
 * xref:developing:astream-cdc.adoc[]
 * xref:developing:gpt-schema-translator.adoc[]
-* xref:streaming-learning:pulsar-io:connectors/index.adoc[IO connectors reference]
+* I/O connectors
+** xref:connectors:index.adoc[]
+** Sinks
+*** xref:connectors:sinks/astra-db.adoc[]
+*** xref:connectors:sinks/cloud-storage.adoc[]
+*** xref:connectors:sinks/elastic-search.adoc[]
+*** xref:connectors:sinks/google-bigquery.adoc[]
+*** xref:connectors:sinks/jdbc-clickhouse.adoc[]
+*** xref:connectors:sinks/jdbc-mariadb.adoc[]
+*** xref:connectors:sinks/jdbc-postgres.adoc[]
+*** xref:connectors:sinks/jdbc-sqllite.adoc[]
+*** xref:connectors:sinks/kafka.adoc[]
+*** xref:connectors:sinks/kinesis.adoc[]
+*** xref:connectors:sinks/snowflake.adoc[]
+** Sources
+*** xref:connectors:sources/data-generator.adoc[]
+*** xref:connectors:sources/debezium-mongodb.adoc[]
+*** xref:connectors:sources/debezium-mysql.adoc[]
+*** xref:connectors:sources/debezium-oracle.adoc[]
+*** xref:connectors:sources/debezium-postgres.adoc[]
+*** xref:connectors:sources/debezium-sqlserver.adoc[]
+*** xref:connectors:sources/kafka.adoc[]
+*** xref:connectors:sources/kinesis.adoc[]
 
 .Operations
 * xref:operations:create-delete-tenants.adoc[]

modules/ROOT/pages/index.adoc

@@ -20,7 +20,7 @@
     Delivered as a fully-managed data streaming and event stream processing service with boundless scale, massive throughput, and low latency, {product} is the simplest way to modernize your event driven architecture and turbo charge your data in motion strategy.
     You can create {pulsar-short} instances, scale across cloud regions, and manage {pulsar-short} resources such as topics, connectors, functions and subscriptions.
 
-    xref:astra-streaming:getting-started:index.adoc[Get started with Astra Streaming]
+    xref:getting-started:index.adoc[Get started with Astra Streaming]
 
   </div>
 
@@ -32,7 +32,7 @@
 
     <h3 class="discrete !text-h2 !m-0 [&>a]:text-white">APIs and integrations</h3>
 
-    <p>{product} goes beyond the {astra-ui}. You can use xref:apis:index.adoc[{product} APIs], xref:developing:clients/index.adoc[{pulsar} clients], and related tools to manage your streaming tenants and resources, develop applications, and xref:streaming-learning:pulsar-io:connectors/index.adoc[integrate] with popular sinks and sources.</p>
+    <p>{product} goes beyond the {astra-ui}. You can use xref:apis:index.adoc[{product} APIs], xref:developing:clients/index.adoc[{pulsar} clients], and related tools to manage your streaming tenants and resources, develop applications, and xref:connectors:index.adoc[integrate] with popular sinks and sources.</p>
 
   </div>
 

modules/ROOT/partials/sinks/edit.adoc

@@ -0,0 +1,26 @@
+To update a connector, pass the new configuration definition to the connector.
+For example, if you created the connector with a configuration file, you can pass an updated configuration file.
+
+You can include the entire configuration or only the properties that you want to change.
+Additionally, some properties can be modified with specific arguments, such as `--parallelism`.
+
+To get the current configuration, see xref:apis:api-operations.adoc#get-sink-connector-configuration-data[Get sink connector configuration data].
+
+.pulsar-admin CLI
+[source,shell,subs="+attributes"]
+----
+./bin/pulsar-admin sinks update \
+  --sink-type {connectorType} \
+  --name "$SINK_NAME" \
+  --inputs "persistent://$TENANT/$NAMESPACE/$TOPIC" \
+  --tenant "$TENANT" \
+  --parallelism 2
+----
+
+.{pulsar-short} Admin API
+[source,shell]
+----
+curl -sS --fail -L -X PUT "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$SINK_NAME" \
+  --header "Authorization: Bearer $PULSAR_TOKEN" \
+  --form "sinkConfig=@configs.json;type=application/json"
+----

modules/ROOT/partials/sinks/get-started.adoc

@@ -0,0 +1,43 @@
+. Optional: If you are using the `pulsar-admin` CLI or {pulsar-short} Admin API, set the following commonly-used environment variables:
++
+[source,shell,subs="+quotes"]
+----
+export TENANT="**TENANT_NAME**"
+export TOPIC="**INPUT_TOPIC_NAME**"
+export NAMESPACE="**NAMESPACE_NAME**" # or default
+export SINK_NAME="**SINK_CONNECTOR_UNIQUE_NAME**"
+export PULSAR_TOKEN="**TENANT_PULSAR_TOKEN**" # API only
+export WEB_SERVICE_URL="**TENANT_PULSAR_WEB_SERVICE_URL**" # API only
+----
++
+`**SINK_NAME**` is the name for your new sink connector.
+{company} recommends a memorable, human-readable name that summarizes the connector's purpose.
+For example: `{connectorType}-sink-prod-us-east-1`.
+
+. Create the connector using JSON-formatted connector configuration data.
+You can pass the configuration directly or with a configuration file.
++
+.pulsar-admin CLI
+[source,shell,subs="+attributes"]
+----
+./bin/pulsar-admin sinks create \
+  --sink-type {connectorType} \
+  --name "$SINK_NAME" \
+  --inputs "persistent://$TENANT/$NAMESPACE/$TOPIC" \
+  --tenant "$TENANT" \
+  --sink-config-file configs.json
+----
++
+.{pulsar-short} Admin API
+[source,shell]
+----
+curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$SINK_NAME" \
+  --header "Authorization: Bearer $PULSAR_TOKEN" \
+  --form "sinkConfig=@configs.json;type=application/json"
+----
++
+.Example configuration data structure
+[source,json]
+----
+include::common:streaming:example$connectors/sinks/{connectorType}/sample-data.json[]
+----

modules/ROOT/partials/sources/edit.adoc

@@ -0,0 +1,26 @@
+To update a connector, pass the new configuration definition to the connector.
+For example, if you created the connector with a configuration file, you can pass an updated configuration file.
+
+You can include the entire configuration or only the properties that you want to change.
+Additionally, some properties can be modified with specific arguments, such as `--parallelism`.
+
+To get the current configuration, see xref:apis:api-operations.adoc#get-source-connector-configuration-data[Get source connector configuration data].
+
+.pulsar-admin CLI
+[source,shell,subs="+attributes"]
+----
+./bin/pulsar-admin sources update \
+  --source-type {connectorType} \
+  --name "$SOURCE_NAME" \
+  --destination-topic-name "persistent://$TENANT/$NAMESPACE/$TOPIC" \
+  --tenant "$TENANT" \
+  --parallelism 2
+----
+
+.{pulsar-short} Admin API
+[source,shell]
+----
+curl -sS --fail -L -X PUT "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE/$SOURCE_NAME" \
+  --header "Authorization: Bearer $PULSAR_TOKEN" \
+  --form "sourceConfig=@mynetty-source-config.json;type=application/json"
+----

modules/ROOT/partials/sources/get-started.adoc

@@ -0,0 +1,43 @@
+. Optional: If you are using the `pulsar-admin` CLI or {pulsar-short} Admin API, set the following commonly-used environment variables:
++
+[source,shell,subs="+quotes"]
+----
+export TENANT="**TENANT_NAME**"
+export TOPIC="**OUTPUT_TOPIC_NAME**"
+export NAMESPACE="**NAMESPACE_NAME**" # or default
+export SOURCE_NAME="**SOURCE_CONNECTOR_UNIQUE_NAME**"
+export PULSAR_TOKEN="**TENANT_PULSAR_TOKEN**" # API only
+export WEB_SERVICE_URL="**TENANT_PULSAR_WEB_SERVICE_URL**" # API only
+----
++
+`**SOURCE_NAME**` is the name for your new source connector.
+{company} recommends a memorable, human-readable name that summarizes the connector's purpose.
+For example: `{connectorType}-source-prod-us-east-1`.
+
+. Create the connector using JSON-formatted connector configuration data.
+You can pass the configuration directly or with a configuration file.
++
+.pulsar-admin CLI
+[source,shell,subs="+attributes"]
+----
+./bin/pulsar-admin sources create \
+  --source-type {connectorType} \
+  --name "$SOURCE_NAME" \
+  --destination-topic-name "persistent://$TENANT/$NAMESPACE/$TOPIC" \
+  --tenant "$TENANT" \
+  --source-config-file configs.json
+----
++
+.{pulsar-short} Admin API
+[source,shell]
+----
+curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE/$SOURCE_NAME" \
+  --header "Authorization: Bearer $PULSAR_TOKEN" \
+  --form "sourceConfig=@mynetty-source-config.json;type=application/json"
+----
++
+.Example configuration data structure
+[source,json]
+----
+include::common:streaming:example$connectors/sources/{connectorType}/sample-data.json[]
+----