Skip to content

Commit 75b61ce

Browse files
aimurphyaimurphy
committed
Merge branch 'main' into doc-5189
2 parents 1d9d895 + 2604afe commit 75b61ce

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

50 files changed

+1713
-408
lines changed

‎antora.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,19 @@ asciidoc:
2828
pulsar-reg: 'Apache Pulsar(TM)'
2929
pulsar-version: '3.1' #DO NOT INCLUDE PATCH VERSION <MAJOR>.<MINOR>.<PATCH>
3030
pulsar-version-patch: '3.1.0'
31+
3132
starlight-kafka: 'Starlight for Kafka'
3233
starlight-rabbitmq: 'Starlight for RabbitMQ'
3334
gpt-schema-translator: 'GPT schema translator'
3435
astra-streaming-examples-repo: 'https://raw.githubusercontent.com/datastax/astra-streaming-examples/master'
36+
scb: 'Secure Connect Bundle (SCB)'
37+
scb-short: 'SCB'
38+
scb-brief: 'Secure Connect Bundle'
39+
40+
# Required for include::common partials that are shared with Luna Streaming
41+
web-ui: '{astra-ui}'
42+
debezium-version: '1.7'
43+
astra: 'Astra'
3544

3645
# LEGACY ATTRIBUTES - CONSIDER REMOVING
3746
langstream: 'RAGStack'

‎modules/ROOT/nav.adoc‎

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,35 @@
1919
** xref:ROOT:astream-subscriptions-shared.adoc[]
2020
** xref:ROOT:astream-subscriptions-failover.adoc[]
2121
** xref:ROOT:astream-subscriptions-keyshared.adoc[]
22-
* xref:developing:astream-functions.adoc[]
22+
* {pulsar-short} functions
23+
** xref:developing:astream-functions.adoc[]
24+
** xref:developing:astream-functions-transform.adoc[]
2325
* xref:developing:starlight.adoc[]
2426
* xref:developing:astream-cdc.adoc[]
2527
* xref:developing:gpt-schema-translator.adoc[]
26-
* xref:streaming-learning:pulsar-io:connectors/index.adoc[IO connectors reference]
28+
* I/O connectors
29+
** xref:connectors:index.adoc[]
30+
** Sinks
31+
*** xref:connectors:sinks/astra-db.adoc[]
32+
*** xref:connectors:sinks/cloud-storage.adoc[]
33+
*** xref:connectors:sinks/elastic-search.adoc[]
34+
*** xref:connectors:sinks/google-bigquery.adoc[]
35+
*** xref:connectors:sinks/jdbc-clickhouse.adoc[]
36+
*** xref:connectors:sinks/jdbc-mariadb.adoc[]
37+
*** xref:connectors:sinks/jdbc-postgres.adoc[]
38+
*** xref:connectors:sinks/jdbc-sqllite.adoc[]
39+
*** xref:connectors:sinks/kafka.adoc[]
40+
*** xref:connectors:sinks/kinesis.adoc[]
41+
*** xref:connectors:sinks/snowflake.adoc[]
42+
** Sources
43+
*** xref:connectors:sources/data-generator.adoc[]
44+
*** xref:connectors:sources/debezium-mongodb.adoc[]
45+
*** xref:connectors:sources/debezium-mysql.adoc[]
46+
*** xref:connectors:sources/debezium-oracle.adoc[]
47+
*** xref:connectors:sources/debezium-postgres.adoc[]
48+
*** xref:connectors:sources/debezium-sqlserver.adoc[]
49+
*** xref:connectors:sources/kafka.adoc[]
50+
*** xref:connectors:sources/kinesis.adoc[]
2751
2852
.Operations
2953
* xref:operations:create-delete-tenants.adoc[]

‎modules/ROOT/pages/index.adoc‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@
2020
Delivered as a fully-managed data streaming and event stream processing service with boundless scale, massive throughput, and low latency, {product} is the simplest way to modernize your event driven architecture and turbo charge your data in motion strategy.
2121
You can create {pulsar-short} instances, scale across cloud regions, and manage {pulsar-short} resources such as topics, connectors, functions and subscriptions.
2222
23-
xref:astra-streaming:getting-started:index.adoc[Get started with Astra Streaming]
23+
xref:getting-started:index.adoc[Get started with Astra Streaming]
2424
2525
</div>
2626
@@ -32,7 +32,7 @@
3232
3333
<h3 class="discrete !text-h2 !m-0 [&>a]:text-white">APIs and integrations</h3>
3434
35-
<p>{product} goes beyond the {astra-ui}. You can use xref:apis:index.adoc[{product} APIs], xref:developing:clients/index.adoc[{pulsar} clients], and related tools to manage your streaming tenants and resources, develop applications, and xref:streaming-learning:pulsar-io:connectors/index.adoc[integrate] with popular sinks and sources.</p>
35+
<p>{product} goes beyond the {astra-ui}. You can use xref:apis:index.adoc[{product} APIs], xref:developing:clients/index.adoc[{pulsar} clients], and related tools to manage your streaming tenants and resources, develop applications, and xref:connectors:index.adoc[integrate] with popular sinks and sources.</p>
3636
3737
</div>
3838

‎modules/ROOT/partials/functions-ui-common-steps-pt1.adoc‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
. In the {astra-ui-link} header, click icon:grip[name="Applications"], and then select *Streaming*.
2+
3+
. Click the name of the tenant where you want to deploy a function.
4+
5+
. On the *Functions* tab, click *Create Function*.
6+
7+
. Enter a function name.

‎modules/ROOT/partials/functions-ui-common-steps-pt2.adoc‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
. Select the namespace and one or more input topics for the function.
2+
3+
. If needed, select an output namespace and topics.
4+
+
5+
Whether you need to specify an output topic depends on the function's logic.
6+
For example, the `cast` xref:developing:astream-functions-transform.adoc[transform function] needs an output topic to receive the transformed messages.
7+
8+
. Optional: Select a log topic.
9+
+
10+
The log topic is a separate output topic for messages containing additional `loglevel`, `fqn`, and `instance` properties.
11+
12+
. Optional: Configure advanced settings:
13+
+
14+
* *Instances*: Enter a number of sink instances to run.
15+
* *Processing Guarantee*: Select one of the following:
16+
** *ATLEAST_ONCE* (default): Each message sent to the function can be processed one or more times.
17+
** *ATMOST_ONCE*: Each message sent to the function is processed 0 or 1 times.
18+
This means there is a change that a message is not processed.
19+
** *EFFECTIVELY_ONCE*: Each message sent to the function has only one output associated with it.
20+
* *Timeout*: Set a timeout limit.
21+
* *Auto Acknowledge*: Enable or disable automatic message acknowledgment.
22+
23+
. If needed, provide a JSON-formatted config key for your function.
24+
This configuration depends on your function code.
25+
26+
. Click *Create*.
27+
28+
. Confirm your function was created on the *Functions* tab.
29+
+
30+
You can also verify the function deployment with the {pulsar-short} CLI:
31+
+
32+
[source,shell]
33+
----
34+
./bin/pulsar-admin functions list --tenant $TENANT
35+
----
36+
37+
Once created, the function starts processing messages from the input topics according to its logic.

‎modules/ROOT/partials/sinks/edit.adoc‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
To update a connector, pass the new configuration definition to the connector.
2+
For example, if you created the connector with a configuration file, you can pass an updated configuration file.
3+
4+
You can include the entire configuration or only the properties that you want to change.
5+
Additionally, some properties can be modified with specific arguments, such as `--parallelism`.
6+
7+
To get the current configuration, see xref:apis:api-operations.adoc#get-sink-connector-configuration-data[Get sink connector configuration data].
8+
9+
.pulsar-admin CLI
10+
[source,shell,subs="+attributes"]
11+
----
12+
./bin/pulsar-admin sinks update \
13+
--sink-type {connectorType} \
14+
--name "$SINK_NAME" \
15+
--inputs "persistent://$TENANT/$NAMESPACE/$TOPIC" \
16+
--tenant "$TENANT" \
17+
--parallelism 2
18+
----
19+
20+
.{pulsar-short} Admin API
21+
[source,shell]
22+
----
23+
curl -sS --fail -L -X PUT "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$SINK_NAME" \
24+
--header "Authorization: Bearer $PULSAR_TOKEN" \
25+
--form "sinkConfig=@configs.json;type=application/json"
26+
----

‎modules/ROOT/partials/sinks/get-started.adoc‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
. Optional: If you are using the `pulsar-admin` CLI or {pulsar-short} Admin API, set the following commonly-used environment variables:
2+
+
3+
[source,shell,subs="+quotes"]
4+
----
5+
export TENANT="**TENANT_NAME**"
6+
export TOPIC="**INPUT_TOPIC_NAME**"
7+
export NAMESPACE="**NAMESPACE_NAME**" # or default
8+
export SINK_NAME="**SINK_CONNECTOR_UNIQUE_NAME**"
9+
export PULSAR_TOKEN="**TENANT_PULSAR_TOKEN**" # API only
10+
export WEB_SERVICE_URL="**TENANT_PULSAR_WEB_SERVICE_URL**" # API only
11+
----
12+
+
13+
`**SINK_NAME**` is the name for your new sink connector.
14+
{company} recommends a memorable, human-readable name that summarizes the connector's purpose.
15+
For example: `{connectorType}-sink-prod-us-east-1`.
16+
17+
. Create the connector using JSON-formatted connector configuration data.
18+
You can pass the configuration directly or with a configuration file.
19+
+
20+
.pulsar-admin CLI
21+
[source,shell,subs="+attributes"]
22+
----
23+
./bin/pulsar-admin sinks create \
24+
--sink-type {connectorType} \
25+
--name "$SINK_NAME" \
26+
--inputs "persistent://$TENANT/$NAMESPACE/$TOPIC" \
27+
--tenant "$TENANT" \
28+
--sink-config-file configs.json
29+
----
30+
+
31+
.{pulsar-short} Admin API
32+
[source,shell]
33+
----
34+
curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$SINK_NAME" \
35+
--header "Authorization: Bearer $PULSAR_TOKEN" \
36+
--form "sinkConfig=@configs.json;type=application/json"
37+
----
38+
+
39+
.Example configuration data structure
40+
[source,json]
41+
----
42+
include::common:streaming:example$connectors/sinks/{connectorType}/sample-data.json[]
43+
----

‎modules/ROOT/partials/sources/edit.adoc‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
To update a connector, pass the new configuration definition to the connector.
2+
For example, if you created the connector with a configuration file, you can pass an updated configuration file.
3+
4+
You can include the entire configuration or only the properties that you want to change.
5+
Additionally, some properties can be modified with specific arguments, such as `--parallelism`.
6+
7+
To get the current configuration, see xref:apis:api-operations.adoc#get-source-connector-configuration-data[Get source connector configuration data].
8+
9+
.pulsar-admin CLI
10+
[source,shell,subs="+attributes"]
11+
----
12+
./bin/pulsar-admin sources update \
13+
--source-type {connectorType} \
14+
--name "$SOURCE_NAME" \
15+
--destination-topic-name "persistent://$TENANT/$NAMESPACE/$TOPIC" \
16+
--tenant "$TENANT" \
17+
--parallelism 2
18+
----
19+
20+
.{pulsar-short} Admin API
21+
[source,shell]
22+
----
23+
curl -sS --fail -L -X PUT "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE/$SOURCE_NAME" \
24+
--header "Authorization: Bearer $PULSAR_TOKEN" \
25+
--form "sourceConfig=@mynetty-source-config.json;type=application/json"
26+
----

‎modules/ROOT/partials/sources/get-started.adoc‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
. Optional: If you are using the `pulsar-admin` CLI or {pulsar-short} Admin API, set the following commonly-used environment variables:
2+
+
3+
[source,shell,subs="+quotes"]
4+
----
5+
export TENANT="**TENANT_NAME**"
6+
export TOPIC="**OUTPUT_TOPIC_NAME**"
7+
export NAMESPACE="**NAMESPACE_NAME**" # or default
8+
export SOURCE_NAME="**SOURCE_CONNECTOR_UNIQUE_NAME**"
9+
export PULSAR_TOKEN="**TENANT_PULSAR_TOKEN**" # API only
10+
export WEB_SERVICE_URL="**TENANT_PULSAR_WEB_SERVICE_URL**" # API only
11+
----
12+
+
13+
`**SOURCE_NAME**` is the name for your new source connector.
14+
{company} recommends a memorable, human-readable name that summarizes the connector's purpose.
15+
For example: `{connectorType}-source-prod-us-east-1`.
16+
17+
. Create the connector using JSON-formatted connector configuration data.
18+
You can pass the configuration directly or with a configuration file.
19+
+
20+
.pulsar-admin CLI
21+
[source,shell,subs="+attributes"]
22+
----
23+
./bin/pulsar-admin sources create \
24+
--source-type {connectorType} \
25+
--name "$SOURCE_NAME" \
26+
--destination-topic-name "persistent://$TENANT/$NAMESPACE/$TOPIC" \
27+
--tenant "$TENANT" \
28+
--source-config-file configs.json
29+
----
30+
+
31+
.{pulsar-short} Admin API
32+
[source,shell]
33+
----
34+
curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE/$SOURCE_NAME" \
35+
--header "Authorization: Bearer $PULSAR_TOKEN" \
36+
--form "sourceConfig=@mynetty-source-config.json;type=application/json"
37+
----
38+
+
39+
.Example configuration data structure
40+
[source,json]
41+
----
42+
include::common:streaming:example$connectors/sources/{connectorType}/sample-data.json[]
43+
----

0 commit comments

Comments
 (0)