Skip to content

DOC-4944 DOC-4945 Pulsar attributes, attribute cleanup #127

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jan 24, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 11 additions & 15 deletions antora.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,24 +12,20 @@ nav:

asciidoc:
attributes:
pulsar_version: '2.10' #DO NOT INCLUDE PATCH VERSION <MAJOR>.<MINOR>.<PATCH>
pulsar_full_version: '2.10.1'
product: 'Astra Streaming'
pulsar: 'Apache Pulsar'
pulsar-short: 'Pulsar'
pulsar-reg: 'Apache Pulsar(TM)'
pulsar-version: '2.10' #DO NOT INCLUDE PATCH VERSION <MAJOR>.<MINOR>.<PATCH>
pulsar-version-patch: '2.10.1'
kafka_for_astra: 'Starlight for Kafka'
starlight_rabbitmq: 'Starlight for RabbitMQ'
gpt-schema-translator: 'GPT schema translator'
base_url: 'https://api.astra.datastax.com'
base_api: '/v2/streaming'
bearer_token: '$BEARER_TOKEN'
tenant_name_with_topics: 'llp-test'
cluster_name: 'astradev-aws'
tenant_name: 'testcreate'
topic_name: 'blah'
request_feature_access_email: '[email protected]'
streaming_signup_url: 'https://astra.datastax.com/streaming'
login_url: 'https://astra.datastax.com'
langstream: 'RAGStack'
ragstack: 'RAGStack'
db-serverless: 'Astra DB Serverless'
url-astra: 'https://astra.datastax.com'
link-astra-portal: '{url-astra}[{astra_ui}^]'
astra-streaming-examples-repo: 'https://raw.githubusercontent.com/datastax/astra-streaming-examples/master'

# LEGACY ATTRIBUTES - CONSIDER REMOVING
langstream: 'RAGStack'
ragstack: 'RAGStack'
streaming_signup_url: 'https://astra.datastax.com/streaming'
4 changes: 2 additions & 2 deletions modules/ROOT/examples/check_tenant_exists.sh
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
curl --location --head '{base_url}{base_api}/tenants/test-llp' \
--header "Authorization: Bearer {bearer_token}"
curl --location --head 'https://api.astra.datastax.com/v2/streaming/tenants/test-llp' \
--header "Authorization: Bearer $BEARER_TOKEN"
6 changes: 3 additions & 3 deletions modules/ROOT/examples/create_tenant.sh
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
curl --location --request POST '{base_url}{base_api}/tenants?topic={topic_name}' \
curl --location --request POST 'https://api.astra.datastax.com/v2/streaming/tenants?topic=blah' \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer {bearer_token}" \
--header "Authorization: Bearer $BEARER_TOKEN" \
--data-raw '{
"cloudProvider": "aws",
"cloudRegion": "useast2",
"tenantName": "{tenant_name}",
"tenantName": "testcreate",
"userEmail": "[email protected]"
}'
2 changes: 1 addition & 1 deletion modules/ROOT/examples/curl_deflate.sh
Original file line number Diff line number Diff line change
@@ -1 +1 @@
curl -v -H 'Authorization: Bearer {bearer_token}' -H 'Accept-encoding: deflate' https://prometheus-aws-useast1.streaming.datastax.com/pulsarmetrics/tenant-1 --compressed
curl -v -H 'Authorization: Bearer $BEARER_TOKEN' -H 'Accept-encoding: deflate' https://prometheus-aws-useast1.streaming.datastax.com/pulsarmetrics/tenant-1 --compressed
2 changes: 1 addition & 1 deletion modules/ROOT/examples/curl_gzip.sh
Original file line number Diff line number Diff line change
@@ -1 +1 @@
curl -v -H 'Authorization: Bearer {bearer_token}' -H 'Accept-encoding: gzip' https://prometheus-aws-useast1.streaming.datastax.com/pulsarmetrics/tenant-1 | zcat
curl -v -H 'Authorization: Bearer $BEARER_TOKEN' -H 'Accept-encoding: gzip' https://prometheus-aws-useast1.streaming.datastax.com/pulsarmetrics/tenant-1 | zcat
4 changes: 2 additions & 2 deletions modules/ROOT/examples/delete_tenant_hard.sh
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
curl --location --request DELETE '{base_url}{base_api}/tenants/{tenant_name}/clusters/{cluster_name}' \
--header "Authorization: Bearer {bearer_token}"
curl --location --request DELETE 'https://api.astra.datastax.com/v2/streaming/tenants/testcreate/clusters/astradev-aws' \
--header "Authorization: Bearer $BEARER_TOKEN"
4 changes: 2 additions & 2 deletions modules/ROOT/examples/delete_tenant_soft.sh
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
curl --location --request DELETE '{base_url}{base_api}/tenants/{tenant_name}/clusters/{cluster_name}?opt={cluster_name}' \
--header "Authorization: Bearer {bearer_token}"
curl --location --request DELETE 'https://api.astra.datastax.com/v2/streaming/tenants/testcreate/clusters/astradev-aws?opt=astradev-aws' \
--header "Authorization: Bearer $BEARER_TOKEN"
4 changes: 2 additions & 2 deletions modules/ROOT/examples/get_clusters.sh
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
curl --location --request GET '{base_url}{base_api}/clusters' \
--header "Authorization: Bearer {bearer_token}"
curl --location --request GET 'https://api.astra.datastax.com/v2/streaming/clusters' \
--header "Authorization: Bearer $BEARER_TOKEN"
4 changes: 2 additions & 2 deletions modules/ROOT/examples/get_providers.sh
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
curl --location --request GET '{base_url}{base_api}/providers' \
--header "Authorization: Bearer {bearer_token}"
curl --location --request GET 'https://api.astra.datastax.com/v2/streaming/providers' \
--header "Authorization: Bearer $BEARER_TOKEN"
6 changes: 3 additions & 3 deletions modules/ROOT/examples/get_tenant_limits.sh
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
curl --location --request GET '{base_url}{base_api}/tenants/{tenant_name_with_topics}/limits' \
--header 'X-DataStax-Pulsar-Cluster: {cluster_name}' \
--header "Authorization: Bearer {bearer_token}"
curl --location --request GET 'https://api.astra.datastax.com/v2/streaming/tenants/llp-test/limits' \
--header 'X-DataStax-Pulsar-Cluster: astradev-aws' \
--header "Authorization: Bearer $BEARER_TOKEN"
4 changes: 2 additions & 2 deletions modules/ROOT/examples/get_tenants.sh
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
curl --location --request GET '{base_url}{base_api}/tenants' \
--header "Authorization: Bearer {bearer_token}"
curl --location --request GET 'https://api.astra.datastax.com/v2/streaming/tenants' \
--header "Authorization: Bearer $BEARER_TOKEN"
2 changes: 1 addition & 1 deletion modules/ROOT/examples/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ for FILE in *;
do
if [[ "$FILE" != "test"* ]]
then
gsed "s#{base_url}#$base_url#; s#{base_api}#$base_api#; s#{tenant_name}#$tenant_name#; s#{tenant_name_with_topics}#$tenant_name_with_topics#; s#{topic_name}#$topic_name#; s#{cluster_name}#$cluster_name#; s#{bearer_token}#$BEARER_TOKEN#;" $FILE > $FILE.tmp;
gsed "s#https://api.astra.datastax.com#$base_url#; s#/v2/streaming#$base_api#; s#testcreate#$tenant_name#; s#llp-test#$tenant_name_with_topics#; s#blah#$topic_name#; s#astradev-aws#$cluster_name#; s#$BEARER_TOKEN#$BEARER_TOKEN#;" $FILE > $FILE.tmp;
chmod 755 $FILE.tmp;
fi
done
Expand Down
8 changes: 4 additions & 4 deletions modules/ROOT/pages/astream-faq.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ See xref:getting-started:index.adoc[].

See xref:operations:astream-pricing.adoc[].

== Why is {product} based on Apache Pulsar?
== Why is {product} based on {pulsar-reg}?

For information about the decision to use Apache Pulsar, see https://www.datastax.com/blog/four-reasons-why-apache-pulsar-essential-modern-data-stack[Four Reasons Why Apache Pulsar is Essential to the Modern Data Stack].
For information about the decision to use {pulsar}, see https://www.datastax.com/blog/four-reasons-why-apache-pulsar-essential-modern-data-stack[Four Reasons Why {pulsar} is Essential to the Modern Data Stack].

== What happened to Kesque?

Expand All @@ -26,11 +26,11 @@ With the launch of {product}, {company} began shutting down the Kesque service a

The pricing model also makes {product} accessible to mid market and small/medium business customers who need event stream processing capabilities to run core parts of their business.

And finally {product} offers a user-friendly interface and free tier to satisfy the needs of individual developers and technologists who want to learn more about Apache Pulsar and data streaming in general.
And finally {product} offers a user-friendly interface and free tier to satisfy the needs of individual developers and technologists who want to learn more about {pulsar} and data streaming in general.

== What is CDC for {astra_db}?

Change Data Capture (CDC) for {astra_db} uses a Pulsar IO source connector that processes changes from the Cassandra Change Agent through a Pulsar topic.
Change Data Capture (CDC) for {astra_db} uses a {pulsar-short} IO source connector that processes changes from the Cassandra Change Agent through a {pulsar-short} topic.
For more information, see xref:developing:astream-cdc.adoc[].

== What are tenants, namespaces, topics, and sinks?
Expand Down
16 changes: 8 additions & 8 deletions modules/ROOT/pages/astream-org-permissions.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -36,22 +36,22 @@ Additional permissions might be required, depending on the tasks the role needs
To control access to specific streaming tenants, you can set granular xref:astra-db-serverless:administration:manage-database-access.adoc#role-scopes[resource scopes] on custom roles.
====

== Authentication and authorization in Pulsar and {astra_db}
== Authentication and authorization in {pulsar-reg} and {astra_db}

Pulsar has the concept of https://pulsar.apache.org/docs/security-authorization/[clients with role tokens].
In Pulsar, authentication is the process of verifying a provided token (JWT), and authorization is the process of determining if the role claimed in that token is allowed to complete the requested action.
{pulsar-short} has the concept of https://pulsar.apache.org/docs/security-authorization/[clients with role tokens].
In {pulsar-short}, authentication is the process of verifying a provided token (JWT), and authorization is the process of determining if the role claimed in that token is allowed to complete the requested action.

{product} uses the {company} version of Apache Pulsar (xref:luna-streaming::index.adoc[Luna Streaming]).
The https://github.com/datastax/pulsar[Luna project] is an open fork of the Pulsar project that maintains feature parity with OSS Pulsar. {product}, as a managed service, abstracts some features/options of Pulsar to ensure continuous, reliable service.
{product} uses the {company} version of {pulsar} (xref:luna-streaming::index.adoc[Luna Streaming]).
The https://github.com/datastax/pulsar[Luna project] is an open fork of the {pulsar-short} project that maintains feature parity with OSS {pulsar-short}. {product}, as a managed service, abstracts some features/options of {pulsar-short} to ensure continuous, reliable service.

On a shared cluster, your {astra_db} organization has one or more tenants on a shared Pulsar cluster.
Each of your tenants is secured by Pulsar authentication and authorization models, as well as your {astra_db} organization's authentication and authorization ({astra_db} RBAC).
On a shared cluster, your {astra_db} organization has one or more tenants on a shared {pulsar-short} cluster.
Each of your tenants is secured by {pulsar-short} authentication and authorization models, as well as your {astra_db} organization's authentication and authorization ({astra_db} RBAC).

{product} shared clusters are created and administered by {product} administrators.
Each tenant is assigned a custom role and permissions limited to that tenant only.
All tokens created within a tenant are assigned roles similar to the assigning tenant.

For programmatic access, you use {astra_db} application tokens or Pulsar JWT, depending on the operation you need to perform.
For programmatic access, you use {astra_db} application tokens or {pulsar-short} JWT, depending on the operation you need to perform.
For more information, see xref:operations:astream-token-gen.adoc[].

== See also
Expand Down
18 changes: 9 additions & 9 deletions modules/ROOT/pages/astream-subscriptions-exclusive.adoc
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
= Exclusive subscriptions in Pulsar
= Exclusive subscriptions in {pulsar-reg}
:navtitle: Exclusive
:page-tag: pulsar-subscriptions,quickstart,admin,dev,pulsar

_Subscriptions_ in Pulsar describe which consumers are consuming data from a topic and how they want to consume that data.
_Subscriptions_ in {pulsar-reg} describe which consumers are consuming data from a topic and how they want to consume that data.

An _exclusive subscription_ describes a basic publish-subscribe (pub-sub) pattern where a single consumer subscribes to a single topic and consumes from it.

This page explains how to use Pulsar's exclusive subscription model to manage your topic consumption.
This page explains how to use {pulsar-short}'s exclusive subscription model to manage your topic consumption.

include::ROOT:partial$subscription-prereq.adoc[]

[#example]
== Exclusive subscription example

. To configure a Pulsar exclusive subscription, define a `pulsarConsumer` object in `SimplePulsarConsumer.java`, as you would for other subscription types.
. To configure a {pulsar-short} exclusive subscription, define a `pulsarConsumer` object in `SimplePulsarConsumer.java`, as you would for other subscription types.
However, you don't need to declare a `subscriptionType`.
Whereas other subscription types required you to declare a specific `subscriptionType`, Pulsar creates an exclusive subscription by default if you don't declare a `subscriptionType`.
Whereas other subscription types required you to declare a specific `subscriptionType`, {pulsar-short} creates an exclusive subscription by default if you don't declare a `subscriptionType`.
+
.SimplePulsarConsumer.java
[source,java]
Expand Down Expand Up @@ -82,7 +82,7 @@ Caused by: org.apache.pulsar.client.api.PulsarClientException$ConsumerBusyExcept

== See also

* xref:astream-subscriptions.adoc[Subscriptions in Pulsar]
* xref:astream-subscriptions-shared.adoc[Shared subscriptions]
* xref:astream-subscriptions-failover.adoc[Failover subscriptions]
* xref:astream-subscriptions-keyshared.adoc[Key shared subscriptions]
* xref:astream-subscriptions.adoc[]
* xref:astream-subscriptions-shared.adoc[]
* xref:astream-subscriptions-failover.adoc[]
* xref:astream-subscriptions-keyshared.adoc[]
20 changes: 10 additions & 10 deletions modules/ROOT/pages/astream-subscriptions-failover.adoc
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
= Failover subscriptions in Pulsar
= Failover subscriptions in {pulsar-reg}
:navtitle: Failover
:page-tag: pulsar-subscriptions,quickstart,admin,dev,pulsar

_Subscriptions_ in Pulsar describe which consumers are consuming data from a topic and how they want to consume that data.
_Subscriptions_ in {pulsar-reg} describe which consumers are consuming data from a topic and how they want to consume that data.

If an xref:astream-subscriptions-exclusive.adoc[exclusive] consumer fails, the topic backlog accumulates stale, unacknowledged messages.
Pulsar solves this problem with failover subscriptions.
In _failover subscriptions_, Pulsar designates one primary consumer and multiple standby consumers.
{pulsar-short} solves this problem with failover subscriptions.
In _failover subscriptions_, {pulsar-short} designates one primary consumer and multiple standby consumers.
If the primary consumer disconnects, the standby consumers begin consuming the subsequent unacknowledged messages.

This page explains how to use Pulsar's failover subscription model to manage your topic consumption.
This page explains how to use {pulsar-short}'s failover subscription model to manage your topic consumption.

include::ROOT:partial$subscription-prereq.adoc[]

[#example]
== Failover subscription example

To try out a Pulsar failover subscription, add `.subscriptionType(SubscriptionType.Failover)` to the `pulsarConsumer` in `SimplePulsarConsumer.java`:
To try out a {pulsar-short} failover subscription, add `.subscriptionType(SubscriptionType.Failover)` to the `pulsarConsumer` in `SimplePulsarConsumer.java`:

.SimplePulsarConsumer.java
[source,java]
Expand Down Expand Up @@ -86,7 +86,7 @@ To test them, you can progressively end each `SimplePulsarConsumer` process, and

== See also

* xref:astream-subscriptions.adoc[Subscriptions in Pulsar]
* xref:astream-subscriptions-exclusive.adoc[Exclusive subscriptions]
* xref:astream-subscriptions-shared.adoc[Shared subscriptions]
* xref:astream-subscriptions-keyshared.adoc[Key shared subscriptions]
* xref:astream-subscriptions.adoc[]
* xref:astream-subscriptions-exclusive.adoc[]
* xref:astream-subscriptions-shared.adoc[]
* xref:astream-subscriptions-keyshared.adoc[]
18 changes: 9 additions & 9 deletions modules/ROOT/pages/astream-subscriptions-keyshared.adoc
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
= Key shared subscriptions in Pulsar
= Key shared subscriptions in {pulsar-reg}
:navtitle: Key shared
:page-tag: pulsar-subscriptions,quickstart,admin,dev,pulsar

_Subscriptions_ in Pulsar describe which consumers are consuming data from a topic and how they want to consume that data.
_Subscriptions_ in {pulsar-reg} describe which consumers are consuming data from a topic and how they want to consume that data.

Pulsar's xref:astream-subscriptions-shared.adoc[shared subscription] model can increase the message processing rate, but it risks losing message ordering guarantees.
{pulsar-short}'s xref:astream-subscriptions-shared.adoc[shared subscription] model can increase the message processing rate, but it risks losing message ordering guarantees.
In a round-robin delivery, there's no way for the broker to know which messages are going to which consumer.

_Key shared subscriptions_ allow multiple consumers to subscribe to a topic, and provide additional metadata in the form of _keys_ that link messages to specific consumers.
Expand All @@ -13,14 +13,14 @@ Keys are generated with hashing that converts arbitrary values like `topic-name`
* *Auto hash*: Uses _consistent hashing_ to balance range values across available consumers without requiring manual setup of hash ranges.
* *Sticky hash*: The client manually assigns consumer range values, and then all hashes within a configured range go to one consumer.

This page explains how to use Pulsar's key shared subscription model to manage your topic consumption.
This page explains how to use {pulsar-short}'s key shared subscription model to manage your topic consumption.

include::ROOT:partial$subscription-prereq.adoc[]

[#example]
== Key shared subscription example

. To try out a Pulsar key shared subscription, add `.subscriptionType(SubscriptionType.Key_Shared)` to the `pulsarConsumer` in `SimplePulsarConsumer.java`:
. To try out a {pulsar-short} key shared subscription, add `.subscriptionType(SubscriptionType.Key_Shared)` to the `pulsarConsumer` in `SimplePulsarConsumer.java`:
+
.SimplePulsarConsumer.java
[source,java]
Expand Down Expand Up @@ -144,7 +144,7 @@ Then, you can launch multiple instances of `SimplePulsarConsumer.java` to consum

== See also

* xref:astream-subscriptions.adoc[Subscriptions in Pulsar]
* xref:astream-subscriptions-exclusive.adoc[Exclusive subscriptions]
* xref:astream-subscriptions-shared.adoc[Shared subscriptions]
* xref:astream-subscriptions-failover.adoc[Failover subscriptions]
* xref:astream-subscriptions.adoc[]
* xref:astream-subscriptions-exclusive.adoc[]
* xref:astream-subscriptions-shared.adoc[]
* xref:astream-subscriptions-failover.adoc[]
18 changes: 9 additions & 9 deletions modules/ROOT/pages/astream-subscriptions-shared.adoc
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
= Shared subscriptions in Pulsar
= Shared subscriptions in {pulsar-reg}
:navtitle: Shared
:page-tag: pulsar-subscriptions,quickstart,admin,dev,pulsar

_Subscriptions_ in Pulsar describe which consumers are consuming data from a topic and how they want to consume that data.
_Subscriptions_ in {pulsar-reg} describe which consumers are consuming data from a topic and how they want to consume that data.

A _shared subscription_ allows multiple consumers to consume messages from a single topic in a round-robin fashion.
More consumers in a shared subscription can increase your Pulsar deployment's rate of message consumption.
More consumers in a shared subscription can increase your {pulsar-short} deployment's rate of message consumption.
However, there is a risk of losing message ordering guarantees and acknowledgment schemes.

This page explains how you can use Pulsar's shared subscription model to manage your topic consumption.
This page explains how you can use {pulsar-short}'s shared subscription model to manage your topic consumption.

include::ROOT:partial$subscription-prereq.adoc[]

[#example]
== Shared subscription example

To try out a Pulsar shared subscription, add `.subscriptionType(SubscriptionType.Shared)` to the `pulsarConsumer` in `SimplePulsarConsumer.java`:
To try out a {pulsar-short} shared subscription, add `.subscriptionType(SubscriptionType.Shared)` to the `pulsarConsumer` in `SimplePulsarConsumer.java`:

.SimplePulsarConsumer.java
[source,java]
Expand Down Expand Up @@ -85,7 +85,7 @@ All the consumers subscribe to the topic and consume messages in a round-robin f

== See also

* xref:astream-subscriptions.adoc[Subscriptions in Pulsar]
* xref:astream-subscriptions-exclusive.adoc[Exclusive subscriptions]
* xref:astream-subscriptions-failover.adoc[Failover subscriptions]
* xref:astream-subscriptions-keyshared.adoc[Key shared subscriptions]
* xref:astream-subscriptions.adoc[]
* xref:astream-subscriptions-exclusive.adoc[]
* xref:astream-subscriptions-failover.adoc[]
* xref:astream-subscriptions-keyshared.adoc[]
Loading
Loading