Installing a custom sink connector

[filipqlrf]

What is the best way to add a custom sink connector to the Pulsar instance deployed using the official helm chart? This is the connector I want to use: https://github.com/streamnative/pulsar-io-lakehouse

Once I have the connector available under /connectors, it should be easy to setup a sink using that connector and referencing it via
"archive": "connectors/pulsar-io-lakehouse-{{connector:version}}-cloud.nar"

Is there a configuration to do it out of the box, should I use initContainers or something else?

[lhotari]

There's a possibility to allow installing directly from an external https:// url. This is disabled by default because of security reasons.

pulsar/conf/functions_worker.yml

Lines 420 to 429 in 4a88721

	connectorsDirectory: ./connectors
	# Whether to enable referencing connectors directory files by file url in connector (sink/source) creation
	enableReferencingConnectorDirectoryFiles: true
	# Regex patterns for enabling creation of connectors by referencing packages in matching http/https urls
	additionalEnabledConnectorUrlPatterns: []
	functionsDirectory: ./functions
	# Whether to enable referencing functions directory files by file url in functions creation
	enableReferencingFunctionsDirectoryFiles: true
	# Regex patterns for enabling creation of functions by referencing packages in matching http/https urls
	additionalEnabledFunctionsUrlPatterns: []

For trusted URLs, it's possible to configure additionalEnabledConnectorUrlPatterns (for connectors) and additionalEnabledFunctionsUrlPatterns (for functions).

In the Apache Pulsar Helm chart, you can set these config values with

broker:
  configData:
    PF_additionalEnabledConnectorUrlPatterns: "https://someurl1/.*,https://someurl2/.*"
    PF_additionalEnabledFunctionsUrlPatterns: "https://someurl1/.*,https://someurl2/.*"

in this case this might work so that you could refer the download urls directly, such as --archive=https://github.com/streamnative/pulsar-io-lakehouse/releases/download/v3.0.10.1/pulsar-io-lakehouse-3.0.10.1.nar, in the pulsar-admin sink create command line.

broker:
  configData:
    PF_additionalEnabledConnectorUrlPatterns: "https://github.com/streamnative/pulsar-io-lakehouse/releases/download/.*"

I haven't tested this so you might run into issues along the way.

[lhotari]

Another solution would be to build a custom docker image with required connectors in the image

1. Save this to a directory as Dockerfile

# example usage: docker buildx build --platform linux/amd64 --build-arg PULSAR_IMAGE=apachepulsar/pulsar:4.0.3 -t myorg/pulsar:4.0.3-custom .
ARG PULSAR_IMAGE=apachepulsar/pulsar:latest
FROM ${PULSAR_IMAGE} AS pulsar
COPY /connectors /pulsar/connectors

2. Create connectors and download the connectors there.

mkdir connectors
cd connectors
curl -LO https://github.com/streamnative/pulsar-io-lakehouse/releases/download/v4.0.3.1/pulsar-io-lakehouse-4.0.3.1.nar

3. Build and push the image

docker buildx build --push --platform linux/amd64 --build-arg PULSAR_IMAGE=apachepulsar/pulsar:4.0.3 -t myorg/pulsar:4.0.3-custom .

(remove --push if you just want to test locally)

If you need arm64 support, add --platform linux/amd64,linux/arm64. To avoid issues, pass --push to the command line when building multi-platform images since otherwise you might not get a multi-platform image in your repository.
example of building and pushing multi-platform image:

docker buildx build --push --platform linux/amd64,linux/arm64 --build-arg PULSAR_IMAGE=apachepulsar/pulsar:4.0.3 -t myorg/pulsar:4.0.3-custom .

And finally use this image in Apache Pulsar Helm chart as the default image

defaultPulsarImageRepository: myorg/pulsar
defaultPulsarImageTag: 4.0.3-custom

It also replaces apachepulsar/pulsar-all with much smaller apachepulsar/pulsar image. If you need all connectors, you can build a custom image based on apachepulsar/pulsar-all.

[filipqlrf]

The first answer worked! Really appreciate the fast and detailed answer.

I managed to create a sink using pulsar-admin command directly on the pulsar toolset pod. However, using the Pulsar Resource Operator and the same configuration as the one I used with pulsar-admin, it fails with:
apply pulsar sink failed on create sink with url: code: 404 reason: 404 Not Found.

I wonder if we have to configure similar option for the Operator to allow the https endpoints.

[lhotari]

@filipqlrf The Resource Operator issue could be different. You could report to https://github.com/streamnative/pulsar-resources-operator/issues with a reproducible example.