Does current implementation of package manager support Oxia?

[anosulchik]

Hi, I'm trying to enable pulsar packages management for functions (also for sinks and sources) in the cluster that uses Oxia as a metadata service. Brokers fails to initialize package management throwing an error java.net.URI.getAuthority() for oxia:// URLs. Does package management supports Oxia at this stage or ZK is required?

Brokers config excerpt:

      # ...
      bookkeeperMetadataServiceUri: metadata-store:oxia://mi-apache-pulsar-staging-oxia-svc:6648/bookkeeper
      configurationMetadataStoreUrl: oxia://mi-apache-pulsar-staging-oxia-svc:6648/broker
      metadataStoreUrl: oxia://mi-apache-pulsar-staging-oxia-svc:6648/broker
      # ...
      enablePackagesManagement: "true"
      packagesManagementStorageProvider: "org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorageProvider"
      packagesReplicas: "1"
      packagesManagementLedgerRootPath: "/ledgers"
      functionsWorkerEnablePackageManagement: "true"

Logs can be found below, please see org.apache.pulsar.broker.PulsarService.startPackagesManagementService in a stacktrace:

2025-02-25T18:51:48,511+0000 [pulsar-client-io-48-2] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/functions/metadata][c-mi-apache-pulsar-staging-fw-mi-apache-pulsar-staging-broker-2.mi-apache-pulsar-staging-broker.mi-apache-pulsar.svc.cluster.local-8080-function-metadata-tailer-reader-c276757b0f] Subscribed to topic on mi-apache-pulsar-staging-broker-1.mi-apache-pulsar-staging-broker.mi-apache-pulsar.svc.cluster.local/<ipadress>:6650 -- consumer: 4
2025-02-25T18:51:48,515+0000 [main] INFO  org.apache.pulsar.functions.worker.FunctionMetaDataManager - MetaData Manager Tailer started
2025-02-25T18:51:48,518+0000 [main] INFO  org.apache.pulsar.functions.worker.PulsarWorkerService - /** Starting Cluster Service Coordinator **/
2025-02-25T18:51:48,518+0000 [main] INFO  org.apache.pulsar.functions.worker.ClusterServiceCoordinator - /** Starting cluster service coordinator **/
2025-02-25T18:51:48,537+0000 [main] INFO  org.apache.pulsar.functions.worker.PulsarWorkerService - /** Started worker id=c-mi-apache-pulsar-staging-fw-mi-apache-pulsar-staging-broker-2.mi-apache-pulsar-staging-broker.mi-apache-pulsar.svc.cluster.local-8080 **/
2025-02-25T18:51:48,537+0000 [main] INFO  org.apache.pulsar.broker.PulsarService - Function worker service started
2025-02-25T18:51:48,596+0000 [main] INFO  org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorage - Applying DLog BookKeeper client configuration setting bkc.tlsHostnameVerificationEnabled=false
2025-02-25T18:51:48,602+0000 [main] ERROR org.apache.pulsar.broker.PulsarService - Failed to start Pulsar service: Cannot invoke "String.replace(java.lang.CharSequence, java.lang.CharSequence)" because the return value of "java.net.URI.getAuthority()" is null
java.lang.NullPointerException: Cannot invoke "String.replace(java.lang.CharSequence, java.lang.CharSequence)" because the return value of "java.net.URI.getAuthority()" is null
	at org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorage.initializeDlogNamespace(BookKeeperPackagesStorage.java:100) ~[org.apache.pulsar-pulsar-package-bookkeeper-storage-4.0.2.jar:4.0.2]
	at org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorage.initialize(BookKeeperPackagesStorage.java:87) ~[org.apache.pulsar-pulsar-package-bookkeeper-storage-4.0.2.jar:4.0.2]
	at org.apache.pulsar.broker.PulsarService.startPackagesManagementService(PulsarService.java:1965) ~[org.apache.pulsar-pulsar-broker-4.0.2.jar:4.0.2]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:1055) [org.apache.pulsar-pulsar-broker-4.0.2.jar:4.0.2]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.call(PulsarBrokerStarter.java:275) [org.apache.pulsar-pulsar-broker-4.0.2.jar:4.0.2]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.call(PulsarBrokerStarter.java:130) [org.apache.pulsar-pulsar-broker-4.0.2.jar:4.0.2]
	at picocli.CommandLine.executeUserObject(CommandLine.java:2041) [info.picocli-picocli-4.7.5.jar:4.7.5]
	at picocli.CommandLine.access$1500(CommandLine.java:148) [info.picocli-picocli-4.7.5.jar:4.7.5]
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461) [info.picocli-picocli-4.7.5.jar:4.7.5]
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2453) [info.picocli-picocli-4.7.5.jar:4.7.5]
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2415) [info.picocli-picocli-4.7.5.jar:4.7.5]
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273) [info.picocli-picocli-4.7.5.jar:4.7.5]
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2417) [info.picocli-picocli-4.7.5.jar:4.7.5]
	at picocli.CommandLine.execute(CommandLine.java:2170) [info.picocli-picocli-4.7.5.jar:4.7.5]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:151) [org.apache.pulsar-pulsar-broker-4.0.2.jar:4.0.2]
org.apache.pulsar.broker.PulsarServerException: java.lang.NullPointerException: Cannot invoke "String.replace(java.lang.CharSequence, java.lang.CharSequence)" because the return value of "java.net.URI.getAuthority()" is null
	at org.apache.pulsar.broker.PulsarServerException.from(PulsarServerException.java:56)
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:1088)
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.call(PulsarBrokerStarter.java:275)
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.call(PulsarBrokerStarter.java:130)
	at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
	at picocli.CommandLine.access$1500(CommandLine.java:148)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
	at picocli.CommandLine.execute(CommandLine.java:2170)
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:151)
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:366)
Caused by: java.lang.NullPointerException: Cannot invoke "String.replace(java.lang.CharSequence, java.lang.CharSequence)" because the return value of "java.net.URI.getAuthority()" is null
	at org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorage.initializeDlogNamespace(BookKeeperPackagesStorage.java:100)
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:366) [org.apache.pulsar-pulsar-broker-4.0.2.jar:4.0.2]
	at org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorage.initialize(BookKeeperPackagesStorage.java:87)
	at org.apache.pulsar.broker.PulsarService.startPackagesManagementService(PulsarService.java:1965)
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:1055)
	... 12 more
2025-02-25T18:51:48,655+0000 [pulsar-service-shutdown] INFO  org.apache.pulsar.functions.worker.FunctionMetaDataTopicTailer - Stopping function metadata tailer
2025-02-25T18:51:48,660+0000 [function-metadata-tailer-thread] INFO  org.apache.pulsar.functions.worker.FunctionMetaDataTopicTailer - metadata tailer thread exiting
2025-02-25T18:51:48,666+0000 [pulsar-client-io-48-2] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/functions/metadata] [c-mi-apache-pulsar-staging-fw-mi-apache-pulsar-staging-broker-2.mi-apache-pulsar-staging-broker.mi-apache-pulsar.svc.cluster.local-8080-function-metadata-tailer-reader-c276757b0f] Closed consumer
2025-02-25T18:51:48,668+0000 [pulsar-service-shutdown] INFO  org.apache.pulsar.functions.worker.FunctionMetaDataTopicTailer - Stopped function metadata tailer
2025-02-25T18:51:48,669+0000 [pulsar-service-shutdown] INFO  org.apache.pulsar.functions.worker.FunctionAssignmentTailer - Closing function assignment tailer
2025-02-25T18:51:48,672+0000 [assignment-tailer-thread] INFO  org.apache.pulsar.functions.worker.FunctionAssignmentTailer - assignment tailer thread exiting
2025-02-25T18:51:48,677+0000 [pulsar-client-io-48-2] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/functions/assignments] [c-mi-apache-pulsar-staging-fw-mi-apache-pulsar-staging-broker-2.mi-apache-pulsar-staging-broker.mi-apache-pulsar.svc.cluster.local-8080-function-assignment-tailer-reader-f137cfeee3] Closed consumer
2025-02-25T18:51:48,678+0000 [pulsar-service-shutdown] WARN  org.apache.pulsar.functions.worker.FunctionRuntimeManager - Will not stop any functions since they are externally managed
2025-02-25T18:51:48,679+0000 [pulsar-service-shutdown] INFO  org.apache.pulsar.functions.worker.ClusterServiceCoordinator - Stopping Cluster Service Coordinator for worker c-mi-apache-pulsar-staging-fw-mi-apache-pulsar-staging-broker-2.mi-apache-pulsar-staging-broker.mi-apache-pulsar.svc.cluster.local-8080
2025-02-25T18:51:48,681+0000 [pulsar-service-shutdown] INFO  org.apache.pulsar.functions.worker.ClusterServiceCoordinator - Stopped Cluster Service Coordinator for worker c-mi-apache-pulsar-staging-fw-mi-apache-pulsar-staging-broker-2.mi-apache-pulsar-staging-broker.mi-apache-pulsar.svc.cluster.local-8080
2025-02-25T18:51:48,682+0000 [pulsar-service-shutdown] INFO  org.apache.pulsar.functions.worker.SchedulerManager - Closing scheduler manager

[lhotari]

No, Oxia isn't supported. The supported option currently is file storage, but that's not available in the Apache Pulsar Helm chart configs. There would need to be NFS or some other shared storage configured in the cluster. Cloud provider k8s support this, but adding the support to the Helm chart would be needed.
An alternative would be to have a package service that uses cloud storage (S3) directly. That's missing from Apache Pulsar.