[QUERY] Correct configuration for spring-cloud-azure-stream-binder-servicebus using credentials as service principal.

[fredysierra]

Query/Question
We have a simple spring app that uses Azure Service Bus with Spring Cloud Stream and spring-cloud-azure-stream-binder-servicebus using credentials as service principal as described here. Our main goal is to disable SAS key authentication for a Service Bus namespace and allow only Azure Active Directory authentication.

I added the following configuration:

spring:
  cloud:
    azure:
      credential:
        client-id: ${AZURE_CLIENT_ID}
        client-secret: ${AZURE_CLIENT_SECRET}
      profile:
        tenant-id: ${AZURE_TENANT_ID}
        subscription-id: ${AZURE_SUBSCRIPTION_ID}
      servicebus:
        enabled: true
        entity-type: topic
        resource:
          resource-group: ${AZURE_RESOURCE_GROUP}
        namespace: ${SERVICEBUS_NAMESPACE}
    stream:
      function.definition: consume
      bindings:
        consume-in-0.destination: topic0
        consume-in-0.group: sub0
        hello-out-0.destination: topic0
      servicebus:
        bindings:
          consume-in-0.consumer.auto-complete: false
          hello-out-0.producer.entity-type: topic
      default-binder: servicebus

With this configuration, it is possible to see that the topic topic0 and subscription sub0 have been automatically created when the application starts.

2023-09-29 12:44:29.897  INFO 27842 --- [           main] zureServiceBusMessagingAutoConfiguration : Service Bus connection string is set from com.azure.spring.cloud.resourcemanager.implementation.connectionstring.ServiceBusArmConnectionStringProvider now.
2023-09-29 12:44:30.157  INFO 27842 --- [           main] o.s.c.s.m.DirectWithAttributesChannel    : Channel 'application.consume-in-0' has 1 subscriber(s).
2023-09-29 12:44:30.225  INFO 27842 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 4 endpoint(s) beneath base path '/actuator'
2023-09-29 12:44:30.327  INFO 27842 --- [           main] o.s.i.monitor.IntegrationMBeanExporter   : Registering MessageChannel errorChannel
2023-09-29 12:44:30.330  INFO 27842 --- [           main] o.s.i.monitor.IntegrationMBeanExporter   : Registering MessageChannel consume-in-0
2023-09-29 12:44:30.335  INFO 27842 --- [           main] o.s.i.monitor.IntegrationMBeanExporter   : Registering MessageChannel nullChannel
2023-09-29 12:44:30.338  INFO 27842 --- [           main] o.s.i.monitor.IntegrationMBeanExporter   : Registering MessageHandler _org.springframework.integration.errorLogger
2023-09-29 12:44:30.347  INFO 27842 --- [           main] o.s.i.endpoint.EventDrivenConsumer       : Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2023-09-29 12:44:30.347  INFO 27842 --- [           main] o.s.i.channel.PublishSubscribeChannel    : Channel 'application.errorChannel' has 1 subscriber(s).
2023-09-29 12:44:30.347  INFO 27842 --- [           main] o.s.i.endpoint.EventDrivenConsumer       : started bean '_org.springframework.integration.errorLogger'
2023-09-29 12:44:30.609  INFO 27842 --- [           main] c.a.s.c.r.i.crud.AbstractResourceCrud    : Fetching ServiceBusSubscription with name 'sub0' ...
2023-09-29 12:44:30.610  INFO 27842 --- [           main] c.a.s.c.r.i.crud.AbstractResourceCrud    : Fetching Topic with name 'topic0' ...
2023-09-29 12:44:30.610  INFO 27842 --- [           main] c.a.s.c.r.i.crud.AbstractResourceCrud    : Fetching ServiceBusNamespace with name 'testrnddesb' ...
2023-09-29 12:44:31.021  INFO 27842 --- [           main] c.a.s.c.r.i.crud.AbstractResourceCrud    : Fetching ServiceBusNamespace with name 'testrnddesb' finished in 0 seconds
2023-09-29 12:44:32.011  INFO 27842 --- [           main] c.a.s.c.r.i.crud.AbstractResourceCrud    : Fetching Topic with name 'topic0' finished in 1 seconds
2023-09-29 12:44:32.011  INFO 27842 --- [           main] c.a.s.c.r.i.crud.AbstractResourceCrud    : Fetching ServiceBusSubscription with name 'sub0' finished in 1 seconds
2023-09-29 12:44:32.011  INFO 27842 --- [           main] c.a.s.c.r.i.crud.AbstractResourceCrud    : Creating ServiceBusSubscription with name 'sub0' ...
2023-09-29 12:44:32.011  INFO 27842 --- [           main] c.a.s.c.r.i.crud.AbstractResourceCrud    : Fetching Topic with name 'topic0' ...
2023-09-29 12:44:32.011  INFO 27842 --- [           main] c.a.s.c.r.i.crud.AbstractResourceCrud    : Fetching ServiceBusNamespace with name 'testrnddesb' ...
2023-09-29 12:44:33.610  INFO 27842 --- [           main] c.a.s.c.r.i.crud.AbstractResourceCrud    : Fetching ServiceBusNamespace with name 'testrnddesb' finished in 1 seconds
2023-09-29 12:44:34.419  INFO 27842 --- [           main] c.a.s.c.r.i.crud.AbstractResourceCrud    : Fetching Topic with name 'topic0' finished in 2 seconds
2023-09-29 12:44:34.419  INFO 27842 --- [           main] c.a.s.c.r.i.crud.AbstractResourceCrud    : Creating Topic with name 'topic0' ...
2023-09-29 12:44:34.419  INFO 27842 --- [           main] c.a.s.c.r.i.crud.AbstractResourceCrud    : Fetching ServiceBusNamespace with name 'testrnddesb' ...
2023-09-29 12:44:36.079  INFO 27842 --- [           main] c.a.s.c.r.i.crud.AbstractResourceCrud    : Fetching ServiceBusNamespace with name 'testrnddesb' finished in 1 seconds
2023-09-29 12:44:38.144  INFO 27842 --- [           main] c.a.s.c.r.i.crud.AbstractResourceCrud    : Creating Topic with name 'topic0' finished in 3 seconds
2023-09-29 12:44:39.521  INFO 27842 --- [           main] c.a.s.c.r.i.crud.AbstractResourceCrud    : Creating ServiceBusSubscription with name 'sub0' finished in 7 seconds
2023-09-29 12:44:39.540  INFO 27842 --- [           main] o.s.c.stream.binder.BinderErrorChannel   : Channel 'topic0.sub0.errors' has 1 subscriber(s).
2023-09-29 12:44:39.541  INFO 27842 --- [           main] o.s.c.stream.binder.BinderErrorChannel   : Channel 'topic0.sub0.errors' has 2 subscriber(s).
2023-09-29 12:44:39.548  INFO 27842 --- [           main] AbstractAzureServiceClientBuilderFactory : Will configure the default credential of type DefaultAzureCredential for class com.azure.identity.ClientSecretCredentialBuilder.
2023-09-29 12:44:39.606  INFO 27842 --- [           main] c.a.m.s.i.ServiceBusConnectionProcessor  : {"az.sdk.message":"Setting next AMQP channel.","entityPath":"N/A"}
2023-09-29 12:44:39.607  INFO 27842 --- [           main] c.a.m.s.ServiceBusClientBuilder          : # of open clients with shared connection: 1
2023-09-29 12:44:39.618  INFO 27842 --- [           main] c.a.m.s.ServiceBusReceiverAsyncClient    : {"az.sdk.message":"Creating consumer.","linkName":"topic0/subscriptions/sub0_bd59d4_1695984279618","entityPath":"topic0/subscriptions/sub0"}
2023-09-29 12:44:39.622  INFO 27842 --- [           main] c.a.m.s.i.ServiceBusReceiveLinkProcessor : Requesting a new AmqpReceiveLink from upstream.
2023-09-29 12:44:39.624

I can see the topic topic0 subscription sub0created in portal.azure.com. However, I can also see that the app has issues when wants to use the topic and subscription.

2023-09-29 12:44:39.927  INFO 27842 --- [ctor-executor-1] c.a.c.a.i.handler.ReceiveLinkHandler     : {"az.sdk.message":"onLinkRemoteOpen","connectionId":"MF_4f992b_1695984279564","entityPath":"$cbs","linkName":"cbs:receiver","remoteSource":"Source{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=null, defaultOutcome=null, outcomes=null, capabilities=null}"}
2023-09-29 12:44:39.944 ERROR 27842 --- [ctor-executor-1] c.a.core.amqp.implementation.RetryUtil   : Failed to create receive link topic0/subscriptions/sub0_bd59d4_1695984279618
status-code: 401, status-description: LocalAuthDisabled: Authorization failed because SAS authentication has been disabled for the namespace. TrackingId:5e9ff2d0-343f-48ec-bd87-140f8291b4ae_G6, SystemTracker:NoSystemTracker, Timestamp:2023-09-29T10:44:39, errorContext[NAMESPACE: testrnddesb.servicebus.windows.net. ERROR CONTEXT: N/A, PATH: $cbs, REFERENCE_ID: cbs:receiver, LINK_CREDIT: 0]
2023-09-29 12:44:39.944  WARN 27842 --- [ctor-executor-1] c.a.m.s.i.ServiceBusReceiveLinkProcessor : {"az.sdk.message":"Non-retryable error occurred in AMQP receive link.","exception":"status-code: 401, status-description: LocalAuthDisabled: Authorization failed because SAS authentication has been disabled for the namespace. TrackingId:5e9ff2d0-343f-48ec-bd87-140f8291b4ae_G6, SystemTracker:NoSystemTracker, Timestamp:2023-09-29T10:44:39, errorContext[NAMESPACE: testrnddesb.servicebus.windows.net. ERROR CONTEXT: N/A, PATH: $cbs, REFERENCE_ID: cbs:receiver, LINK_CREDIT: 0]","linkName":"n/a","entityPath":"n/a"}
2023-09-29 12:44:39.944 ERROR 27842 --- [ctor-executor-1] .s.FluxAutoLockRenew$LockRenewSubscriber : Errors occurred upstream.
status-code: 401, status-description: LocalAuthDisabled: Authorization failed because SAS authentication has been disabled for the namespace. TrackingId:5e9ff2d0-343f-48ec-bd87-140f8291b4ae_G6, SystemTracker:NoSystemTracker, Timestamp:2023-09-29T10:44:39, errorContext[NAMESPACE: testrnddesb.servicebus.windows.net. ERROR CONTEXT: N/A, PATH: $cbs, REFERENCE_ID: cbs:receiver, LINK_CREDIT: 0]
2023-09-29 12:44:39.946  INFO 27842 --- [ctor-executor-1] c.a.m.s.ServiceBusProcessorClient        : Error receiving messages.
2023-09-29 12:44:39.950 ERROR 27842 --- [ctor-executor-1] .s.i.s.i.ServiceBusInboundChannelAdapter : Error in the operation RECEIVE occurred on entity topic0/subscriptions/sub0. Error: {}

com.azure.messaging.servicebus.ServiceBusException: status-code: 401, status-description: LocalAuthDisabled: Authorization failed because SAS authentication has been disabled for the namespace. TrackingId:5e9ff2d0-343f-48ec-bd87-140f8291b4ae_G6, SystemTracker:NoSystemTracker, Timestamp:2023-09-29T10:44:39, errorContext[NAMESPACE: testrnddesb.servicebus.windows.net. ERROR CONTEXT: N/A, PATH: $cbs, REFERENCE_ID: cbs:receiver, LINK_CREDIT: 0]
	at com.azure.messaging.servicebus.ServiceBusReceiverAsyncClient.mapError(ServiceBusReceiverAsyncClient.java:1676) ~[azure-messaging-servicebus-7.14.0.jar:7.14.0]
	at com.azure.messaging.servicebus.ServiceBusReceiverAsyncClient.lambda$receiveMessagesWithContext$21(ServiceBusReceiverAsyncClient.java:892) ~[azure-messaging-servicebus-7.14.0.jar:7.14.0]
	at reactor.core.publisher.Flux.lambda$onErrorMap$28(Flux.java:7070) ~[reactor-core-3.4.26.jar:3.4.26]
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94) ~[reactor-core-3.4.26.jar:3.4.26]
	at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.doError(FluxPublishOn.java:511) ~[reactor-core-3.4.26.jar:3.4.26]
	

Why is it allowing us to create the topic and at the same time complain about using it with an Authorization failed because SAS authentication has been disabled for the namespace?

I continued exploring alternatives and I ended up trying the same configuration above but using spring.cloud.azure.servicebus instead of spring.cloud.azure.

spring:
  cloud:
    azure:
      servicebus:
        enabled: true
        entity-type: topic
        credential:
          client-id: ${AZURE_CLIENT_ID}
          client-secret: ${AZURE_CLIENT_SECRET}
        profile:
          tenant-id: ${AZURE_TENANT_ID}
          subscription-id: ${AZURE_SUBSCRIPTION_ID}
        resource:
          resource-group:  ${AZURE_RESOURCE_GROUP}
        namespace: ${SERVICEBUS_NAMESPACE}
    stream:
      function.definition: consume
      bindings:
        consume-in-0.destination: topic0
        consume-in-0.group: sub0
        hello-out-0.destination: topic0
      servicebus:
        bindings:
          consume-in-0.consumer.auto-complete: false
          hello-out-0.producer.entity-type: topic
      default-binder: servicebus

With that configuration works fine accessing the topics and allowing us to send a receive messages. ONLY if the topic topic0 and subscription sub0 are created manually using the portal.azure.com. With this configuration, the automatic creation of topics and subscriptions is not happening.

In spring-cloud-azure-stream-binder-servicebus, I can see in ServiceBusBinderConfiguration two ProvisioningProvider: ServiceBusChannelResourceManagerProvisioner and ServiceBusChannelProvisioner. The logic for creating topics and subscriptions seems to be only present in ServiceBusChannelResourceManagerProvisioner. When our configuration uses the prefix spring.cloud.azure.servicebus the bean ServiceBusChannelProvisioner is used and it does not have any topic creation logic.

Setup:

• spring-cloud-azure-stream-binder-servicebus:5.5.0
• com.azure:azure-core:1.42.0
• Spring boot 3.1.4
• Spring Cloud 2022.0.4

Can someone help me with the correct configuration for spring apps with credentials as service principal and auto-creation of topics at application startup?

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Query Added
• Setup information Added

[joshfree]

@yiliuTo could you please assist?

[Netyyyy]

Hi @fredysierra , thanks for reaching out.
We have received your submission and will take it into consideration. We appreciate your input and will review this matter as soon as possible.
Please feel free to provide any additional information or context that you think may be helpful. We'll keep you updated on the progress of our review.
Thank you for your contribution to improving our project.