update-space-users failing in v1.0.88

[supp429]

Description
update-space-users is failing after upgrade from v1.0.43 to v1.0.88 with the following error message.

2024/05/29 17:28:40 D0529 17:28:40.861327 52 initialize.go:78] Using Version: [1.0.88], Commit: [01a5d93ea44bf4c2cea85238f7b8a90431a9f8b2] of cf-mgmt
2024/05/29 17:28:46 D0529 17:28:46.747409 52 manager.go:87] Found 8328 roles from space API
2024/05/29 17:28:46 Sending quit message and waiting for confirmation
2024/05/29 17:28:46 Shutting down - quit message received
2024/05/29 17:28:46 Closing network connection
error: got errors processing update space users [role for type space with GUID[07548fe0-cac2-11ed-bcec-005056a01210] is returned multiple times, prior role [{"guid":"07548fe0-cac2-11ed-bcec-005056a01210","created_at":"2023-03-25T04:03:36Z","updated_at":"2023-03-25T04:03:40Z","type":"space_developer","relationships":{"space":{"data":{"guid":"dfc721d9-471e-4100-a2af-2439d48943b9"}},"user":{"data":{"guid":"1b53a4b2-bb19-471b-bf9b-1393b28a46eb"}},"organization":{"data":null}},"links":{"self":{"href":"https://api.acme.net/v3/roles/07548fe0-cac2-11ed-bcec-005056a01210"},"space":{"href":"https://api.acme.net/v3/spaces/dfc721d9-471e-4100-a2af-2439d48943b9"},"user":{"href":"https://api.acme.net/v3/users/1b53a4b2-bb19-471b-bf9b-1393b28a46eb"}}}] and current role [{"guid":"07548fe0-cac2-11ed-bcec-005056a01210","created_at":"2023-03-25T04:03:36Z","updated_at":"2023-03-25T04:03:40Z","type":"space_developer","relationships":{"space":{"data":{"guid":"dfc721d9-471e-4100-a2af-2439d48943b9"}},"user":{"data":{"guid":"1b53a4b2-bb19-471b-bf9b-1393b28a46eb"}},"organization":{"data":null}},"links":{"self":{"href":"https://api.acme.net/v3/roles/07548fe0-cac2-11ed-bcec-005056a01210"},"space":{"href":"https://api.acme.net/v3/spaces/dfc721d9-471e-4100-a2af-2439d48943b9"},"user":{"href":"https://api.acme.net/v3/users/1b53a4b2-bb19-471b-bf9b-1393b28a46eb"}}}]]

To Reproduce
Steps to reproduce the behavior:

1. Run update-space-users command with necessary config

Expected behavior
User sync completes as configured.

Screenshots
Added the DEBUG logs above.

CF Versions
TAS 2.11.x
Opsman 2.10.x
cf-mgmt 1.0.88

Additional context
Possibly related to #423

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[calebwashburn]

@supp429 Can you provide the specific version of TAS (which 2.11.x) to see if we can recreate on this version as unable to recreate on later versions?

[calebwashburn]

@supp429 trying to see if this isolates and fixes the error. This can be tested using the develop tag from dockerhub image.

https://hub.docker.com/r/pivotalservices/cf-mgmt/tags

[calebwashburn]

If this doesn't resolve your issue believe you will need to upgrade to 4.x to get a newer version of CAPI than 2.11 ships with as this pagination fix is in 1.111 of capi - cloudfoundry/cloud_controller_ng#2249 which the latest vs of TAS 2.13 ships with 1.109

[supp429]

@calebwashburn Apologize for the delay in responding. We are on TAS 2.11.19 which ships with capi 1.109.9 as per the release notes. As you have predicted, we may have wait until TAS 4.x upgrade. I will also try your recommendation to test with the development image.