Parallel build support

[mikezerosix]

We have a lot of proxies and running deploy takes looooong time.

We structured our solution as Maven multi module project. So simply running from root all we can easily deploy them all and
we also can easily choose to deploy them individually from each submodule.

But would be great if we would deploy all of them at once using Maven parallel build: mvn -T 10 install
(install phase runs the apigee plugin)

We have tried parallel build, but it fail with strange errors like :

[ERROR] Failed to execute goal io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.5.0:deploy (deploy-bundle) on project shared-request-player: MojoExecutionException: com.google.api.client.http.HttpResponseException: 404 Not Found
[ERROR] {
[ERROR]   "error": {
[ERROR]     "code": 404,
[ERROR]     "message": "generic::not_found: organizations/xxx/sharedflows/shared-xxx/revisions/218 not found",
[ERROR]     "status": "NOT_FOUND",
[ERROR]     "details": [
[ERROR]       {
[ERROR]         "@type": "type.googleapis.com/google.rpc.RequestInfo",
[ERROR]         "requestId": "11261845108213792315"
[ERROR]       }
[ERROR]     ]
[ERROR]   }
[ERROR] }

That shared flow has no dependencies and it deploys without any issues in non-parallel build.
Cutting down the half hour deploy time to just 3 minutes being able to run it in parallel would be amazing.

So if possible, please suppport Maven parallel builds.

[ssvaidyanathan]

@mikezerosix - I believe you just want to perform fire and forget style deployments and not wait for the deployments to complete. The default option is that so that you can include triggering the automated testing after knowing that deployment is complete.

If you do not want that - you can change the deploy options. Just use -Dapigee.options=async or update the apigee.options value in the pom profile to <apigee.options>async</apigee.options>

That will just run the import and deploy command and not poll for the deployment to complete. The same is documented as well

[mikezerosix]

@ssvaidyanathan Thanks. I am aware of the async option. But I do want to wait for confirmation, I just do not run deployments sequentially. I have maven multimodule project with 20 sub modules, each is apigee flow or proxy. if I run the maven normally mvn install it takes maven almost an hour to run the build as it runs each proxy deployment one at time wating for one to complete before running the next. What I do want to, is to run Maven with mvn -T 20 which would run maven in 20 threads, running all 20 submodules at once and it would complete in 2 minutes (time of single polled deployment)
But when I try to run maven in parallel (even just "-T 2", cutting build time to half), I get errors from this plugin.

[ERROR] Failed to execute goal io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.5.0:deploy (deploy-bundle) on project shared-request-player: MojoExecutionException: com.google.api.client.http.HttpResponseException: 404 Not Found
[ERROR] {
[ERROR]   "error": {
[ERROR]     "code": 404,
[ERROR]     "message": "generic::not_found: organizations/xxx/sharedflows/shared-xxx/revisions/218 not found",

So it looks like this plugin does not support multithreading and it would be amazing if it did.

Or I would love have ability to run async deploy and then run separate process to poll projects deployment

1. First run mvn apigee-edge-maven-plugin:deploy -Dapigee.options=async
   Which would run the 20 sub modules each with proxy in less than minute.

2. Then run mvn apigee-edge-maven-plugin:POLL-ASYNC-DEPLOYMENTS
   Which would poll and untill each of the previously deployed proxies and shared flows report having been deployes or report failure. This should complete in 2-3 minutes assuming Apigee itself runs deployments paraller.

Problem here would be that this would need to persist information between these two runs (like mvn release plugin) so it knows which revisions to poll for. So it does not get confused from 3rd party deploys.

[ssvaidyanathan]

@mikezerosix - Thank you for the detailed summary. Great pointers and suggestions.
Let me see what can be done. Will try them out.
This is the first time someone's asking this. So thats great 😄

I will need a favor - if I end up implementing, I will need you to help test it for me and see if it works as expected. We can fine tune as you provide more inputs.

[mikezerosix]

I created sample project for the issue. I have not tested it that it actually runs, but it at least demonstrates the structure
https://gitlab.com/mike06/public/apigee-paraller-deploy-sample

Failure is random, but on thread value 4 or higher it should be almost 100%.

[ssvaidyanathan]

Brilliant.. I will use this to test it out

[mikezerosix]

I updated the sample little bit. I also added the Maven warning about threads

[WARNING] *****************************************************************
[WARNING] * Your build is requesting parallel execution, but this *
[WARNING] * project contains the following plugin(s) that have goals not *
[WARNING] * marked as thread-safe to support parallel execution. *
[WARNING] * While this /may/ work fine, please look for plugin updates *
[WARNING] * and/or request plugins be made thread-safe. *
[WARNING] * If reporting an issue, report it against the plugin in *
[WARNING] * question, not against Apache Maven. *
[WARNING] *****************************************************************
[WARNING] The following plugins are not marked as thread-safe in proxynn:
[WARNING] io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.5.0
[WARNING]
[WARNING] Enable debug to see precisely which goals are not marked as thread-safe.
[WARNING] *****************************************************************

[ssvaidyanathan]

@mikezerosix - I cloned your repo
I had to make some changes for example

• add <apigee.apiversion>v1</apigee.apiversion> to main pom.xml
• add <module>proxy18</module> to the list of modules in the main pom.xml
• update the <BasePath> in Proxy04 module.
• remove the apigee.googletoken.email and oauthClientId from the profile in the main pom.xml

I ran the test with -T4C and also with -T10C, I saw errors popping out mainly 503s from the Apigee APIs (I think its rate limiting) when the plugin is polling for the Deployment state

However, when I ran the same test with <apigee.options>async</apigee.options>, it worked without any issues

The entire build ran in 10 seconds

[INFO] Reactor Summary for apigee-maven-parallel 0.1.0-SNAPSHOT:
[INFO] 
[INFO] apigee-maven-parallel .............................. SUCCESS [  0.910 s]
[INFO] proxy01 ............................................ SUCCESS [  4.447 s]
[INFO] proxy02 ............................................ SUCCESS [  4.220 s]
[INFO] proxy03 ............................................ SUCCESS [  8.624 s]
[INFO] proxy04 ............................................ SUCCESS [  5.868 s]
[INFO] proxy05 ............................................ SUCCESS [  7.956 s]
[INFO] proxy06 ............................................ SUCCESS [  3.782 s]
[INFO] proxy07 ............................................ SUCCESS [  5.510 s]
[INFO] proxy08 ............................................ SUCCESS [  7.240 s]
[INFO] proxy09 ............................................ SUCCESS [  4.837 s]
[INFO] proxy10 ............................................ SUCCESS [  3.058 s]
[INFO] proxy11 ............................................ SUCCESS [  5.210 s]
[INFO] proxy12 ............................................ SUCCESS [  6.941 s]
[INFO] proxy13 ............................................ SUCCESS [  6.275 s]
[INFO] proxy14 ............................................ SUCCESS [  7.607 s]
[INFO] proxy15 ............................................ SUCCESS [  6.581 s]
[INFO] proxy16 ............................................ SUCCESS [  9.361 s]
[INFO] proxy17 ............................................ SUCCESS [  2.542 s]
[INFO] proxy18 ............................................ SUCCESS [  3.417 s]
[INFO] proxy19 ............................................ SUCCESS [  8.312 s]
[INFO] proxy20 ............................................ SUCCESS [  8.954 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  10.383 s (Wall Clock)
[INFO] Finished at: 2024-05-29T17:52:04-07:00
[INFO] ------------------------------------------------------------------------

I think you can ignore the threadsafe warning. Thats not really impacting the plugin.

With the default "override" option, the plugin is polling the deployments API and if the response from the Apigee API is 503, then the plugin just exits and marks the build as Failed.. But the proxy is eventually getting deployed.

What I tried is pipe the mvn response to a log file

mvn clean install -P destination1 -Dbearer="$(gcloud auth print-access-token)" -T10C > a.log

Then looked at the log in detail

[mikezerosix]

@ssvaidyanathan
I added your fixes to repo. Thanks.

I forgot to mention that parallel deploy run might not fail on empty apigee. Because errors were about revisions. When the deploy actually overrides the existing revisions, it should fail. I have once successfull ran maven parallel with -T1C on 2 core cpu for 16 proxies and flows. But that could just been because the other core was buzy and never ran the second thread. As re-running it failed. Also most times error seems to come from flows, but proxies do fail also. If you can not reproduce the problem with this sample, I can add flows to it.

I know that warning is not critical, I added it just for information.

I would be interested to know how long does that take to run in default override option, in single thread. It should be something like 40 minutes, even though the proxies have no policies. I have no Apigee that I can run this sample myself as I do not want to mess around in customer's enviroment.

We tried asýnc option and it run the project without problems. But that async does not meet our requirements. We need to assert the proxies are successfully deployed. Using async the solution would be just horrible; unreliable and a lot of work to create and maintain
If we used async, we would

1. we would need parse the list of proxies that are supposed to be deployes. (parse the pom.xml at deploy time to prevent human error.
2. query currently deployed revision of each of those proxies
3. store this information for later
4. run deploy async
5. create ad hoc poller to check all those proxies and flows for status and deployed revision.
6. compare these to stored revisions
7. hope and pray one one else deloys any of those proxies while this one hour long process is running !

We could do the work, but the fact that it would be unreliable is a deal breaker.

[ssvaidyanathan]

Got it.. The issue with "override" is that the Apigee APIs are rate limiting.. So you really cannot do anything in the plugin.
I tried the async multiple times so that new revisions of the proxy gets created. It always worked.

But I was always having issues with "override" since the Apigee APIs were rate limiting and that was causing failures. Dont think adding flows or policies would matter here.

Let me think about your mvn apigee-edge-maven-plugin:POLL-ASYNC-DEPLOYMENTS option. The issue here is - its the same set of GET API that I will have to call for polling which is used with the "override". So not sure how this will help.

[mikezerosix]

@ssvaidyanathan About ratelimit, what about adding plugin option for poll rate ?
At what rate does the plugin now poll Apigee ?
Does Apigee give specific error on rate limit exceeded, so could pluging throttle & wait when receiving rate limit exceed error and resume polling ?

[mikezerosix]

There is a throttling trick I have used in polling, instead of running immediately and sleeping for pollRate. I start the poll on
even clock seconds of the pollRate. Setting different projects pollRate to prime numbers, you can get five ticks that never tick at same time.
For example is I would set 4 sub modules <apigee.poll.rate>31</apigee.poll.rate>
another 4 to <apigee.poll.rate>33</apigee.poll.rate>, another 4 to <apigee.poll.rate>35</apigee.poll.rate> and so on; 37, 39

In Java language:
Instead of doing
sleet(31) ...
Might cause all pols to tick at same time even if rate is different

But doing:
cronSchedule("0/31 * * * * ?") ...

Guarantees that with 20 divided to prime pollRates the parallel polls to only ever run 4 polls per 3 seconds and 20 polls a minute.

[ssvaidyanathan]

@mikezerosix - did u close this issue by mistake?

[mikezerosix]

yes, I closed by mistake