Hawtio-Online installs are displaying an nginx "429 Too Many Requests" error

[phantomjinx]

Consider how to modify the rate limiting of the web server.

[phantomjinx]

Logs from app:

2023/12/20 06:09:35 [error] 23#23: *58720 limiting connections by zone "limitconnbyaddr", client: 172.17.45.21, server: localhost, request: "GET /maste
r/api/v1/namespaces/openshift-operators/pods?watch=true HTTP/1.1", host: "hawtio-online-hawtio.hawtio-cluster-XXXXX"

172.17.45.21 - - [20/Dec/2023:06:09:35 +0000] "GET /master/api/v1/namespaces/openshift-operators/pods?watch=true HTTP/1.1" 429 564 "-" "Mozilla/5.0 (X1
1; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" "10.244.0.5"

2023/12/20 06:09:35 [error] 23#23: *58721 limiting connections by zone "limitconnbyaddr", client: 172.17.11.234, server: localhost, request: "GET /mast
er/api/v1/namespaces/xxxxx-cert-store/pods?watch=true HTTP/1.1", host: "hawtio-online-hawtio.hawtio-cluster-XXXXX"

172.17.11.234 - - [20/Dec/2023:06:09:35 +0000] "GET /master/api/v1/namespaces/xxxxx-cert-store/pods?watch=true HTTP/1.1" 429 564 "-" "Mozilla/5.0 (X11; L
inux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" "10.244.0.4"

2023/12/20 06:09:35 [error] 23#23: *58722 limiting connections by zone "limitconnbyaddr", client: 172.17.11.234, server: localhost, request: "GET /mast
er/api/v1/namespaces/hawtio/pods?watch=true HTTP/1.1", host: "hawtio-online-hawtio.hawtio-cluster-XXXXX"

172.17.11.234 - - [20/Dec/2023:06:09:35 +0000] "GET /master/api/v1/namespaces/hawtio/pods?watch=true HTTP/1.1" 429 564 "-" "Mozilla/5.0 (X11; Linux x86
_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" 

[phantomjinx]

Trying to implement limitations, either through max connections by IP or rate limiting causes parts of the application to fail in different ways. For example, rate limiting to 40 connections / second seems like a good solution (even inserting only in the /online location) yet opening up the app in 5 different tabs kills the 5th tab.

Implementation of any conn/rate limiting is going to require a much more sophisticated approach if it is indeed necessary.

@tadayosi glad for your input on this.

[phantomjinx]

Created #273 in the interim to workaround the too many requests errors.

[phantomjinx]

Pushing to subsequent milestone.