ddos ing the metadata server

[benofben]

It looks like the CFT runs about against ddos protections for the metadata service at scale (100 nodes):

Complete!
This node is part of the autoscaling group that contains the rally point.

An error occurred (RequestLimitExceeded) when calling the DescribeTags operation (reached max retries: 4): Request limit exceeded.

An error occurred (RequestLimitExceeded) when calling the DescribeInstances operation (reached max retries: 4): Request limit exceeded.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 41 100 41 0 0 80078 0 --:--:-- --:--:-- --:--:-- 41000
Using the setting

[benofben]

This is subtle --- the timeout is happening for the tags operation. That's causing the script to pick the wrong rally point. It guessed badly and picked a non-inited node which caused it to block until I added that manually.

n1 oniguruma.x86_64 0:5.9.1-3.1.2.amzn1

Complete!
This node is part of the autoscaling group that contains the rally point.

An error occurred (RequestLimitExceeded) when calling the DescribeTags operation (reached max retries: 4): Request limit exceeded.

An error occurred (RequestLimitExceeded) when calling the DescribeTags operation (reached max retries: 4): Request limit exceeded.

An error occurred (RequestLimitExceeded) when calling the DescribeTags operation (reached max retries: 4): Request limit exceeded.

An error occurred (RequestLimitExceeded) when calling the DescribeTags operation (reached max retries: 4): Request limit exceeded.

An error occurred (RequestLimitExceeded) when calling the DescribeTags operation (reached max retries: 4): Request limit exceeded.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 42 100 42 0 0 6825 0 --:--:-- --:--:-- --:--:-- 7000
Using the settings:
adminUsername 'couchbase'
adminPassword 'foo123!'
services 'data,index,query,fts'
stackName 'prebaked'
rallyPublicDNS 'ec2-54-85-58-168.compute-1.amazonaws.com'
region 'us-east-1'
instanceID 'i-08c7ff49764abe9df'
nodePublicDNS 'ec2-34-207-221-187.compute-1.amazonaws.com'
Running couchbase-cli node-init
node-init output 'SUCCESS: Node initialized'
Running couchbase-cli server-add
server-add output 'ERROR: Cluster is not initialized, use cluster-init to initialize the cluster'
server-add output 'ERROR: Cluster is not initialized, use cluster-init to initialize the cluster'
server-add output 'ERROR: Cluster is not initialized, use cluster-init to initialize the cluster'
server-add output 'ERROR: Cluster is not initialized, use cluster-init to initialize the cluster'
server-add output 'ERROR: Cluster is not initialized, use cluster-init to initialize the cluster'
server-add output 'ERROR: Cluster is not initialized, use cluster-init to initialize the cluster'

[benofben]

It's here: https://github.com/couchbase-partners/amazon-cloud-formation-couchbase/blob/master/scripts/util.sh#L57

[benofben]

One solution would be to grab all the instance data and then loop through locally. That might lead to some really complex logic though.