code 107: Database connection error.

[Innixma]

From runbenchmark.20201223T011330.log file saved to S3 (same error occurred on all datasets)

[DEBUG] [amlb.benchmark:01:13:30.196] Using constraint definition: { 'cores': 8,
  'folds': 10,
  'max_runtime_seconds': 3600,
  'min_vol_size_mb': 1000000,
  'name': '1h8c'}.
[INFO] [amlb.benchmarks.openml:01:13:30.196] Loading openml suite 269.
[ERROR] [amlb:01:13:30.498] https://www.openml.org/api/v1/xml/study/269 returned code 107: Database connection error. Usually due to high server load. Please wait for N seconds and try again. - None
Traceback (most recent call last):
  File "runbenchmark.py", line 118, in <module>
    bench = amlb.Benchmark(args.framework, args.benchmark, args.constraint)
  File "/repo/amlb/benchmark.py", line 75, in __init__
    self.benchmark_def, self.benchmark_name, self.benchmark_path = rget().benchmark_definition(benchmark_name, self.constraint_def)
  File "/repo/amlb/resources.py", line 181, in benchmark_definition
    hard_defaults, tasks, benchmark_path, benchmark_name = benchmark_load(name, self.config.benchmarks.definition_dir)
  File "/repo/amlb/benchmarks/parser.py", line 19, in benchmark_load
    benchmark_name, benchmark_path, tasks = load_oml_benchmark(name)
  File "/repo/amlb/benchmarks/openml.py", line 38, in load_oml_benchmark
    suite = openml.study.get_suite(oml_id)
  File "/repo/venv/lib/python3.7/site-packages/openml/study/functions.py", line 29, in get_suite
    suite = cast(OpenMLBenchmarkSuite, _get_study(suite_id, entity_type="task"))
  File "/repo/venv/lib/python3.7/site-packages/openml/study/functions.py", line 71, in _get_study
    xml_string = openml._api_calls._perform_api_call(call_suffix, "get")
  File "/repo/venv/lib/python3.7/site-packages/openml/_api_calls.py", line 62, in _perform_api_call
    __check_response(response, url, file_elements)
  File "/repo/venv/lib/python3.7/site-packages/openml/_api_calls.py", line 143, in __check_response
    raise __parse_server_exception(response, url, file_elements=file_elements)
openml.exceptions.OpenMLServerException: https://www.openml.org/api/v1/xml/study/269 returned code 107: Database connection error. Usually due to high server load. Please wait for N seconds and try again. - None

This was caused by the following:

python runbenchmark.py AutoGluon openml/s/269 1h8c -m aws -f 0 -p 100

Note: This is on a fork of the automlbenchmark repo where I have set max p to 100.

Running test datasets works correctly.

When I run the classification datasets, ~45 of the 66 datasets fail with similar errors, while the others succeed.

python runbenchmark.py AutoGluon openml/s/271 1h8c -m aws -f 0 -p 100

This used to work fine in previous versions of automlbenchmark (From May 28th) where I would sometimes even use -p 400 equivalents with the original 39 datasets.

It seems odd that it is trying to fetch the study instead of a particular dataset. Why does an EC2 instance need to fetch the study? It is only supposed to be training a single dataset. Also, perhaps it makes sense to have a logarithmic retry loop on accessing the data? This currently seems quite brittle.

[PGijsbers]

It seems odd that it is trying to fetch the study instead of a particular dataset. Why does an EC2 instance need to fetch the study?

It's an artifact from the fact that benchmark definitions were/can be defined locally, and in that setting a task is only uniquely defined within a benchmark file. For this reason the benchmark (in this case an openml study) is loaded locally before determining the task. You're right that we can optimize this, but I'd prefer to wait with this until I can talk with @sebhrusen about this.

Also, perhaps it makes sense to have a logarithmic retry loop on accessing the data?

openml-python should already do this internally (I don't recall if it's logarithmic though).

Could you please try this change? It should reduce the number of calls to the server by a factor of the number of datasets in the benchmark (at the loss of having a text description, which is fine). The one call itself is also light-weight. I don't have the resources you do, so I couldn't verify this does enough to alleviate the problem, though I suspect it does.

[Innixma]

This solved the problem, thanks!

[Innixma]

After some additional runs, I notice that sometimes this issue still occurs, although only for a small portion of the datasets on only a random subset of folds. (This is doing all 10 folds with -p 100). I suspect that when all 10 EC2 instances for a given dataset are accessing the database at the same time, it can cause the error to occur still.

I think this can be solved by a try: except: loop that attempts 10 times to get the data. If a failure was detected, it will try again after sleeping an increasingly long time per failure + a random offset, up to 5 minutes.

[PGijsbers]

Thanks. I know there is something similar already in openml-python already. It makes more sense to me to change the policy there and/or make sure the right openml-python configuration is used.

[mfeurer]

openml-python currently does two retries. You should be able to change this in the OpenML config file by setting connection_n_retries = 5 or something else that's higher than 2. We'll also be changing this in the next release of openml-python.

[Innixma]

@mfeurer Would it make sense to add extra sleep time + random offset for each retry (logarithmic increase) so the servers steadily get fewer requests at a time until it can handle them all? (vs getting DDOSed by many machines doing many retries rapidly)

[mfeurer]

Would it make sense to add extra sleep time + random offset for each retry (logarithmic increase) so the servers steadily get fewer requests at a time until it can handle them all? (vs getting DDOSed by many machines doing many retries rapidly)

That's a good question and we haven't investigated this yet. We currently increase the time in a linear fashion and haven't had any issues when we sufficiently increased the repetitions.

[PGijsbers]

The new openml-python release has a retry_policy which can be set to robot which quickly increases time between retries up to a minute. With #312 that is now used for the benchmark, hopefully that saves the server. Additionally we used some new parameters to avoid downloading qualities.xml altogether. Finally #303 should also reduce some more server spam for benchmarks directly defined as openml suite.

At the moment I believe people are working on production server infrastructure as well. Not entirely sure how noticeable this will be as a user (it might temporarily be worse).

[PGijsbers]

Closing this issue, if you are still experiencing issues please re-open.