Improvement: make startup resilient to bad file paths

[gabeschine]

Current:

• If a config YML references a bad file path, ie for an .mbtiles or .pmtiles file, startup fails completely.
• In contrast, if the config references a bad Postgres table, startup warns but continues to initialize & serve the remaining sources.

Desired state:

• Consistently ignore bad sources, allow startup to finish, and serve good known sources.

Reasoning:

• Better production resilience: a single bad path should not "take down" the serving infrastructure.

[gabeschine]

See: #1689 for relevant context and code patterns.

[CommanderStorm]

Once configurable, the default should likely be warn.

Aborting might still be valuable, especially in CI 😉

CC @nyurik because I think this was your decision.

[gabeschine]

Once configurable, the default should likely be warn.

Aborting might still be valuable, especially in CI 😉

CC @nyurik because I think this was your decision.

Copied from Slack discussion:

I opted to go simple, and not provide any configuration, for this round. I noticed the PR you sent me was never merged - while I appreciate the ability to configure what to do on a config with validation problems, my view is that the default for running martin should be a best-effort initialization (warn, not abort).

[nyurik]

martin foo.mbtiles -- should it fail to start if foo.mbtiles does not exist, i.e. nothing to serve? What if the connection is a DB server that may later come back up? What about the dev experience vs production? I think dev experience should be simpler - if you gave wrong params, fail right away. In a prod, devops might be ok to configure a bit more complex configuration, e.g. start and continuously refresh. Lastly, if a file is missing from a local disk, the chances of it becoming available are much smaller than for a DB server or an S3 bucket to come back up / get the needed file.

In a devops env, e.g. kubernetes, the best "signal" of something is bad is for the service to crash - k8s would automatically restart the failed pod, often fixing the problem. If a service starts but fails to deliver needed results may require complex configuration using liveness / rediness / startup probes.

In short, I don't have a magical answer. We should make on-boarding experience simple, i.e. crash rather than show some warning. And yet we should account for the long tail of the edge cases that are possible. This may also intertwine with the automtaic refresh - #288

[gabeschine]

martin foo.mbtiles -- should it fail to start if foo.mbtiles does not exist, i.e. nothing to serve?

My view: if there is nothing to serve (ie, all sources failed to resolve), then a hard failure is always appropriate.

What if the connection is a DB server that may later come back up? What about the dev experience vs production? I think dev experience should be simpler - if you gave wrong params, fail right away. In a prod, devops might be ok to configure a bit more complex configuration, e.g. start and continuously refresh. Lastly, if a file is missing from a local disk, the chances of it becoming available are much smaller than for a DB server or an S3 bucket to come back up / get the needed file.

I fully agree and appreciate the nuances here. This PR is intending to be a baby step towards consistency across the source types as they are today.

For my organization's use-case, we have tiles served via Postgres, and via PMTiles from S3 (soon). This configuration allows the Martin instances to be stupid immutable horizontally-scaled dyno instances, and lets us update the source(s) asynchronously from our data ingest pipeline by publishing to S3 or to Postgres. For production serving, we would never want an outage in one region of AWS's S3 infrastructure to take down the entire tiles serving stack, just because one of the PMTiles files is hosted there.

My opinion is that until more dimensions of configuration are added, the policy for Martin should be graceful degradation.

In a devops env, e.g. kubernetes, the best "signal" of something is bad is for the service to crash - k8s would automatically restart the failed pod, often fixing the problem. If a service starts but fails to deliver needed results may require complex configuration using liveness / rediness / startup probes.

In short, I don't have a magical answer. We should make on-boarding experience simple, i.e. crash rather than show some warning. And yet we should account for the long tail of the edge cases that are possible. This may also intertwine with the automtaic refresh - #288

Agreed. Runtime polling/refresh is definitely attractive when coupled with graceful degradation. This way, a dev-ops engineer does not need to intervene to recover a degraded system.

If it's good with you, I'd like to land this PR more or less in the shape it's in, and I would be happy to further discuss the topic and come up with a plan everyone can get behind.