feat(mbtiles) - Add validation to martin .mbtiles #741 #1689
Conversation
martin/src/mbtiles/mod.rs
Outdated
| validation_error.to_string(), | ||
| )); | ||
| } | ||
| OnInvalid::Warn => { |
There was a problem hiding this comment.
We might want to do further processing of the validation_error to determine if this is a critical issue or not.
There was a problem hiding this comment.
I keep wondering if "fast" and "skip" are actually the same thing. We should always do some basic validation - like making sure the tiles table exists with the right columns - or else we can't even use it. The only difference is in what we do afterwards - ignore the source or continue with it... So perhaps:
- validation level is
fastvsthorough(no skip) - severity would be
good,warn,error - and the action could be
abortorignore sourceforerror, andabortorignore sourceorwarnonwarning? - we may even introduce
warnings-as-errorsparameter? (slowly building a compiler here :) )
There was a problem hiding this comment.
I really like where you are going with this! I left a few minor comments inline.
A few overall thoughts:
- we may have more than one type of "local source validation" (pmtiles, sprites, fonts, cog files, ...) - so we should keep CLI parameters simple and global (not mbtiles specific)
- all configuration sources may introduce some format-specific validation steps, but we should try to keep them all consistent from configuration file perspective
- we should have multi-level config: CLI / config file root / per source type / specific path/file
- lets cleanup error handling dup code somehow
- as i wrote more in depth in a comment, we may want to have
warning-as-errornon-default mode
overall, awesome work! I think it is pretty close to the working state. One last thing - usually you do not want to submit PRs from the main branch of your fork, but that's really a minor thing :)
martin/src/mbtiles/mod.rs
Outdated
| validation_error.to_string(), | ||
| )); | ||
| } | ||
| OnInvalid::Warn => { |
There was a problem hiding this comment.
I keep wondering if "fast" and "skip" are actually the same thing. We should always do some basic validation - like making sure the tiles table exists with the right columns - or else we can't even use it. The only difference is in what we do afterwards - ignore the source or continue with it... So perhaps:
- validation level is
fastvsthorough(no skip) - severity would be
good,warn,error - and the action could be
abortorignore sourceforerror, andabortorignore sourceorwarnonwarning? - we may even introduce
warnings-as-errorsparameter? (slowly building a compiler here :) )
|
hi @Auspicus, let me know if you need any help moving it forward, seems like a really good feature to have in the longer term |
|
Hey @nyurik! Thanks for checking in. I'm working on this in the interest of using Martin as a replacement for MapTiler Server for my day job. Just had to pause for a bit to tackle some other things that have come up but should be able to get back to this next week, hopefully. Cheers! |
|
Thx! You might be interested in my work on https://github.com/nyurik/maplibre-native-rs - ability to do server-side rendering... still some time away, but already shows promising results |
Wow. Really excited for this! Sadly I know nothing about cpp ( need some cpp knowledge to get involved in I guess?) |
…to new_sources_url in case tile sources from URLs produce this error
for more information, see https://pre-commit.ci
…he source to result if Ok and returns all other errors
for more information, see https://pre-commit.ci
| .collect::<TileInfoSources>(); | ||
|
|
||
| let mut sources_to_prune: Vec<usize> = vec![]; | ||
| for (idx, source) in resolved_sources.iter().enumerate() { |
There was a problem hiding this comment.
need to do parallel validation rather than this.
| let cfg = serde_yaml::from_str::<FileConfigEnum<MbtConfig>>(indoc! {" | ||
| let cfg: FileConfigEnum<MbtConfig> = | ||
| serde_yaml::from_str::<FileConfigEnum<MbtConfig>>(indoc! {" | ||
| validate: thorough |
There was a problem hiding this comment.
might want to make consistent with validation_level name
|
Hey @nyurik I've adjusted this one a bit to be more in line with what you were talking about:
I don't think it's 100% over the line yet but before I polish this off I'd like to know if this general direction makes sense to you. Let me know. Cheers! 🍻 |
closes #741