Other ways to export metrics in hyperparameter tuning jobs

[dennis-ec]

Hello,

First of all thank you for implementing Hyperparameter tuning jobs.
This seems like a really useful feature I'd like to use very often.

However, I don't understand why the metrics are collected via logs.
Using regex to search over the logs seems complicated.
The documentation is also confusing without showing an concrete output which will match the shown regex code.

Why are the metrics not gathered via an output json file (in case for custom code)?
E.g. metrics.json which could be created in /op/ml/ouput/

[dennis-ec]

A little Update: This blog post shows a regex expression as example, which seems more reasonable:
'Validation-accuracy=([0-9\\.]+)'

Despite that, the approach to search over logs still seems very inconvenient. So the Feature request for an alternative method still stands.

[leopd]

Thanks for the feedback. To explain the "why", one of our goals was to enable hyperparameter tuning on training code that customers don't necessarily understand or control -- e.g. you download a learning algorithm from github or get something from a colleague, and you aren't necessarily comfortable or even able to change the code, but still want to tune it. Regex's solve this problem nicely. As such, it's compatible with nearly every piece of ML code ever written, although it does bring its own set of challenges like crafting the right regex.

I agree the documentation & examples can be better. I've developed some tools to help with testing & debugging the MetricDefinition regexes. I'll look into how we might release such tools publicly.