Prevention of inconsistent label specifications

[rfloca]

Currently some models have inconsistent label ids if you compare their REST specification and the results.

E.g.:
Inconsistent label ids for deepedit:

• "labels":{"spleen": 1, "right kidney": 2, "left kidney": 3, "liver": 6, "stomach": 7, "aorta": 8, "inferior vena cava": 9, "background": 0}
• However, the nifty file response has the label ids contiguous ie. 0 – 7
• ‘Secret’ request parameter restore_label_idx introduced but remains undocumented.

Incorrect labels specification for localization_spine: Model returns binary mask even though there is a whole lot of labels exposed in the API.

May be we could establish a basic smoke/consistency testing protocol, to catch such problems. E.g. linter/schema checker for the profile JSONs; generate a test result and checking if it is matching to specification. May be with easy tooling and good documentation it would be also easier to motivate devs that contribute a algorithm to provide this simple unit test for a model.