Clarify static validators

[Matistjati]

Static validators are going to be implemented in problemtools soon. IMO, there are is one large points of ambiguity:
I think it's clear that a testgroup can contain exactly 0 or 1 static validation test case. If there is a static validation testcase, can there be normal testcases? My guess would be no, and if a testgroup is static, it contains exactly one static testcase and no other testcases. However, why is there the a score key in the static_validation map then? A testdata.yaml already has the property score. At the same time, I see no way to distinguish which testcase is the static one. Whichever way it is, this should really be clarified.

I definitely think there is enough expressive power in not allowing "normal" testcases, given that require_pass exists.

Also, I think we should clarify that all submission files are will be in the same working directory as the static validator. Currently, this is described very far away from static validators in the document. A reference probably suffices. https://www.kattis.com/problem-package-format/spec/2023-07-draft.html#working-directory

When the ambiguity has been clarified, I'll make a PR. @niemela @Tagl

[evouga]

My understanding is that any test group can have a static validation test case (even if it also has regular test cases). This is somewhat desirable as it allows the problem author to lump a static validation test case and all regular test cases inside a single secret test group.

[Tagl]

Yes, the idea was that you can have a static validation test case with regular test cases in the same group.

[niemela]

I talked with @Matistjati earlier, but am also recording the answers here...

Static validators are going to be implemented in problemtools soon. IMO, there are is one large points of ambiguity: I think it's clear that a testgroup can contain exactly 0 or 1 static validation test case.

Correct.

If there is a static validation testcase, can there be normal testcases?

Yes.

My guess would be no, and if a testgroup is static, it contains exactly one static testcase and no other testcases. However, why is there the a score key in the static_validation map then? A testdata.yaml already has the property score. At the same time, I see no way to distinguish which testcase is the static one. Whichever way it is, this should really be clarified.

The static test case is a "virtual" one. I.e. there is no .in file for the static test case.

I definitely think there is enough expressive power in not allowing "normal" testcases, given that require_pass exists.

That's a good point, but that would be quite ugly and cumbersome, you would then create an (otherwise) empty group for the static test case, and that directory (i.e. group) would only contain a test_group.yaml.

Also, I think we should clarify that all submission files are will be in the same working directory as the static validator. Currently, this is described very far away from static validators in the document. A reference probably suffices. https://www.kattis.com/problem-package-format/spec/2023-07-draft.html#working-directory

👍

(Generally, we should add a lot of links to things).

When the ambiguity has been clarified, I'll make a PR. @niemela @Tagl

Sounds great!

[niemela]

@Matistjati Do you have all the clarifications you need?

[Matistjati]

Yes. I will implement static validators in the coming days, so I'm holding off on making the PR until I've actually implemented it.
To clarify: due to the division of work in the group, I'm only implementing the runner code, not the "parse problem into the db" part. (At least not yet). I think the runner part should be entirely non-controversial. If not, please speak up.