Output visualizer on interactive/multipass problem

[Matistjati]

What is the expected behavior when we run output visualizers on interactive or multipass problems? I can see two reasonable alternatives for interactive:

• Run them with an empty file as .ans. Write any info you need to the feedback directory
• For interactive, the ans file is the complete log of the contestant submissions' output during the interaction. I dislike this one, since it's a pain to implement. Maybe we could send the full communication log, but I would argue that that isn't the .ans file anymore.
• For multipass, the last .ans file produced
• Don't run them at all

I personally prefer option 1, especially given that there is no "obvious expected behavior" for multipass and interactive. I think there's a fair argument for option 3, but I'd much rather for it to blow up hard by feeding an empty .ans than the judge maybe, maybe not triggering a bug in their program. Any other thoughts?

[mzuenni]

I think we should not support an OutputVisualizer for interactive problems simply because there is no output that we could feed into it.

For multipass IMO it should be [last input] [original ans file] < [last output] and it can read the feedbackdir. More or less like for other problems.

Note that we can't simply pass an empty .ans file as second argument because the .ans file may should already exists and could contain required data. (The .ans file must not -- and this case can't -- be valid output)

[Matistjati]

When I made this issue, I had mixed up ans file vs contestant output. Talked with @niemela , and we came to the following conclusions:

• Interactive problems shall be run with an empty file as contestant output. If there is no .ans file, an empty one shall be created
• Multipass problems shall be run as @mzuenni suggested, with [last input] [original ans file] < [last output]

In this setup, everything is still possible by writing appropriate information to the feedback directory.

A follow-up question, should you be able to specify a limit on how long an output visualizer can run/how much memory it uses, or should it be up to the judge system? I personally think it's fine for the judge system to set reasonable defaults for this.

Also, I think we should explicitly specify how they will be called. The current format for output validators is <output_validator_program> input_file answer_file feedback_dir [additional_arguments] < team_output [ > team_input ]. However, there I don't see any way to specify additional_arguments, and team_input will never be used.
<output_validator_program> input_file answer_file feedback_dir < team_output is IMO much clearer.

[thorehusfeldt]

I think we should not support an OutputVisualizer for interactive problems simply because there is no output that we could feed into it.

Can’t we just specify that the output_visualizer be run just like the output_validator? It can then produce an image. I am happy to write an interactive output visualizer that does exactly that, say for guess. Such an output_visualizer would (for an interactive problem) to a large extent mimik the output_validator, but produce an image (rather than judgemessage.txt etc.)

In practice, if I ever made a problem like that, I’d probably put most of the code into output_validator and then symlink to it from output_visualizer and call it with an argument like --visualize.

As observed elsewhere we do indeed need to be able to specify output_visualizer_args, preferably next to {input, output}_validator_args.

[mzuenni]

I think we should not support an OutputVisualizer for interactive problems simply because there is no output that we could feed into it.

Can’t we just specify that the output_visualizer be run just like the output_validator? It can then produce an image. I am happy to write an interactive output visualizer that does exactly that, say for guess. Such an output_visualizer would (for an interactive problem) to a large extent mimik the output_validator, but produce an image (rather than judgemessage.txt etc.)

In practice, if I ever made a problem like that, I’d probably put most of the code into output_validator and then symlink to it from output_visualizer and call it with an argument like --visualize.

I think this on itself does not really make sense. Submissions can be random, so the visualizer could get a different interaction than the validator. Also, the visualizer would be a complete validator, just with extra steps... so in that scenario it would be way easier to simply allow the validator to produce an image?

• Interactive problems shall be run with an empty file as contestant output. If there is no .ans file, an empty one shall be created
• [...]

In this setup, everything is still possible by writing appropriate information to the feedback directory.

Yes that works (not sure how useful it is though).

[thorehusfeldt]

Also, the visualizer would be a complete validator, just with extra steps... so in that scenario it would be way easier to simply allow the validator to produce an image?

Yes, I think that is what will happen. This phenomenon is not qualitatively different from what we already accept for noninteractive output visualization.

A good output_visiualizer of, say, convex hull problem would need to parse the points, check if they describe a polygon, draw the polygon, and (ideally) highlight in red the points described in .in that are outside the polygon. It then could also visualize the solution described by .ans (in green, say) and the polygon described by .out, possibly highlighting (in red) the corners that are not convex. That’s almost the same work that output_validator needs to also do. (If I were to implement either, I’d try to share as much code as I can.)

So: output visualizers and output validators (no matter the mode of interaction) share many features. We should consider invoking them very similarly, the difference is just which args are passed to them.

[thorehusfeldt]

OK, changed my mind and now think that an output visualizer for an interactive problem need not itself be interactive. It suffices to have access to judgemessage.txt. Here’s an example judgeimage for guess:

And the vlsualizer that produces such images is here: https://github.com/RagnarGrootKoerkamp/BAPCtools/tree/draft-visualizer/test/problems/guess/output_visualizer_disabled