Example for output_visualizer is missing

[vmcj]

I would expect an example for the output_visualizer in the maximal problem. This is currently missing. Can this be added before the draft becomes official?

[niemela]

@RagnarGrootKoerkamp Could you add an example of visualizers?

[RagnarGrootKoerkamp]

I'll pass it on to either @mpsijm or @mzuenni ; don't have sufficient time at the momen.

[mpsijm]

Coincidentally, I was chatting about this with @mzuenni last week 😄 I think I'd like to have a working implementation in BAPCtools as well before the draft becomes final (see also RagnarGrootKoerkamp/BAPCtools#433), but we weren't 100% sure about some things. Forgot to actually ask the people here, but let's do that now then 😛 I'll just write from the top of my head what BAPCtools should do with visualizers (both input and output), and then I'd like to hear from others whether that's indeed what they expect tooling to do, based on the spec:

• The input visualizer will work roughly the same as the visualizer: key we currently have in generators.yaml. The spec does not prescribe how this program should be called, so for the simplest implementation, we can simply keep the same interface but simply remove the visualizer: key from generators.yaml and always look for a visualizer Program in input_visualizer/ instead.
  • The input visualizer is run as part of bt generate, after generating <tc>.in and <tc>.ans files, and the resulting images (<tc>.png/<tc>.svg/etc.) also end up in data/.
  • Q1: Note that currently, the input visualizer also has access to the answer file. Is that reasonable? @mzuenni thought that in that case, it would be more like a test_case_visualizer rather than an input_visualizer.
• The output visualizer will work roughly the same as a combination of the input visualizer and the output validator, and it should generate {team,judge}image.<ext> files.
  • When running bt run, the image files stay in the feedback directory of $(bt tmp)/runs/<submission>/<test_case>/, and do not end up in data/.
  • Q2: if we want to use the output visualizer to generate images for submission output, then we may as well use the same program to generate an image for the canonical jury answer (i.e. the .ans file) during bt generate. If so, then we would like this to end up in data/ as well, but that potentially gives name clashes with images generated by the input visualizer. Even when the input visualizer does not have access to the .ans file, it may still be the case that we want to visualize both the input and the answer/output. If so, then how do we resolve the name conflict of the image files in data/? For example, we could have both <tc>.in.<ext> and <tc>.ans.<ext> (with <ext> from png/svg/etc.)?

[niemela]

Coincidentally, I was chatting about this with @mzuenni last week 😄

Great!

I think I'd like to have a working implementation in BAPCtools as well before the draft becomes final

I'm aiming to declare the draft feature final at the end of the month. (We'll see how that goes...)

(see also RagnarGrootKoerkamp/BAPCtools#433), but we weren't 100% sure about some things. Forgot to actually ask the people here, but let's do that now then 😛 I'll just write from the top of my head what BAPCtools should do with visualizers (both input and output), and then I'd like to hear from others whether that's indeed what they expect tooling to do, based on the spec:

• The input visualizer will work roughly the same as the visualizer: key we currently have in generators.yaml. The spec does not prescribe how this program should be called,

We definitely should specify how if should be called. My strong preference would be to follow the pattern from the input/static/output validators, so probably something like:
<input_visualizer_program> feedback_dir [additional_arguments] < inputfile
?

• so for the simplest implementation, we can simply keep the same interface but simply remove the visualizer: key from generators.yaml and always look for a visualizer Program in input_visualizer/ instead.
  • The input visualizer is run as part of bt generate, after generating <tc>.in and <tc>.ans files, and the resulting images (<tc>.png/<tc>.svg/etc.) also end up in data/.

👍

• • Q1: Note that currently, the input visualizer also has access to the answer file. Is that reasonable? @mzuenni thought that in that case, it would be more like a test_case_visualizer rather than an input_visualizer.

Is there any other reason why an input visualizer would need the answer file, then that it want to produce a visualization that indicates the correct solution as well?

My though was always that the input visualizer would not indicate the solution, since you could (should?) run the output visualizer on the answer file (or .out or .ans.statement ?) to get a visualization of the expected solution.

That said, the intended audience for input visualizations are judges, teachers, coaches, spectators, and such, not contestants and students, so it's not obviously broken for it to be more of a "test case visualization" and include the solution.

I would prefer a pure input visualization over a test case visualization though.

• The output visualizer will work roughly the same as a combination of the input visualizer and the output validator, and it should generate {team,judge}image.<ext> files.

  • When running bt run, the image files stay in the feedback directory of $(bt tmp)/runs/<submission>/<test_case>/, and do not end up in data/.

👍

• • Q2: if we want to use the output visualizer to generate images for submission output, then we may as well use the same program to generate an image for the canonical jury answer (i.e. the .ans file) during bt generate. If so, then we would like this to end up in data/ as well, but that potentially gives name clashes with images generated by the input visualizer. Even when the input visualizer does not have access to the .ans file, it may still be the case that we want to visualize both the input and the answer/output. If so, then how do we resolve the name conflict of the image files in data/? For example, we could have both <tc>.in.<ext> and <tc>.ans.<ext> (with <ext> from png/svg/etc.)?

Wouldn't it make more sense for the tool running the output visualizer to move the resulting filer from feedback_dir/judgeimage.<ext> (would we also use the team image?) to data/? The feedback_dir is always temporary anyway, it's just "where the ouput goes".

[mzuenni]

Is there any other reason why an input visualizer would need the answer file, then that it want to produce a visualization that indicates the correct solution as well?

the question is: what do we want. I think visualizing a test case can very much mean visualizing the solution and in the past this has shown to be very useful. Same goes for the "output" visualizer, do we want it to have access to the jury solution? I would again think that this can be useful.

IMO it would be nicer to have:

• a testcase_visualizer that gets .in and .ans and produces one image. It is then up to the implementation if it visualizes the solution or just the input.
• an output_visualizer that gets access to .in, ans and the team output. Again, the visualizer can decide what should be shown.

[niemela]

Is there any other reason why an input visualizer would need the answer file, then that it want to produce a visualization that indicates the correct solution as well?

the question is: what do we want.

Agreed, that is the more important question, and I was discussing it in the section after what you quoted :). My point here was "even if we want a pure input visualization, would there still be a reason for the input visualizer to have the answer file". But, you're right, we should answer the other question first.

I think visualizing a test case can very much mean visualizing the solution and in the past this has shown to be very useful.

Makes sense.

Same goes for the "output" visualizer, do we want it to have access to the jury solution? I would again think that this can be useful.

I already does, it uses the same arguments as the output validator, and I agree.

IMO it would be nicer to have:

• a testcase_visualizer that gets .in and .ans and produces one image. It is then up to the implementation if it visualizes the solution or just the input.

• This could still be called an input visualizer. (And I think it should be)
• I don't think we should just leave the expected purpose unspecified. If we think it's reasonable/acceptable/useful for this to visualize the solution as well, we should say so.
• The output visualizer can produce both a team and judge image, could that be the difference between "with solution and without"?
• So you're suggesting something like <input_visualizer_program> input_file answer_file feedback_dir [additional_arguments]? Mirroring the output validator rather than the input validator?

What would the difference be between what is generated by this input visualizer that gets the solution, versus running the output visualizer on the correct solution?

• an output_visualizer that gets access to .in, ans and the team output. Again, the visualizer can decide what should be shown.

The current spec for the output visualizer, which includes what you say.

[mzuenni]

So you're suggesting something like <input_visualizer_program> input_file answer_file feedback_dir [additional_arguments]? Mirroring the output validator rather than the input validator?

yes. (except that output validator is also called with < team output, so its more like something inbetween)

What would the difference be between what is generated by this input visualizer that gets the solution, versus running the output visualizer on the correct solution?

Their purpose is entirely different. The input_visualizer wants to tell me something about the test case. The other one is more focused on the submission output.

A random example would be that the input_visualizer could give me a plot of the degree sequence of the graph (just some random statictic) and the output_visualizer shows me the diff between jury and team output.

[niemela]

Ok, that works.

Let me verify something else. We all agree that "what the input visualizer produces" is exactly the "Illustrations" described in the test data section, right? Meaning that the expectations for the latter should apply to the output of the former. Specifically:

• Illustration files are meant to be privileged information.
• An illustration provides a visualization of the associated test case.

If we have more expectations (for example: "an illustration could contain information indicating the expected solution"), we should add that. Furthermore, since we are discussing this, it is obviously not obvious, so we should definitely (I wanted to say "obviously") specify it one way or the other.

[RagnarGrootKoerkamp]

So to clarify, there are three types of generated illustrations:

• input/testcase illustrations, stored as eg data/secret/*.png, not shown to teams. I don't really care what we call it, but clarifying that it illustrates the testcase as a whole, including possible an answer, sgtm.
• output illustrations, written to {team,judge}image.png. One is shown to teams, one is not.

I'm a bit confused now about public illustrations:

• We do not have public input/testcase illustrations.
• We do have public output illustrations.

But what is the expected use case of teamimage.png? Should it only be generated for samples? Or always generated and only shown for samples? Can input/testcase illustrations be shown for samples?

It seems that the public/private figure hinges more on whether the testcase is a sample, than what is rendered in the figure.

• Is there a use case for showing teamimage.png for secret cases?
• Can we instead consolidate teamimage.png and juryimage.png into one and only show it for samples (if enabled/supported)?
• Are there cases where we want to not leak the juryimage.png for the samples? I think that can be, eg when we draw some hints towards the solution.

[niemela]

I'm a bit confused now about public illustrations:

Sorry if I caused this confusion.

• We do not have public input/testcase illustrations.

Correct.

• We do have public output illustrations.

Also correct. This would be the teamimage.<ext>.

But what is the expected use case of teamimage.png? Should it only be generated for samples?

It should be for all full_feedback cases, which by default is the samples.

Or always generated and only shown for samples?

If a tree falls...?

Does it matter whether it's generated, if it's not used? This seems like an implementation detail of a system using the problem package.

Can input/testcase illustrations be shown for samples?

Great question, that we have not answered (or even thought that much about). I would prefer if the answer is "yes". But a good argument can certainly be made for "no".

• Is there a use case for showing teamimage.png for secret cases?

To judges/teachers? Maybe a choice could be made to show teamimage.<ext> (but not judgeimage.<ext> to spectators?

• Can we instead consolidate teamimage.png and juryimage.png into one and only show it for samples (if enabled/supported)?

Good point. 👍

• Are there cases where we want to not leak the juryimage.png for the samples? I think that can be, eg when we draw some hints towards the solution.

But this would be hints that are not conveyed by getting both .in and .ans (or .out?). What kind of information could that be? Why would we want that in judgeimage.<ext>?

[RagnarGrootKoerkamp]

But this would be hints that are not conveyed by getting both .in and .ans (or .out?). What kind of information could that be? Why would we want that in judgeimage.?

Maybe something like: the figure illustrates the steps of Dijkstra's algorithm towards finding a shortest path, rather than just showing the shortest path. So you would show the solution to the team, but not the way the solution is obtained.

It should be for all full_feedback cases, which by default is the samples.

Makes sense 👍

I don't have strong opinions on this, but I think it would be good to make a little list of some usecases we want to support and what exactly their requirements are. E.g., does one ever want to actually generate both teamimage and judgeimage at the same time? Otherwise, this could also be a setting in problem.yaml (output_visualizations_are_private: bool). (I don't really think that's better, but who knows.)

[niemela]

This is also somewhat related to my comment in #393.

If output validators get access to full_feedback, then input/output visualizer should as well. then they could choose to not leak privileged information in those cases. What I'm saying is, that if visualizers know when a case is full_feedback, we might not need separate team and judge images?

[mpsijm]

To aid in the discussion, RagnarGrootKoerkamp/BAPCtools#438 implements the visualizers as they are currently written in the draft:

• The input visualizer only reads the .in file from stdin.
  • It writes testcase.<ext>. This is currently not specified, but the closest thing to the implementation that we already had.
• The output visualizer is executed in the same way as the output validator: in/ans/feedbackdir as args, output is passed to stdin.
  • It writes image files to the feedbackdir, which are then left untouched (but from the discussion above, we may want to do something for the visualizations of the answer, a.k.a. the canonical jury output).

[niemela]

• The input visualizer only reads the .in file from stdin.

Could we specify this as "same interface as input validators"? Would it make sense to give it the same arguments as the input validator?

• • It writes testcase.<ext>. This is currently not specified, but the closest thing to the implementation that we already had.

Where does it write them. In the same directory?

• The output visualizer is executed in the same way as the output validator: in/ans/feedbackdir as args, output is passed to stdin.

👍

• • It writes image files to the feedbackdir, which are then left untouched (but from the discussion above, we may want to do something for the visualizations of the answer, a.k.a. the canonical jury output).

👍