Separate stdout and stderr in determine_result

[rkkautsar]

In #84 I see that it is intended to not hide any output from the user and separating these will remove the time relationship between them. But I argue that the separation can be done without removing the time relationship. For example, it can be done by attaching metadata to the output, maybe like a tuple (STDOUT, 'something') for the lines in output. Then, when needed it can be filtered to separate stdin and stderr, though this change will break backwards compatibility.

[PhilippWendler]

Yes, this would in theory be possible, but it would come with the disadvantage of increased overhead during benchmarking. For this to implement, runexec would need to spawn two threads that listen on a pipe for stdout/stderr and read each line, attach metadata, and write it to disk. Currently we just connect stdout/stderr to a file descriptor.

Because of the complexity of the implementation and the overhead it is not likely that we implement this, except if there is an important use case for this. What is your use case? Is there no alternative possibility for your use case?

[rkkautsar]

Thanks for the quick response.

My use case is to evaluate on multiline outputs, such as sudoku tables. For now I manage to just output the table to a file and later evaluate the file. Yup I agree this will increase complexity. When I think of it, I can also just write a wrapper anyway to output the stdout to a file or even remove the stderr altogether. Though I'll be glad if there's a way to separate them without adding complexity and overhead.

Will close this for now

[PhilippWendler]

Reopening because we might eventually support this once we implement for example #222.