There are a few situations where runexec can produce warnings during run execution, for example:
- If swapping occurs.
- If thermal throttling of the CPU occurs.
- If the tool output is too large and the log is truncated.
These trigger a warning message, but with executions of large benchmark sets, the messages are likely overlooked. In order to increase the visibility of these potential benchmark problems, we could make these warnings part of the actual run result.
RunExecutor (when used via its Python API) could return a list of warnings as part of its result dict (e.g., ["swapping", "throttling", "log truncation"] or so, maybe as an enum).
runexec could similarly either return a key with a list of comma-separated warnings, or individual keys like warning-swapping=True.
benchexec could either add a column with warnings, or one column for every warning.
We need to think about which of the variants is more convenient. Typically one should not have cells of type list in a table, but in practice this might be more convenient because warnings are expected to occur rarely.
There are a few situations where
runexeccan produce warnings during run execution, for example:These trigger a warning message, but with executions of large benchmark sets, the messages are likely overlooked. In order to increase the visibility of these potential benchmark problems, we could make these warnings part of the actual run result.
RunExecutor(when used via its Python API) could return a list of warnings as part of its result dict (e.g.,["swapping", "throttling", "log truncation"]or so, maybe as an enum).runexeccould similarly either return a key with a list of comma-separated warnings, or individual keys likewarning-swapping=True.benchexeccould either add a column with warnings, or one column for every warning.We need to think about which of the variants is more convenient. Typically one should not have cells of type list in a table, but in practice this might be more convenient because warnings are expected to occur rarely.