i#7480: Move syscall injection after all pre-syscall event markers

[abhinav92003]

Moves injection point for syscall traces to the point after all syscall related markers added by the pre-syscall callback, but before the ones added by the post-syscall callback.

Specifically, moves injection to after these markers: TRACE_MARKER_TYPE_MAYBE_BLOCKING_SYSCALL, the syscall function tracing related ones including TRACE_MARKER_TYPE_FUNC_ID and TRACE_MARKER_TYPE_FUNC_ARG, and the scheduling related ones added by raw2trace sub-classes based on the syscall function tracing ones including TRACE_MARKER_TYPE_SYSCALL_SCHEDULE, TRACE_MARKER_TYPE_SYSCALL_UNSCHEDULE, TRACE_MARKER_TYPE_SYSCALL_ARG_TIMEOUT, and TRACE_MARKER_TYPE_DIRECT_THREAD_SWITCH. These markers are not present for all syscalls, but for only the ones that are considered non-blocking by raw2trace, and the ones that are specified in -record_syscall. This change is made for both static injection in raw2trace, and dynamic injection in the drmemtrace scheduler.

Adds a view test that shows how the syscall trace is dynamically injected on a deterministic pure assembly test app that includes maybe-blocking syscalls and syscalls traced using -record_syscall. Updates the mock syscall sequence file to include dummy sequences for the syscalls in the pure asm app.

Updates the burst_syscall_inject test which verifies static injection. Updates one of the syscalls in burst_syscall_inject to SYS_membarrier which is identified as a maybe blocking syscall by raw2trace, and add the other one to -record_syscall.

Updates the scheduler_unit_test and raw2trace_unit_test to show injection in various different cases.

Adds more trace invariant checks to the drmemtrace invariant checker to ensure the location of injected records is correct. Also adds tests in invariant_checker_unit_test and an invariant checker run on the pure-asm app dynamically injected trace.

Disallows empty templates for any kernel sequence in the drmemtrace scheduler.

Verifies some more view tool output for the existing allasm-record-syscall test, which lets a reader see the non-injected version.

Also manually tested on a large internal trace that the new static and dynamic injection logic generates traces without any invariant error.

Issue: #7480, #7157, #6495

[abhinav92003]

Moves injection point for syscall traces to the point after all syscall related markers added by the pre-syscall callback, but before the ones added by the post-syscall callback.

@derekbruening Would be useful to get some feedback on this design decision.

Initially I just wanted to keep the syscall and maybe_blocking_syscall markers together because there's an invariant check that fails unless that's true. But then I decided that the syscall trace ideally is placed between the pre- and post-syscall callback markers. But the scheduling related markers (syscall_schedule, syscall_unschedule, syscall_arg_timeout, direct_thread_switch) are also written at that point. Then I ran into issues injecting the syscall trace before the scheduling markers because of the way they are inserted today in raw2trace, so I ended up deciding to inject the trace after them.

Do you expect any hard-to-overcome issue in the drmemtrace scheduler with this injection point? Maybe this makes some scheduler stuff trickier for the four scheduling related markers if they set some state that we generally clear at instructions, which should be kept until the next scheduling decision (which is the next user-space instr). We'll need to make sure to preserve such state through the syscall trace. Maybe that's not too complicated?

[derekbruening]

Moves injection point for syscall traces to the point after all syscall related markers added by the pre-syscall callback, but before the ones added by the post-syscall callback.

@derekbruening Would be useful to get some feedback on this design decision.

Initially I just wanted to keep the syscall and maybe_blocking_syscall markers together because there's an invariant check that fails unless that's true. But then I decided that the syscall trace ideally is placed between the pre- and post-syscall callback markers. But the scheduling related markers (syscall_schedule, syscall_unschedule, syscall_arg_timeout, direct_thread_switch) are also written at that point. Then I ran into issues injecting the syscall trace before the scheduling markers because of the way they are inserted today in raw2trace, so I ended up deciding to inject the trace after them.

Do you expect any hard-to-overcome issue in the drmemtrace scheduler with this injection point? Maybe this makes some scheduler stuff trickier for the four scheduling related markers if they set some state that we generally clear at instructions, which should be kept until the next scheduling decision (which is the next user-space instr). We'll need to make sure to preserve such state through the syscall trace. Maybe that's not too complicated?

I believe the state is input->{unscheduled,blocked_time,skip_next_unscheduled} which should still be there while the syscall template delays check_for_input_switch(): so long as the syscall template never has these schedule markers.