atf-c-api cannot expect both failure and timeout

[jmmv]

From asomers on January 16, 2014 18:34:38

What steps will reproduce the problem? 1. Create an ATF testcase in C that will both fail and timeout
2. Add atf_tc_expect_fail() and atf_tc_expect_timeout() statements to the test
3. Run it What is the expected output? What do you see instead? The expected output is something like "expected failure: blah blah blah". Instead, this is the output:
timeout_and_fail:expect_both -> failed: Test case was expecting a failure but none were raised [0.008s] What version of the product are you using? On what operating system? kyua 0.8

uname -a

FreeBSD alans-fbsd-head 11.0-CURRENT FreeBSD 11.0-CURRENT #23 r260786M: Thu Jan 16 13:06:43 MST 2014 alans@ns1.eng.sldomain.com:/vmpool/obj/usr/home/alans/freebsd/head/sys/GENERIC amd64 Please provide any additional information below. Attaching a test program that demonstrates the issues.

Attachment: timeout_and_fail.c Makefile Kyuafile

Original issue: http://code.google.com/p/kyua/issues/detail?id=76

[jmmv]

From julio@meroh.net on January 24, 2014 15:20:53

Is this the same as issue 27 ?

[jmmv]

From julio@meroh.net on January 24, 2014 15:38:11

Oh, I see. This is because you are using ATF_CHECK_* instead of ATF_REQUIRE_*; otherwise you'd never get to the timeout part because the test would abort right away.

Interesting, need to think about it some more.

Can you share some more details on the actual test you are writing, not just the simplified demonstration? Thanks.

[jmmv]

From asomers on January 24, 2014 15:29:04

Nope. This test is deterministic. It always fails and always times out. The bug is that there isn't any way to set an expectation for that behavior.

[jmmv]

From asomers on January 24, 2014 15:49:35

The actual test is in FreeBSD at sys/kern/unix_seqpacket_test:pipe_128k_8k. It times out because of kern/185812, which drops packets. And ATF_CHECK_EQ_MSG fails in the reader thread due to a miscompare, which is also due to the dropped packets.