Skip to content

libtbx testing broken #422

Open
Open
@Anthchirp

Description

@Anthchirp

Test in #421 is failing, but libtbx testing mistakenly treats it as success and only notes the standard error output. The test is not listed in the "This-is-broken-this-is-fine" category.

libtbx.python "/Users/dlshudson/jenkins_slave/workspace/cctbx_mac_python3/build_dials/modules/cctbx_project/scitbx/suffixtree/test/tst_single.py" [OK] 0.2s
  Standard error:
(...)
    FAILED (errors=1)
libtbx.python "/Users/dlshudson/jenkins_slave/workspace/cctbx_mac_python3/build_dials/modules/cctbx_project/scitbx/glmtbx/tests/tst.py" [OK] 0.7s
libtbx.python "/Users/dlshudson/jenkins_slave/workspace/cctbx_mac_python3/build_dials/modules/cctbx_project/scitbx/iso_surface/tst_iso_surface.py" [OK] 9.2s
================================================================================

Tests finished. Elapsed time: 351.59s

(..)

Summary:
  Tests run                    : 423
  Failures                     : 0
  Warnings (possible failures) : 0
  Known Failures (  4)         : 4
  Known Unstable (  1)         : 0
  Stderr output (discouraged)  : 5

That is probably why this test failure was not picked up earlier.

I guess the reason for this is that this test seems to run unittest under the hood but the failure behaviour was not tested. I suggest converting the tests to pytest. That works in a predictable manner and doesn't swallow errors.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions