Add execution level tagging

[nprizal]

Add execution level tagging using execution_tag custom marker. The test collector will send all tags marked on each test to Buildkite Test Engine. See pytest's custom markers documentation.

tagging individual tests using decorator:

import pytest

@pytest.mark.execution_tag("key", "value")
@pytest.mark.execution_tag("another.key", "value")
def test_add():
     assert 1 + 1 == 2

tagging all tests in a test class using decorator:

import pytest

@pytest.mark.execution_tag("key", "value")
@pytest.mark.execution_tag("another.key", "value")
class TestSubstract:
    def test_positive(self):
        assert 5 - 4 == 1

    def test_negative(self):
        assert 2 - 3 = -1 

tagging all tests in a module using pytestmark helper

import pytest

pytestmark=[
    pytest.mark.execution_tag("key", "value"),
    pytest.mark.execution_tag("another.key", "value")
]

def test_add():
     assert 1 + 1 == 2

tagging tests from a hook:

def pytest_runtest_setup(item):
    item.add_marker(pytest.mark.execution_tag("key", "value"))
    item.add_marker(pytest.mark.execution_tag("another.key", "value"))

[nprizal]

Defer the execution colection after the test teardown to allow adding execution_tag during the call hook.

[zhming0]

I don't quite understand this bit. I thought logreport is always after the teardown hook? What does this if condition do?

[zhming0]

Cool, looks great 🙂

[zhming0]

not a blocker, I wonder what will happen if user put non-string key value here.

[nprizal]

Ah! I wasn't aware that tag only support string. I'll handle non-string scenario.

[zhming0]

I don't quite understand this bit. I thought logreport is always after the teardown hook? What does this if condition do?

[nprizal]

I don't quite understand this bit. I thought logreport is always after the teardown hook? What does this if condition do?

The logreport hook is called after setup, call, and teardown. Previously, the test is reported (i.e. appended to the payload) on the logreport hook following a call. Because we are allowing user to add tag during a hook (most likely setup/call, we need to collect that tag at the end, which is when the test is teardown.

[zhming0]

@nprizal please give me a sec, I will have another read at this.

Taking another look, gaining more pytest understanding

[zhming0]

I understand it fully now. Let's 🚀 .

I don't think we need an abstraction on top of the marker just yet.

[nprizal]

I pushed changes to validate the type of key and value and raise error when it's not a string (to follow the behavior of ruby collector)

[zhming0]

🚀