e2etest: redesign framework

[ewienik]

This PR provides redesigning test framework by introducing macros for defining groups (group!) and tests (test!). It implements new e2etest-macros crate for procedural macros and new integration tests to check framework itself.

The new framework can be used like that:

mod sample {

use std::net::Ipv4Addr;
use std::sync::Arc;
use std::time::Duration;

#[derive(clap::Args)]
pub struct Args {
    #[arg(short, long, default_value = "127.0.100.1")]
    dns_ip: Ipv4Addr,
}

pub async fn init(args: &Args, fixtures: &e2etest::Fixtures) {
    fixtures.add_permanent(FixtureCfg { dns_ip : args.dns_ip }).await;
}

#[derive(Clone, Copy)]
pub struct FixtureCfg {
    dns_ip: Ipv4Addr,
}

#[derive(Clone, Copy)]
pub struct FixtureOne {
    dns_ip: Ipv4Addr,
}

impl e2etest::Fixture for FixtureOne {
    async fn setup(setup: &mut impl e2etest::Setup) -> Self {
        let cfg = setup.get::<FixtureCfg>().await.unwrap();
        Self { dns_ip: cfg.dns_ip }
    }

    async fn teardown(self) { }
}

#[derive(Clone, Copy)]
pub struct FixtureTwo {
    dns_ip: Ipv4Addr,
}

impl e2etest::Fixture for FixtureTwo {
    async fn setup(setup: &mut impl e2etest::Setup) -> Self {
        setup.setup::<FixtureOne>().await;
        let one = setup.get::<FixtureOne>().await.unwrap();
        Self { dns_ip: one.dns_ip }
    }

    async fn teardown(self) { }
}

e2etest::group!(name = root, fixtures = (FixtureOne));

e2etest::group!(name = group, fixtures = (FixtureOne), parent = root);

#[e2etest::test(group = group, timeout = Duration::from_secs(5))]
async fn dns_ip_100(one: Arc<FixtureOne>, two: Arc<FixtureTwo>) {
    assert_eq!(one.dns_ip, Ipv4Addr::new(127, 0, 100, 1));
    assert_eq!(two.dns_ip, Ipv4Addr::new(127, 0, 100, 1));
}

#[e2etest::test(group = group, skip = true)]
async fn dns_ip_200(one: Arc<FixtureOne>) {
    assert_eq!(one.dns_ip, Ipv4Addr::new(127, 0, 200, 1));
}

}

tokio::runtime::Runtime::new().unwrap().block_on(async move {
    use std::time::Duration;
    e2etest::run(["validator", "run"], sample::init, sample::root(), Duration::from_secs(10)).await.unwrap();
});

Fixes: VECTOR-153

[QuerthDP]

I don't understand this example:

#[derive(Clone, Copy)]
pub struct FixtureOne {
    dns_ip: Ipv4Addr,
}

impl e2etest::Fixture for FixtureOne {
    async fn setup(setup: &mut impl e2etest::Setup) -> Self {
        let cfg = setup.get::<FixtureCfg>().await.unwrap();
        Self { dns_ip: cfg.dns_ip }
    }

    async fn teardown(self) { }
}

#[derive(Clone, Copy)]
pub struct FixtureTwo {
    dns_ip: Ipv4Addr,
}

impl e2etest::Fixture for FixtureTwo {
    async fn setup(setup: &mut impl e2etest::Setup) -> Self {
        setup.setup::<FixtureOne>().await;
        let one = setup.get::<FixtureOne>().await.unwrap();
        Self { dns_ip: one.dns_ip }
    }

    async fn teardown(self) { }
}

Why would we need a FixtureTwo which all it does is setup FixtureOne and expose it's dns_ip?
That should be some explanatory example I suppose.

e2etest::group!(name = root, fixtures = (FixtureOne));

e2etest::group!(name = group, fixtures = (FixtureOne), parent = root);

What's the diference between those groups if the fixture set is the same? I would expect that the group fixtures would differ to showcase the possibilities.

Additionally, if we set the root as parent, why do we need to add the containing features in the group declaration as well?

#[e2etest::test(group = group, timeout = Duration::from_secs(5))]
async fn dns_ip_100(one: Arc<FixtureOne>, two: Arc<FixtureTwo>) {
    assert_eq!(one.dns_ip, Ipv4Addr::new(127, 0, 100, 1));
    assert_eq!(two.dns_ip, Ipv4Addr::new(127, 0, 100, 1));
}

#[e2etest::test(group = group, skip = true)]
async fn dns_ip_200(one: Arc<FixtureOne>) {
    assert_eq!(one.dns_ip, Ipv4Addr::new(127, 0, 200, 1));
}

Since those are defined to be in a group, why do we need to specify the fixtures in declaration?
I would get it if those were additional fixtures like FixtureTwo, but why do we need the FixtureOne as well?

[ewienik]

#[derive(Clone, Copy)]
pub struct FixtureOne {
    dns_ip: Ipv4Addr,
}

impl e2etest::Fixture for FixtureOne {
    async fn setup(setup: &mut impl e2etest::Setup) -> Self {
        let cfg = setup.get::<FixtureCfg>().await.unwrap();
        Self { dns_ip: cfg.dns_ip }
    }

    async fn teardown(self) { }
}

#[derive(Clone, Copy)]
pub struct FixtureTwo {
    dns_ip: Ipv4Addr,
}

impl e2etest::Fixture for FixtureTwo {
    async fn setup(setup: &mut impl e2etest::Setup) -> Self {
        setup.setup::<FixtureOne>().await;
        let one = setup.get::<FixtureOne>().await.unwrap();
        Self { dns_ip: one.dns_ip }
    }

    async fn teardown(self) { }
}

Why would we need a FixtureTwo which all it does is setup FixtureOne and expose it's dns_ip? That should be some explanatory example I suppose.

This is an example what e2etest framework can do, not a specific test case. You can base one fixture on the other one - FixtureTwo uses FixtureOne during setup.

e2etest::group!(name = root, fixtures = (FixtureOne));

e2etest::group!(name = group, fixtures = (FixtureOne), parent = root);

What's the diference between those groups if the fixture set is the same? I would expect that the group fixtures would differ to showcase the possibilities.

It shows that you can use different mix of fixtures - it could be the same set, it could be different set. It is hard to show all possibilities and keep it short.

Additionally, if we set the root as parent, why do we need to add the containing features in the group declaration as well?

You mean fixtures not features? Each group can have different set of fixtures and each groups are independent - each group must define fixtures by itself.

#[e2etest::test(group = group, timeout = Duration::from_secs(5))]
async fn dns_ip_100(one: Arc<FixtureOne>, two: Arc<FixtureTwo>) {
    assert_eq!(one.dns_ip, Ipv4Addr::new(127, 0, 100, 1));
    assert_eq!(two.dns_ip, Ipv4Addr::new(127, 0, 100, 1));
}

#[e2etest::test(group = group, skip = true)]
async fn dns_ip_200(one: Arc<FixtureOne>) {
    assert_eq!(one.dns_ip, Ipv4Addr::new(127, 0, 200, 1));
}

Since those are defined to be in a group, why do we need to specify the fixtures in declaration? I would get it if those were additional fixtures like FixtureTwo, but why do we need the FixtureOne as well?

First, tests are independent from groups - they don't know group fixtures (there is no introspection in Rust); you define fixture for tests independently from groups. Second, test! is a procedural macro which needs to parse an async method with its parameters; fixtures are parameters so must be named in its argument list - you need to know the arguments to correctly write function body.

[QuerthDP]

Each group can have different set of fixtures and each groups are independent - each group must define fixtures by itself.

Then what does it imply that the root is a parent of group?

#[e2etest::test(group = group, timeout = Duration::from_secs(5))]
async fn dns_ip_100(one: Arc<FixtureOne>, two: Arc<FixtureTwo>) {
    assert_eq!(one.dns_ip, Ipv4Addr::new(127, 0, 100, 1));
    assert_eq!(two.dns_ip, Ipv4Addr::new(127, 0, 100, 1));
}

#[e2etest::test(group = group, skip = true)]
async fn dns_ip_200(one: Arc<FixtureOne>) {
    assert_eq!(one.dns_ip, Ipv4Addr::new(127, 0, 200, 1));
}

Since those are defined to be in a group, why do we need to specify the fixtures in declaration? I would get it if those were additional fixtures like FixtureTwo, but why do we need the FixtureOne as well?

First, tests are independent from groups - they don't know group fixtures (there is no introspection in Rust); you define fixture for tests independently from groups. Second, test! is a procedural macro which needs to parse an async method with its parameters; fixtures are parameters so must be named in its argument list - you need to know the arguments to correctly write function body.

Do I understand correctly that the main difference is the "group" fixtures are shared and the additional "test" fixtures are independently added to each test which defines them?

[ewienik]

Each group can have different set of fixtures and each groups are independent - each group must define fixtures by itself.

Then what does it imply that the root is a parent of group?

Each group have a set of subgroups and tests. You can make a hierarchy of groups. Running group means that:

• setup a set of fixtures assigned to this group
• get fixtures assigned to this group (increase strong counter in Arc)
• run subgroup in a loop
• run tests in a loop
• drop stored fixtures (decrease strong counter in Arc)
• run teardown on all fixtures which are not used anywhere

Running test means:

• setup a set of fixtures assigned to this test
• get fixtures assigned to this test (increase strong counter in Arc)
• run defined test function with taken fixtures (consume fixtures, so drop strong counter in Arc)
• run teardown on all fixtures which are not used anywhere

#[e2etest::test(group = group, timeout = Duration::from_secs(5))]
async fn dns_ip_100(one: Arc<FixtureOne>, two: Arc<FixtureTwo>) {
    assert_eq!(one.dns_ip, Ipv4Addr::new(127, 0, 100, 1));
    assert_eq!(two.dns_ip, Ipv4Addr::new(127, 0, 100, 1));
}

#[e2etest::test(group = group, skip = true)]
async fn dns_ip_200(one: Arc<FixtureOne>) {
    assert_eq!(one.dns_ip, Ipv4Addr::new(127, 0, 200, 1));
}

Since those are defined to be in a group, why do we need to specify the fixtures in declaration? I would get it if those were additional fixtures like FixtureTwo, but why do we need the FixtureOne as well?

First, tests are independent from groups - they don't know group fixtures (there is no introspection in Rust); you define fixture for tests independently from groups. Second, test! is a procedural macro which needs to parse an async method with its parameters; fixtures are parameters so must be named in its argument list - you need to know the arguments to correctly write function body.

Do I understand correctly that the main difference is the "group" fixtures are shared and the additional "test" fixtures are independently added to each test which defines them?

Fixtures in groups or tests are independent, they belongs to the specific group or to the specific test. Each fixture could depend on other fixtures - during setup they have access to the cache (Fixtures type) of fixtures.

I understand that we need a mdbook for this crate :-)

[ewienik]

Changelog for bad6a06

• add async_runtime::framed as a macro for test function
• use std::future::Future in macros

range diff

1:  a02f4ee ! 1:  4493a6b e2etest-macros: implement group! and test! macros
    @@ crates/e2etest-macros/src/lib.rs (new)
     +/// The test function must be async, return `()`, and take as arguments a list of `Arc<Fixture>`
     +/// as a list of fixtures used inside the test.
     +///
    -+/// If you use this macro you should add `linkme` as a dependency in your crate.
    ++/// If you use this macro you should add `linkme` and `async_runtime` as a dependency in your crate.
     +#[proc_macro_attribute]
     +pub fn test(attr: TokenStream, item: TokenStream) -> TokenStream {
     +    let params = parse_macro_input!(attr as TestParams);
    @@ crates/e2etest-macros/src/lib.rs (new)
     +
     +            #skip
     +
    -+            fn run(&self, fixture: std::sync::Arc<#test_fixture>) -> impl Future<Output = ()> + Send + 'static {
    ++            fn run(&self, fixture: std::sync::Arc<#test_fixture>) -> impl std::future::Future<Output = ()> + Send + 'static {
     +                async move {
     +                    #name(#(std::sync::Arc::clone(&fixture.#fixtures_range)),*).await;
     +                }
    @@ crates/e2etest-macros/src/lib.rs (new)
     +            Box::new(#test_type)
     +        }
     +
    ++        #[async_backtrace::framed]
     +        #run
     +    };
     +
2:  12724c3 = 2:  1cdb633 e2etest: integrate redisigned framework
3:  90157f3 = 3:  bad6a06 e2etest: implement integration tests

[ewienik]

Changelog for 9bc3698

• fix doc comment

range diff

1:  4493a6b ! 1:  f5bba78 e2etest-macros: implement group! and test! macros
    @@ crates/e2etest-macros/src/lib.rs (new)
     +/// The test function must be async, return `()`, and take as arguments a list of `Arc<Fixture>`
     +/// as a list of fixtures used inside the test.
     +///
    -+/// If you use this macro you should add `linkme` and `async_runtime` as a dependency in your crate.
    ++/// If you use this macro you should add `linkme` and `async-backtrace` as a dependency in your crate.
     +#[proc_macro_attribute]
     +pub fn test(attr: TokenStream, item: TokenStream) -> TokenStream {
     +    let params = parse_macro_input!(attr as TestParams);
2:  1cdb633 = 2:  2dcc73b e2etest: integrate redisigned framework
3:  bad6a06 = 3:  9bc3698 e2etest: implement integration tests

[ewienik]

Changelog for 5b06a1d

• rebase to master
• refactor Fixtures::setup to return Arc
• refactor e2etest-macros by switching from panics to syn::Error

range diff

 1:  92dd7ac =  1:  cf78eaa e2etest: refactor Backtrace struct
 2:  85db4ad =  2:  7f63fb3 e2etest: refactor Statistics struct
 3:  29b1590 !  3:  174d770 e2etest: implement Fixture trait and Fixtures struct
    @@ crates/e2etest/src/fixture.rs (new)
     +        self.0.lock().await.add_permanent(a);
     +    }
     +
    -+    pub fn setup<F: Fixture>(&self) -> impl Future<Output = ()> + Send + use<F> {
    ++    pub fn setup<F: Fixture>(&self) -> impl Future<Output = Arc<F>> + Send + use<F> {
     +        let inner = Arc::clone(&self.0);
    -+        async move {
    -+            inner.lock().await.setup::<F>().await;
    -+        }
    ++        async move { inner.lock().await.setup::<F>().await }
     +    }
     +
     +    pub async fn get<A: Any + Send + Sync + 'static>(&self) -> Option<Arc<A>> {
    @@ crates/e2etest/src/fixture.rs (new)
     +/// is passed to the `setup` method of a fixture.
     +pub trait Setup: Send {
     +    /// Set up a new fixture. If the fixture is already set up, this will do nothing.
    -+    fn setup<F: Fixture>(&mut self) -> impl Future<Output = ()> + Send;
    ++    fn setup<F: Fixture>(&mut self) -> impl Future<Output = Arc<F>> + Send;
     +
     +    /// Get a fixture. If the fixture is not set up, this will return `None`.
     +    fn get<F: Send + Sync + 'static>(&self) -> impl Future<Output = Option<Arc<F>>> + Send;
    @@ crates/e2etest/src/fixture.rs (new)
     +}
     +
     +impl Setup for Inner {
    -+    async fn setup<F: Fixture>(&mut self) {
    ++    async fn setup<F: Fixture>(&mut self) -> Arc<F> {
     +        if self.permanent.contains_key(&TypeId::of::<F>())
     +            || self.cache.contains_key(&TypeId::of::<F>())
     +        {
    -+            return;
    ++            return self.get::<F>().await.unwrap();
     +        }
    -+        let fixture = F::setup(self).await;
    ++        let fixture = Arc::new(F::setup(self).await);
     +        self.cache
    -+            .insert(TypeId::of::<F>(), Arc::new(fixture) as Arc<dyn Teardown>);
    ++            .insert(TypeId::of::<F>(), Arc::clone(&fixture) as Arc<dyn Teardown>);
    ++        fixture
     +    }
     +
     +    async fn get<F: Send + Sync + 'static>(&self) -> Option<Arc<F>> {
 4:  1c2ef43 !  4:  ca55eaa e2etest: refactor tasks for fixtures and tests
    @@ crates/e2etest/src/lib.rs
      mod testcase;
      
      use async_backtrace::frame;
    -@@ crates/e2etest/src/lib.rs: pub use testcase::TestCase;
    - use tokio::fs;
    +@@ crates/e2etest/src/lib.rs: use std::time::Duration;
    + pub use testcase::TestCase;
      use tokio::runtime::Builder;
      use tokio::runtime::Handle;
     -use tokio::task;
    @@ crates/e2etest/src/task.rs (new)
     +use tracing::info;
     +
     +#[framed]
    -+pub(crate) async fn setup(
    ++pub(crate) async fn setup<T: Send + Sync + 'static>(
     +    name: &str,
    -+    setup: impl Future<Output = ()> + Send + 'static,
    ++    setup: impl Future<Output = T> + Send + 'static,
     +    timeout: Duration,
     +    backtrace: Backtrace,
    -+) -> Result<Statistics, Statistics> {
    ++) -> Result<(T, Statistics), Statistics> {
     +    single(
     +        error_span!("setup"),
     +        "setup",
    @@ crates/e2etest/src/task.rs (new)
     +    timeout: Duration,
     +    backtrace: Backtrace,
     +) -> Statistics {
    -+    match single(
    ++    match single::<()>(
     +        error_span!("teardown"),
     +        "teardown",
     +        name,
    @@ crates/e2etest/src/task.rs (new)
     +    )
     +    .await
     +    {
    -+        Ok(stats) => stats,
    ++        Ok((_, stats)) => stats,
     +        Err(stats) => stats,
     +    }
     +}
    @@ crates/e2etest/src/task.rs (new)
     +    timeout: Duration,
     +    backtrace: Backtrace,
     +) -> Statistics {
    -+    match single(error_span!("run"), "run", name, run, timeout, backtrace).await {
    -+        Ok(stats) => stats,
    ++    match single::<()>(error_span!("run"), "run", name, run, timeout, backtrace).await {
    ++        Ok((_, stats)) => stats,
     +        Err(stats) => stats,
     +    }
     +}
     +
     +#[framed]
    -+pub(crate) async fn single(
    ++pub(crate) async fn single<T: Send + Sync + 'static>(
     +    span: Span,
     +    operation: &str,
     +    name: &str,
    -+    fut: impl Future<Output = ()> + Send + 'static,
    ++    fut: impl Future<Output = T> + Send + 'static,
     +    timeout: Duration,
     +    backtrace: Backtrace,
    -+) -> Result<Statistics, Statistics> {
    ++) -> Result<(T, Statistics), Statistics> {
     +    let mut stats = Statistics::new();
     +
     +    stats.increment_launched();
     +
     +    let task_result = tokio::spawn(frame!(
    -+        async move {
    -+            time::timeout(timeout, fut).await.expect("test timed out");
    -+        }
    -+        .instrument(span.clone())
    ++        async move { time::timeout(timeout, fut).await.expect("test timed out") }
    ++            .instrument(span.clone())
     +    ))
     +    .await;
     +
    -+    if let Err(err) = task_result {
    -+        stats.record_failure(format!("{name}::{operation}"));
    -+        let backtrace = backtrace.get();
    -+        error!(parent: &span, "test failed: {err}\n{backtrace}");
    -+        Err(stats)
    -+    } else {
    -+        stats.increment_ok();
    -+        info!(parent: &span, "test ok");
    -+        Ok(stats)
    ++    match task_result {
    ++        Err(err) => {
    ++            stats.record_failure(format!("{name}::{operation}"));
    ++            let backtrace = backtrace.get();
    ++            error!(parent: &span, "test failed: {err}\n{backtrace}");
    ++            Err(stats)
    ++        }
    ++        Ok(t) => {
    ++            stats.increment_ok();
    ++            info!(parent: &span, "test ok");
    ++            Ok((t, stats))
    ++        }
     +    }
     +}
 5:  41bb53f !  5:  3eee3e3 e2etest: introduce Test and RunTest traits
    @@ crates/e2etest/src/test.rs (new)
     +            stats.increment_total(3);
     +
     +            // Setup the fixture. If it fails, we skip the test and teardown.
    -+            match task::setup(
    ++            let fixture = match task::setup(
     +                &name,
     +                fixtures.setup::<F>(),
     +                F::timeout_setup().unwrap_or(default_timeout),
    @@ crates/e2etest/src/test.rs (new)
     +            )
     +            .await
     +            {
    -+                Ok(fixture_stats) => stats.append(&fixture_stats),
    ++                Ok((fixture, fixture_stats)) => {
    ++                    stats.append(&fixture_stats);
    ++                    fixture
    ++                }
     +                Err(fixture_stats) => {
     +                    stats.append(&fixture_stats);
     +                    stats.increment_error_skipped(2);
     +                    return stats;
     +                }
    -+            }
    -+            let fixture = fixtures.get::<F>().await.unwrap();
    ++            };
     +
     +            // Run the test if not skipped. If it fails, we still run teardown.
     +            if self.skip() {
 6:  2f0ee84 !  6:  87bdecb e2etest: introduce Group and RunGroup traits and refactor filtering
    @@ Cargo.lock: dependencies = [
      ]
     
      ## Cargo.toml ##
    -@@ Cargo.toml: clap = { version = "4.5.40", features = ["derive"] }
    - e2etest = { path = "crates/e2etest", version = "0.1.0" }
    +@@ Cargo.toml: async-backtrace = "0.2.7"
    + clap = { version = "4.5.40", features = ["derive"] }
      futures = "0.3.31"
      hickory-server = "0.26.1"
     +itertools = "0.14.0"
    @@ crates/e2etest/src/group.rs (new)
     +            stats.increment_total(2);
     +
     +            // Setup the fixture. If it fails, we skip the tests
    -+            match task::setup(
    ++            let fixture = match task::setup(
     +                self.name(),
     +                fixtures.setup::<F>(),
     +                F::timeout_setup().unwrap_or(default_timeout),
    @@ crates/e2etest/src/group.rs (new)
     +            )
     +            .await
     +            {
    -+                Ok(fixture_stats) => stats.append(&fixture_stats),
    ++                Ok((fixture, fixture_stats)) => {
    ++                    stats.append(&fixture_stats);
    ++                    fixture
    ++                }
     +                Err(fixture_stats) => {
     +                    stats.append(&fixture_stats);
     +                    stats.increment_error_skipped(1);
     +                    return stats;
     +                }
    -+            }
    -+            let fixture = fixtures.get::<F>().await.unwrap();
    ++            };
     +
     +            // Run groups
     +            for group in self.groups().iter().filter(|group| {
 7:  962af05 =  7:  2eb91a7 e2etest: refactor running tests
 8:  f5bba78 !  8:  b60396a e2etest-macros: implement group! and test! macros
    @@ Cargo.lock: dependencies = [
     + "convert_case",
     + "itertools",
     + "linkme",
    ++ "proc-macro2",
     + "quote",
     + "syn",
     +]
    @@ Cargo.toml: license = "MIT OR Apache-2.0"
      async-backtrace = "0.2.7"
      clap = { version = "4.5.40", features = ["derive"] }
     +convert_case = "0.11.0"
    - e2etest = { path = "crates/e2etest", version = "0.1.0" }
     +e2etest-macros = { path = "crates/e2etest-macros", version = "0.1.0" }
      futures = "0.3.31"
      hickory-server = "0.26.1"
    @@ Cargo.toml: license = "MIT OR Apache-2.0"
     +linkme = "0.3.36"
      neli = { version = "0.7.4", default-features = false, features = ["async"] }
      num-bigint = "0.4"
    ++proc-macro2 = "1.0.106"
     +quote = "1.0.0"
      rustls = "0.23"
      rustls-pki-types = "1.13.1"
    @@ crates/e2etest-macros/Cargo.toml (new)
     +convert_case.workspace = true
     +itertools.workspace = true
     +linkme.workspace = true
    ++proc-macro2.workspace = true
     +quote.workspace = true
     +syn.workspace = true
     
    @@ crates/e2etest-macros/src/lib.rs (new)
     +use syn::parse::Parse;
     +use syn::parse::ParseStream;
     +use syn::parse_macro_input;
    ++use syn::spanned::Spanned;
     +
     +fn group_groups_name(name: &str) -> String {
     +    format!("_E2ETEST_{name}_GROUPS", name = ccase!(constant, name))
    @@ crates/e2etest-macros/src/lib.rs (new)
     +                    .into_iter()
     +                    .map(|elem| {
     +                        let Type::Path(type_path) = elem.ty else {
    -+                            panic!("Expected a tuple of Fixture");
    ++                            return Err(syn::Error::new(
    ++                                elem.span(),
    ++                                "Expected a type implementing Fixture",
    ++                            ));
     +                        };
    -+                        type_path.path.segments
    ++                        Ok(type_path.path.segments)
     +                    })
     +                    .map(|segments| {
    -+                        segments
    -+                            .first()
    -+                            .expect("Expected a tuple of Fixture")
    -+                            .clone()
    ++                        segments.and_then(|segments| {
    ++                            let span = segments.span();
    ++                            (segments.len() == 1)
    ++                            .then_some(segments)
    ++                            .and_then(|segments| segments.first().cloned())
    ++                            .ok_or(syn::Error::new(
    ++                                span,
    ++                                "Expected a single pathtuple of Fixture with at least one element",
    ++                            ))
    ++                        })
     +                    })
    -+                    .map(|path_segment| path_segment.ident)
    -+                    .collect();
    ++                    .map_ok(|path_segment| path_segment.ident)
    ++                    .collect::<syn::Result<_>>()?;
     +            } else if name == "parent" {
     +                let _: Token![=] = input.parse()?;
     +                parent = Some(input.parse()?);
    @@ crates/e2etest-macros/src/lib.rs (new)
     +#[proc_macro]
     +pub fn group(item: TokenStream) -> TokenStream {
     +    let params = parse_macro_input!(item as GroupParams);
    ++    generate_group(params)
    ++        .unwrap_or_else(syn::Error::into_compile_error)
    ++        .into()
    ++}
     +
    ++fn generate_group(params: GroupParams) -> syn::Result<proc_macro2::TokenStream> {
     +    let name = params.name;
     +    let name_string = name.to_string();
     +    let fixtures = params.fixtures;
    @@ crates/e2etest-macros/src/lib.rs (new)
     +    let group_fixture = Ident::new(&group_fixture_name(&name_string), name.span());
     +    let group_type = Ident::new(&group_type_name(&name_string), name.span());
     +    let register_group = if let Some(parent) = &params.parent {
    -+        let parent_name = parent.segments.last().unwrap().ident.to_string();
    ++        let Some(last) = parent.segments.last() else {
    ++            return Err(syn::Error::new(
    ++                parent.segments.span(),
    ++                "Expected parent path to have at least one segment",
    ++            ));
    ++        };
    ++        let parent_name = last.ident.to_string();
     +        let mut parent_groups = parent.clone();
    -+        parent_groups
    -+            .segments
    -+            .last_mut()
    -+            .map(|last| {
    -+                last.ident = Ident::new(&group_groups_name(&parent_name), name.span());
    -+            })
    -+            .expect("Parent path must have at least one segment");
    ++        let Some(last) = parent_groups.segments.last_mut() else {
    ++            return Err(syn::Error::new(
    ++                parent_groups.segments.span(),
    ++                "Expected parent path to have at least one segment",
    ++            ));
    ++        };
    ++        last.ident = Ident::new(&group_groups_name(&parent_name), name.span());
     +        quote! {
     +            #[linkme::distributed_slice(#parent_groups)]
     +        }
    @@ crates/e2etest-macros/src/lib.rs (new)
     +        struct #group_fixture(#(std::sync::Arc<#fixtures>),*);
     +        impl e2etest::Fixture for #group_fixture {
     +            async fn setup(setup: &mut impl e2etest::Setup) -> Self {
    -+                #(setup.setup::<#fixtures>().await;)*
    -+                Self(#(setup.get::<#fixtures>().await.unwrap()),*)
    ++                Self(#(setup.setup::<#fixtures>().await),*)
     +            }
     +            async fn teardown(self) { }
     +        }
    @@ crates/e2etest-macros/src/lib.rs (new)
     +            Box::new(#group_type)
     +        }
     +    };
    -+    TokenStream::from(expanded)
    ++
    ++    Ok(expanded)
     +}
     +
     +struct TestParams {
    @@ crates/e2etest-macros/src/lib.rs (new)
     +    }
     +}
     +
    -+fn take_fixtures(run: &ItemFn) -> Vec<TypePath> {
    -+    let fixtures = run
    ++fn take_fixtures(run: &ItemFn) -> syn::Result<Vec<TypePath>> {
    ++    let fixtures: Vec<_> = run
     +        .sig
     +        .inputs
     +        .iter()
     +        .map(|arg| {
    -+            let FnArg::Typed(pat_type) = arg else {
    -+                panic!("Expected arguments of type Arc<Fixture>");
    -+            };
    -+            pat_type
    -+        })
    -+        .map(|pat_type| {
    -+            let Type::Path(type_path) = &*pat_type.ty else {
    -+                panic!("Expected arguments of type Arc<Fixture>");
    -+            };
    -+            type_path
    -+        })
    -+        .map(|type_path| {
    -+            let Some(path_segment) = type_path.path.segments.first() else {
    -+                panic!("Expected arguments of type Arc<Fixture>");
    -+            };
    -+            assert_eq!(
    -+                path_segment.ident, "Arc",
    -+                "Expected arguments of type Arc<Fixture>"
    -+            );
    -+            path_segment
    -+        })
    -+        .map(|path_segment| {
    -+            let PathArguments::AngleBracketed(angle_bracketed) = &path_segment.arguments else {
    -+                panic!("Expected arguments of type Arc<Fixture>");
    -+            };
    -+            angle_bracketed
    -+        })
    -+        .map(|angle_bracketed| {
    -+            let Some(generic_arg) = angle_bracketed.args.first() else {
    -+                panic!("Expected arguments of type Arc<Fixture>");
    -+            };
    -+            generic_arg
    -+        })
    -+        .map(|generic_arg| {
    -+            let GenericArgument::Type(Type::Path(fixture_type)) = generic_arg else {
    -+                panic!("Expected arguments of type Arc<Fixture>");
    -+            };
    -+            fixture_type.clone()
    ++            if let FnArg::Typed(pat_type) = arg
    ++                && let Type::Path(type_path) = &*pat_type.ty
    ++                && let Some(path_segment) = type_path.path.segments.first()
    ++                && let PathArguments::AngleBracketed(angle_bracketed) = &path_segment.arguments
    ++                && let Some(generic_arg) = angle_bracketed.args.first()
    ++                && let GenericArgument::Type(Type::Path(fixture_type)) = generic_arg
    ++            {
    ++                Ok(fixture_type.clone())
    ++            } else {
    ++                Err(syn::Error::new(
    ++                    arg.span(),
    ++                    "Expected arguments of type Arc<Fixture>",
    ++                ))
    ++            }
     +        })
    -+        .collect_vec();
    -+    assert!(
    -+        !fixtures.is_empty(),
    -+        "Expected the test function to have at least one argument of type Arc<Fixture>"
    -+    );
    -+    fixtures
    ++        .collect::<syn::Result<_>>()?;
    ++    if fixtures.is_empty() {
    ++        Err(syn::Error::new(
    ++            run.sig.inputs.span(),
    ++            "Expected the test function to have at least one argument of type Arc<Fixture>",
    ++        ))
    ++    } else {
    ++        Ok(fixtures)
    ++    }
     +}
     +
     +/// Macro for defining a test.
    @@ crates/e2etest-macros/src/lib.rs (new)
     +#[proc_macro_attribute]
     +pub fn test(attr: TokenStream, item: TokenStream) -> TokenStream {
     +    let params = parse_macro_input!(attr as TestParams);
    -+    let group_name = params.group.segments.last().unwrap().ident.to_string();
    ++    let run = parse_macro_input!(item as ItemFn);
    ++    generate_test(params, run)
    ++        .unwrap_or_else(syn::Error::into_compile_error)
    ++        .into()
    ++}
    ++
    ++fn generate_test(params: TestParams, run: ItemFn) -> syn::Result<proc_macro2::TokenStream> {
    ++    let Some(last) = params.group.segments.last() else {
    ++        return Err(syn::Error::new(
    ++            params.group.segments.span(),
    ++            "Expected group path to have at least one segment",
    ++        ));
    ++    };
    ++    let group_name = last.ident.to_string();
     +    let mut group_tests = params.group.clone();
    -+    group_tests
    -+        .segments
    -+        .last_mut()
    -+        .map(|last| {
    -+            last.ident = Ident::new(&group_tests_name(&group_name), last.ident.span());
    -+        })
    -+        .expect("Group path must have at least one segment");
    ++    let Some(last) = group_tests.segments.last_mut() else {
    ++        return Err(syn::Error::new(
    ++            group_tests.segments.span(),
    ++            "Expected group path to have at least one segment",
    ++        ));
    ++    };
    ++    last.ident = Ident::new(&group_tests_name(&group_name), last.ident.span());
     +
    -+    let run = parse_macro_input!(item as ItemFn);
     +    let name = run.sig.ident.clone();
     +    let name_string = name.to_string();
     +    let test_fixture = Ident::new(&test_fixture_name(&name_string), name.span());
    @@ crates/e2etest-macros/src/lib.rs (new)
     +        quote! {}
     +    };
     +
    -+    let fixtures = take_fixtures(&run);
    ++    let fixtures = take_fixtures(&run)?;
     +    let fixtures_range = (0..fixtures.len()).map(Index::from);
     +
     +    let expanded = quote! {
     +        struct #test_fixture(#(std::sync::Arc<#fixtures>),*);
     +        impl e2etest::Fixture for #test_fixture {
     +            async fn setup(setup: &mut impl e2etest::Setup) -> Self {
    -+                #(setup.setup::<#fixtures>().await;)*
    -+                Self(#(setup.get::<#fixtures>().await.unwrap()),*)
    ++                Self(#(setup.setup::<#fixtures>().await),*)
     +            }
     +            async fn teardown(self) { }
     +        }
    @@ crates/e2etest-macros/src/lib.rs (new)
     +        #run
     +    };
     +
    -+    TokenStream::from(expanded)
    ++    Ok(expanded)
     +}
 9:  2dcc73b !  9:  4b253e4 e2etest: integrate redisigned framework
    @@ crates/e2etest/src/lib.rs
     -//! async fn init_testcase(fixture: Fixture) {
     +//! #[derive(Clone, Copy)]
     +//! pub struct FixtureTwo {
    -+//!     dns_ip: Ipv4Addr,
    ++//!     octet: u8,
     +//! }
     +//!
     +//! impl e2etest::Fixture for FixtureTwo {
     +//!     async fn setup(setup: &mut impl e2etest::Setup) -> Self {
    -+//!         setup.setup::<FixtureOne>().await;
    -+//!         let one = setup.get::<FixtureOne>().await.unwrap();
    -+//!         Self { dns_ip: one.dns_ip }
    ++//!         let one = setup.setup::<FixtureOne>().await;
    ++//!         Self { octet: one.dns_ip.octets()[2] }
     +//!     }
     +//!
     +//!     async fn teardown(self) { }
    - //! }
    - //!
    --//! async fn cleanup_testcase(fixture: Fixture) {
    ++//! }
    ++//!
    ++//! #[derive(Clone, Copy)]
    ++//! pub struct FixtureThree {
    ++//!     number: usize,
    ++//! }
    ++//!
    ++//! impl e2etest::Fixture for FixtureThree {
    ++//!     async fn setup(setup: &mut impl e2etest::Setup) -> Self {
    ++//!         let two = setup.setup::<FixtureTwo>().await;
    ++//!         Self { number: two.octet as usize * 1024 }
    ++//!     }
    ++//!
    ++//!     async fn teardown(self) { }
    ++//! }
    ++//!
     +//! e2etest::group!(name = root, fixtures = (FixtureOne));
     +//!
    -+//! e2etest::group!(name = group, fixtures = (FixtureOne), parent = root);
    ++//! e2etest::group!(name = group, fixtures = (FixtureTwo), parent = root);
     +//!
     +//! #[e2etest::test(group = group, timeout = Duration::from_secs(5))]
     +//! async fn dns_ip_100(one: Arc<FixtureOne>, two: Arc<FixtureTwo>) {
     +//!     assert_eq!(one.dns_ip, Ipv4Addr::new(127, 0, 100, 1));
    -+//!     assert_eq!(two.dns_ip, Ipv4Addr::new(127, 0, 100, 1));
    ++//!     assert_eq!(two.octet, 100);
      //! }
      //!
    --//! async fn test_dns_ip(fixture: Fixture) {
    --//!     assert_eq!(fixture.dns_ip, Ipv4Addr::new(127, 0, 100, 1));
    +-//! async fn cleanup_testcase(fixture: Fixture) {
     +//! #[e2etest::test(group = group, skip = true)]
     +//! async fn dns_ip_200(one: Arc<FixtureOne>) {
     +//!     assert_eq!(one.dns_ip, Ipv4Addr::new(127, 0, 200, 1));
      //! }
      //!
    +-//! async fn test_dns_ip(fixture: Fixture) {
    +-//!     assert_eq!(fixture.dns_ip, Ipv4Addr::new(127, 0, 100, 1));
    ++//! #[e2etest::test(group = group)]
    ++//! async fn number_and_octet(two: Arc<FixtureTwo>, three: Arc<FixtureThree>) {
    ++//!     assert_eq!(two.octet, 100);
    ++//!     assert_eq!(three.number, 100 * 1024);
    + //! }
    + //!
     -//! async fn register() -> Vec<(String, TestCase<Fixture>)> {
     -//!     let timeout = Duration::from_secs(10);
     -//!     let testcase = TestCase::empty()
    @@ crates/e2etest/src/lib.rs: mod run;
     +pub use e2etest_macros::group;
     +pub use e2etest_macros::test;
      use std::ffi::OsString;
    - use std::os::unix::fs::PermissionsExt;
      use std::panic;
    - use std::path::Path;
     -use std::sync::Arc;
      use std::time::Duration;
     -pub use testcase::TestCase;
    - use tokio::fs;
     -use tokio::runtime::Builder;
      use tokio::runtime::Handle;
      use tokio::time;
      use tracing::error;
    -@@ crates/e2etest/src/lib.rs: pub async fn executable_exists(path: &Path) -> bool {
    -     metadata.is_file() && (metadata.permissions().mode() & 0o111 != 0)
    +@@ crates/e2etest/src/lib.rs: enum Command<T: clap::Args> {
    +     },
      }
      
     -#[derive(Clone, Copy, Debug, Eq, PartialEq)]
    @@ crates/e2etest/src/lib.rs: pub async fn executable_exists(path: &Path) -> bool {
     -            ),
     -        ]
     -    }
    --
    ++    };
    + 
     -    #[test]
     -    fn test_no_filters_runs_all() {
     -        let test_cases = make_test_cases();
    @@ crates/e2etest/src/lib.rs: pub async fn executable_exists(path: &Path) -> bool {
     -        let result = parse_test_filters(&filters, &test_cases);
     -        assert!(result.is_empty());
     -    }
    --
    ++    let filter = Filter::new(&filters, group.as_ref());
    + 
     -    #[test]
     -    fn test_empty_filters_runs_all() {
     -        let test_cases = make_test_cases();
    @@ crates/e2etest/src/lib.rs: pub async fn executable_exists(path: &Path) -> bool {
     -        assert!(result["full_scan"].is_empty());
     -        assert!(result["other"].is_empty());
     -    }
    --
    ++    let report = run::run(fixtures, group, filter, default_timeout).await;
    + 
     -    #[test]
     -    fn test_file_partial_match() {
     -        let test_cases = make_test_cases();
    @@ crates/e2etest/src/lib.rs: pub async fn executable_exists(path: &Path) -> bool {
     -        assert!(result["crud"].contains("simple_create"));
     -        assert_eq!(result.len(), 1);
     -    }
    -+    };
    - 
    +-
     -    #[test]
     -    fn test_file_and_empty_test_case_syntax() {
     -        let test_cases = make_test_cases();
    @@ crates/e2etest/src/lib.rs: pub async fn executable_exists(path: &Path) -> bool {
     -        assert!(result["crud"].is_empty());
     -        assert_eq!(result.len(), 1);
     -    }
    -+    let filter = Filter::new(&filters, group.as_ref());
    - 
    +-
     -    #[test]
     -    fn test_empty_file_and_test_case_syntax() {
     -        let test_cases = make_test_cases();
    @@ crates/e2etest/src/lib.rs: pub async fn executable_exists(path: &Path) -> bool {
     -        assert!(result["other"].contains("simple_misc"));
     -        assert_eq!(result.len(), 2);
     -    }
    -+    let report = run::run(fixtures, group, filter, default_timeout).await;
    - 
    +-
     -    #[test]
     -    fn test_exact_file_match_syntax() {
     -        let test_cases = make_overlapping_test_cases();
10:  9bc3698 = 10:  5b06a1d e2etest: implement integration tests

[ewienik]

Changelog for e27c80e

• fix filtering for hierarchy of groups

range diff

1:  87bdecb ! 1:  f35902f e2etest: introduce Group and RunGroup traits and refactor filtering
    @@ crates/e2etest/src/filter.rs (new)
     +pub struct Filter {
     +    /// - Key: test file name (e.g., "crud", "full_scan")
     +    /// - Value: HashSet of specific test names within that file (empty means run all tests in file)
    -+    map: Arc<HashMap<String, HashSet<String>>>,
    ++    tests: Arc<HashMap<String, HashSet<String>>>,
    ++    groups: Arc<HashSet<String>>,
     +}
     +
     +impl Filter {
    @@ crates/e2etest/src/filter.rs (new)
     +    /// Returns a HashMap where:
     +    pub(crate) fn new(filters: &[String], group: &dyn RunGroup) -> Self {
     +        let mut filter_map = HashMap::new();
    ++        let mut group_set = HashSet::new();
     +
     +        if filters.is_empty() {
     +            // Run all tests
     +            return Self {
    -+                map: Arc::new(filter_map),
    ++                tests: Arc::new(filter_map),
    ++                groups: Arc::new(group_set),
     +            };
     +        }
     +
    ++        let parent_name = group.name().to_string();
    ++        let mut update_group_set = |group_name: &str| {
    ++            group_name
    ++                .split("::")
    ++                .fold(parent_name.clone(), |acc, name| {
    ++                    let acc = format!("{acc}::{name}");
    ++                    group_set.insert(acc.clone());
    ++                    acc
    ++                })
    ++        };
     +        for filter in filters {
    -+            // Check for <file>::<test> syntax
    -+            if let Some((file_part, test_part)) = filter.rsplit_once("::") {
    -+                let file_filter = FilterMatcher::new(file_part);
    ++            // Check for <group>::<test> syntax
    ++            if let Some((group_part, test_part)) = filter.rsplit_once("::") {
    ++                let group_filter = FilterMatcher::new(group_part);
     +                let test_filter = FilterMatcher::new(test_part);
     +
     +                for (group_name, test_name) in group
    @@ crates/e2etest/src/filter.rs (new)
     +                    .iter()
     +                    .filter_map(|name| name.rsplit_once("::"))
     +                {
    -+                    if !file_filter.matches(group_name) {
    ++                    if !group_filter.matches(group_name) {
     +                        continue;
     +                    }
     +
    ++                    let parent_group_name = format!("{parent_name}::{group_name}");
     +                    if matches!(test_filter, FilterMatcher::Any) {
    -+                        filter_map.entry(group_name.to_string()).or_default();
    ++                        filter_map.entry(parent_group_name).or_default();
    ++                        update_group_set(group_name);
     +                        continue;
     +                    }
     +                    if test_filter.matches(test_name) {
     +                        filter_map
    -+                            .entry(group_name.to_string())
    ++                            .entry(parent_group_name)
     +                            .or_default()
     +                            .insert(test_name.to_string());
    ++                        update_group_set(group_name);
     +                    }
     +                }
     +            } else {
    @@ crates/e2etest/src/filter.rs (new)
     +                    .iter()
     +                    .filter_map(|name| name.rsplit_once("::"))
     +                {
    ++                    let parent_group_name = format!("{parent_name}::{group_name}");
     +                    if filter.matches(group_name) {
    -+                        filter_map.entry(group_name.to_string()).or_default();
    ++                        filter_map.entry(parent_group_name).or_default();
    ++                        update_group_set(group_name);
     +                        continue;
     +                    }
     +                    if filter.matches(test_name) {
     +                        filter_map
    -+                            .entry(group_name.to_string())
    ++                            .entry(parent_group_name)
     +                            .or_default()
     +                            .insert(test_name.to_string());
    ++                        update_group_set(group_name);
     +                    }
     +                }
     +            }
     +        }
     +
     +        Self {
    -+            map: Arc::new(filter_map),
    ++            tests: Arc::new(filter_map),
    ++            groups: Arc::new(group_set),
     +        }
     +    }
     +
    @@ crates/e2etest/src/filter.rs (new)
     +            .map(|v| v.as_ref().to_string())
     +            .chain(iter::once(group_name.to_string()))
     +            .join("::");
    -+        self.map.is_empty() || self.map.contains_key(&group_name)
    ++        self.groups.is_empty() || self.groups.contains(&group_name)
     +    }
     +
     +    pub(crate) fn consider_test(
    @@ crates/e2etest/src/filter.rs (new)
     +        group_names: impl IntoIterator<Item = impl AsRef<str>>,
     +        test_name: &str,
     +    ) -> bool {
    -+        if self.map.is_empty() {
    ++        if self.tests.is_empty() {
     +            return true;
     +        }
     +        let group_name = group_names
     +            .into_iter()
     +            .map(|v| v.as_ref().to_string())
     +            .join("::");
    -+        if let Some(tests) = self.map.get(&group_name) {
    ++        if let Some(tests) = self.tests.get(&group_name) {
     +            tests.is_empty() || tests.contains(test_name)
     +        } else {
     +            false
    @@ crates/e2etest/src/filter.rs (new)
     +
     +    fn make_test_cases() -> Box<dyn RunGroup> {
     +        Box::new(GroupImpl {
    -+            name: String::new(),
    ++            name: "root".to_string(),
     +            tests: vec![],
     +            groups: vec![
     +                make_dummy_group("crud", &["simple_create", "drop_index"]),
    @@ crates/e2etest/src/filter.rs (new)
     +
     +    fn make_overlapping_test_cases() -> Box<dyn RunGroup> {
     +        Box::new(GroupImpl {
    -+            name: String::new(),
    ++            name: "root".to_string(),
     +            tests: vec![],
     +            groups: vec![
     +                make_dummy_group("crud", &["simple_create", "simple_create_extra"]),
    @@ crates/e2etest/src/filter.rs (new)
     +        let test_cases = make_test_cases();
     +        let filters: Vec<String> = vec![];
     +        let result = Filter::new(&filters, test_cases.as_ref());
    -+        assert!(result.map.is_empty());
    ++        assert!(result.groups.is_empty());
    ++        assert!(result.tests.is_empty());
     +    }
     +
     +    #[test]
    @@ crates/e2etest/src/filter.rs (new)
     +        let filters: Vec<String> = vec!["::".to_string()];
     +        let result = Filter::new(&filters, test_cases.as_ref());
     +        // It should contain all available test files with empty test cases (running all)
    -+        assert_eq!(result.map.len(), 3);
    -+        assert!(result.map["crud"].is_empty());
    -+        assert!(result.map["full_scan"].is_empty());
    -+        assert!(result.map["other"].is_empty());
    ++        assert_eq!(result.groups.len(), 3);
    ++        assert_eq!(result.tests.len(), 3);
    ++        assert!(result.groups.contains("root::crud"));
    ++        assert!(result.tests["root::crud"].is_empty());
    ++        assert!(result.groups.contains("root::full_scan"));
    ++        assert!(result.tests["root::full_scan"].is_empty());
    ++        assert!(result.groups.contains("root::other"));
    ++        assert!(result.tests["root::other"].is_empty());
     +    }
     +
     +    #[test]
    @@ crates/e2etest/src/filter.rs (new)
     +        let test_cases = make_test_cases();
     +        let filters = vec!["crud".to_string()];
     +        let result = Filter::new(&filters, test_cases.as_ref());
    -+        assert!(result.map.contains_key("crud"));
    -+        assert!(result.map["crud"].is_empty());
    -+        assert_eq!(result.map.len(), 1);
    ++        assert!(result.groups.contains("root::crud"));
    ++        assert!(result.tests.contains_key("root::crud"));
    ++        assert!(result.tests["root::crud"].is_empty());
    ++        assert_eq!(result.groups.len(), 1);
    ++        assert_eq!(result.tests.len(), 1);
     +    }
     +
     +    #[test]
    @@ crates/e2etest/src/filter.rs (new)
     +        let test_cases = make_test_cases();
     +        let filters = vec!["simple".to_string()];
     +        let result = Filter::new(&filters, test_cases.as_ref());
    -+        assert!(result.map["crud"].contains("simple_create"));
    -+        assert!(result.map["other"].contains("simple_misc"));
    -+        assert_eq!(result.map.len(), 2);
    ++        assert!(result.groups.contains("root::crud"));
    ++        assert!(result.tests["root::crud"].contains("simple_create"));
    ++        assert!(result.groups.contains("root::other"));
    ++        assert!(result.tests["root::other"].contains("simple_misc"));
    ++        assert_eq!(result.groups.len(), 2);
    ++        assert_eq!(result.tests.len(), 2);
     +    }
     +
     +    #[test]
    @@ crates/e2etest/src/filter.rs (new)
     +        let test_cases = make_test_cases();
     +        let filters = vec!["crud::simple".to_string()];
     +        let result = Filter::new(&filters, test_cases.as_ref());
    -+        assert!(result.map["crud"].contains("simple_create"));
    -+        assert_eq!(result.map.len(), 1);
    ++        assert!(result.groups.contains("root::crud"));
    ++        assert!(result.tests["root::crud"].contains("simple_create"));
    ++        assert_eq!(result.groups.len(), 1);
    ++        assert_eq!(result.tests.len(), 1);
     +    }
     +
     +    #[test]
    @@ crates/e2etest/src/filter.rs (new)
     +        let test_cases = make_test_cases();
     +        let filters = vec!["crud::".to_string()];
     +        let result = Filter::new(&filters, test_cases.as_ref());
    -+        assert!(result.map.contains_key("crud"));
    -+        assert!(result.map["crud"].is_empty());
    -+        assert_eq!(result.map.len(), 1);
    ++        assert!(result.groups.contains("root::crud"));
    ++        assert!(result.tests.contains_key("root::crud"));
    ++        assert!(result.tests["root::crud"].is_empty());
    ++        assert_eq!(result.groups.len(), 1);
    ++        assert_eq!(result.tests.len(), 1);
     +    }
     +
     +    #[test]
    @@ crates/e2etest/src/filter.rs (new)
     +        let test_cases = make_test_cases();
     +        let filters = vec!["::simple".to_string()];
     +        let result = Filter::new(&filters, test_cases.as_ref());
    -+        assert!(result.map["crud"].contains("simple_create"));
    -+        assert!(result.map["other"].contains("simple_misc"));
    -+        assert_eq!(result.map.len(), 2);
    ++        assert!(result.groups.contains("root::crud"));
    ++        assert!(result.tests["root::crud"].contains("simple_create"));
    ++        assert!(result.groups.contains("root::other"));
    ++        assert!(result.tests["root::other"].contains("simple_misc"));
    ++        assert_eq!(result.groups.len(), 2);
    ++        assert_eq!(result.tests.len(), 2);
     +    }
     +
     +    #[test]
    @@ crates/e2etest/src/filter.rs (new)
     +        let test_cases = make_overlapping_test_cases();
     +        let filters = vec!["\"crud\"::".to_string()];
     +        let result = Filter::new(&filters, test_cases.as_ref());
    -+        assert!(result.map.contains_key("crud"));
    -+        assert!(result.map["crud"].is_empty());
    -+        assert_eq!(result.map.len(), 1);
    ++        assert!(result.groups.contains("root::crud"));
    ++        assert!(result.tests.contains_key("root::crud"));
    ++        assert!(result.tests["root::crud"].is_empty());
    ++        assert_eq!(result.groups.len(), 1);
    ++        assert_eq!(result.tests.len(), 1);
     +    }
     +
     +    #[test]
    @@ crates/e2etest/src/filter.rs (new)
     +        let test_cases = make_overlapping_test_cases();
     +        let filters = vec!["::\"simple_create\"".to_string()];
     +        let result = Filter::new(&filters, test_cases.as_ref());
    -+        assert!(result.map["crud"].contains("simple_create"));
    -+        assert!(!result.map["crud"].contains("simple_create_extra"));
    -+        assert!(result.map["crud_extra"].contains("simple_create"));
    -+        assert!(!result.map["crud_extra"].contains("simple_create_additional"));
    -+        assert_eq!(result.map.len(), 2);
    ++        assert!(result.groups.contains("root::crud"));
    ++        assert!(result.tests["root::crud"].contains("simple_create"));
    ++        assert!(!result.tests["root::crud"].contains("simple_create_extra"));
    ++        assert!(result.groups.contains("root::crud_extra"));
    ++        assert!(result.tests["root::crud_extra"].contains("simple_create"));
    ++        assert!(!result.tests["root::crud_extra"].contains("simple_create_additional"));
    ++        assert_eq!(result.groups.len(), 2);
    ++        assert_eq!(result.tests.len(), 2);
     +    }
     +
     +    #[test]
    @@ crates/e2etest/src/filter.rs (new)
     +        let test_cases = make_overlapping_test_cases();
     +        let filters = vec!["\"crud\"::\"simple_create\"".to_string()];
     +        let result = Filter::new(&filters, test_cases.as_ref());
    -+        assert!(result.map["crud"].contains("simple_create"));
    -+        assert!(!result.map["crud"].contains("simple_create_extra"));
    -+        assert_eq!(result.map.len(), 1);
    ++        assert!(result.groups.contains("root::crud"));
    ++        assert!(result.tests["root::crud"].contains("simple_create"));
    ++        assert!(!result.tests["root::crud"].contains("simple_create_extra"));
    ++        assert_eq!(result.groups.len(), 1);
    ++        assert_eq!(result.tests.len(), 1);
     +    }
     +}
     
2:  2eb91a7 ! 2:  2ea1835 e2etest: refactor running tests
    @@ crates/e2etest/src/run.rs (new)
     +use crate::group::RunGroup;
     +use crate::statistics::Statistics;
     +use async_backtrace::framed;
    -+use std::iter;
     +use std::time::Duration;
     +use tracing::Instrument;
     +use tracing::error;
    @@ crates/e2etest/src/run.rs (new)
     +) -> Statistics {
     +    let backtrace = backtrace::setup_panic_hook();
     +
    -+    let stats = if filter.consider_group(iter::once(""), group.name()) {
    -+        group
    -+            .run_group(vec![], fixtures, filter, backtrace, default_timeout)
    -+            .instrument(error_span!("group", "{}", group.name()))
    -+            .await
    -+    } else {
    -+        Statistics::new()
    -+    };
    ++    let stats = group
    ++        .run_group(vec![], fixtures, filter, backtrace, default_timeout)
    ++        .instrument(error_span!("group", "{}", group.name()))
    ++        .await;
     +
     +    backtrace::clear_panic_hook();
     +
3:  b60396a = 3:  e8a2c76 e2etest-macros: implement group! and test! macros
4:  4b253e4 = 4:  f77b302 e2etest: integrate redisigned framework
5:  5b06a1d ! 5:  e27c80e e2etest: implement integration tests
    @@ crates/e2etest/Cargo.toml: futures.workspace = true
     +[dev-dependencies]
     +linkme.workspace = true
     
    - ## crates/e2etest/tests/integration/hierarchy.rs (new) ##
    + ## crates/e2etest/tests/integration/filter.rs (new) ##
     @@
     +/*
     + * Copyright 2026-present ScyllaDB
    @@ crates/e2etest/tests/integration/hierarchy.rs (new)
     +use e2etest::Fixture;
     +use e2etest::Fixtures;
     +use e2etest::Setup;
    -+use std::collections::HashSet;
     +use std::sync::Arc;
     +use std::sync::atomic::AtomicUsize;
     +use std::sync::atomic::Ordering;
    @@ crates/e2etest/tests/integration/hierarchy.rs (new)
     +
     +struct Counter(Arc<AtomicUsize>);
     +
    -+impl Fixture for Counter {
    -+    async fn setup(_: &mut impl Setup) -> Self {
    -+        unreachable!()
    -+    }
    -+    async fn teardown(self) {
    -+        unreachable!()
    ++#[derive(Clone)]
    ++struct FixtureCount(Arc<Counter>);
    ++
    ++impl Fixture for FixtureCount {
    ++    async fn setup(setup: &mut impl Setup) -> Self {
    ++        let counter = setup.get::<Counter>().await.unwrap();
    ++        Self(counter)
     +    }
    ++    async fn teardown(self) {}
    ++}
    ++
    ++e2etest::group!(name = filter_root);
    ++e2etest::group!(name = filter_group1, parent = filter_root);
    ++e2etest::group!(name = filter_group1_1, parent = filter_group1);
    ++e2etest::group!(name = filter_group1_2, parent = filter_group1);
    ++e2etest::group!(name = filter_group2, parent = filter_root);
    ++e2etest::group!(name = filter_group2_1, parent = filter_group2);
    ++e2etest::group!(name = filter_group2_2, parent = filter_group2);
    ++
    ++#[e2etest::test(group = filter_group1)]
    ++async fn filter_test1_1(fixture: Arc<FixtureCount>) {
    ++    fixture.0.0.fetch_add(1, Ordering::Relaxed);
    ++}
    ++
    ++#[e2etest::test(group = filter_group1)]
    ++async fn filter_test1_2(fixture: Arc<FixtureCount>) {
    ++    fixture.0.0.fetch_add(1, Ordering::Relaxed);
    ++}
    ++
    ++#[e2etest::test(group = filter_group1_1)]
    ++async fn filter_test1_1_1(fixture: Arc<FixtureCount>) {
    ++    fixture.0.0.fetch_add(1, Ordering::Relaxed);
    ++}
    ++
    ++#[e2etest::test(group = filter_group1_1)]
    ++async fn filter_test1_1_2(fixture: Arc<FixtureCount>) {
    ++    fixture.0.0.fetch_add(1, Ordering::Relaxed);
    ++}
    ++
    ++#[e2etest::test(group = filter_group1_2)]
    ++async fn filter_test1_2_1(fixture: Arc<FixtureCount>) {
    ++    fixture.0.0.fetch_add(1, Ordering::Relaxed);
    ++}
    ++
    ++#[e2etest::test(group = filter_group1_2)]
    ++async fn filter_test1_2_2(fixture: Arc<FixtureCount>) {
    ++    fixture.0.0.fetch_add(1, Ordering::Relaxed);
    ++}
    ++
    ++#[e2etest::test(group = filter_group2)]
    ++async fn filter_test2_1(fixture: Arc<FixtureCount>) {
    ++    fixture.0.0.fetch_add(1, Ordering::Relaxed);
    ++}
    ++
    ++#[e2etest::test(group = filter_group2)]
    ++async fn filter_test2_2(fixture: Arc<FixtureCount>) {
    ++    fixture.0.0.fetch_add(1, Ordering::Relaxed);
    ++}
    ++
    ++#[e2etest::test(group = filter_group2_1)]
    ++async fn filter_test2_1_1(fixture: Arc<FixtureCount>) {
    ++    fixture.0.0.fetch_add(1, Ordering::Relaxed);
    ++}
    ++
    ++#[e2etest::test(group = filter_group2_1)]
    ++async fn filter_test2_1_2(fixture: Arc<FixtureCount>) {
    ++    fixture.0.0.fetch_add(1, Ordering::Relaxed);
    ++}
    ++
    ++#[e2etest::test(group = filter_group2_2)]
    ++async fn filter_test2_2_1(fixture: Arc<FixtureCount>) {
    ++    fixture.0.0.fetch_add(1, Ordering::Relaxed);
    ++}
    ++
    ++#[e2etest::test(group = filter_group2_2)]
    ++async fn filter_test2_2_2(fixture: Arc<FixtureCount>) {
    ++    fixture.0.0.fetch_add(1, Ordering::Relaxed);
    ++}
    ++
    ++#[tokio::test]
    ++async fn filter_by_group() {
    ++    let counter = Arc::new(AtomicUsize::new(0));
    ++    let init = async |_: &Args, fixtures: &Fixtures| {
    ++        fixtures.add_permanent(Counter(Arc::clone(&counter))).await;
    ++    };
    ++
    ++    e2etest::run(
    ++        ["validator", "run", "group1::"],
    ++        init,
    ++        filter_root(),
    ++        Duration::from_secs(1),
    ++    )
    ++    .await
    ++    .unwrap();
    ++
    ++    assert_eq!(counter.load(Ordering::Relaxed), 6);
     +}
     +
    ++#[tokio::test]
    ++async fn filter_by_test() {
    ++    let counter = Arc::new(AtomicUsize::new(0));
    ++    let init = async |_: &Args, fixtures: &Fixtures| {
    ++        fixtures.add_permanent(Counter(Arc::clone(&counter))).await;
    ++    };
    ++
    ++    e2etest::run(
    ++        ["validator", "run", "::test2_2"],
    ++        init,
    ++        filter_root(),
    ++        Duration::from_secs(1),
    ++    )
    ++    .await
    ++    .unwrap();
    ++
    ++    assert_eq!(counter.load(Ordering::Relaxed), 3);
    ++}
    +
    + ## crates/e2etest/tests/integration/hierarchy.rs (new) ##
    +@@
    ++/*
    ++ * Copyright 2026-present ScyllaDB
    ++ * SPDX-License-Identifier: MIT OR Apache-2.0
    ++ */
    ++
    ++use e2etest::Fixture;
    ++use e2etest::Fixtures;
    ++use e2etest::Setup;
    ++use std::collections::HashSet;
    ++use std::sync::Arc;
    ++use std::sync::atomic::AtomicUsize;
    ++use std::sync::atomic::Ordering;
    ++use std::time::Duration;
    ++
    ++#[derive(clap::Args)]
    ++struct Args {}
    ++
    ++struct Counter(Arc<AtomicUsize>);
    ++
     +#[derive(Clone)]
     +struct FixtureRoot(Arc<Counter>);
     +
    @@ crates/e2etest/tests/integration/main.rs (new)
     + * SPDX-License-Identifier: MIT OR Apache-2.0
     + */
     +
    ++mod filter;
     +mod hierarchy;
     +mod skip;
     +mod timeout;
    @@ crates/e2etest/tests/integration/skip.rs (new)
     +
     +struct Counter(Arc<AtomicUsize>);
     +
    -+impl e2etest::Fixture for Counter {
    -+    async fn setup(_: &mut impl Setup) -> Self {
    -+        unreachable!()
    -+    }
    -+    async fn teardown(self) {
    -+        unreachable!()
    -+    }
    -+}
    -+
     +#[derive(Clone)]
     +struct Fixture(Arc<Counter>);
     +
    @@ crates/e2etest/tests/integration/timeout.rs (new)
     +
     +struct Counter(Arc<AtomicUsize>);
     +
    -+impl e2etest::Fixture for Counter {
    -+    async fn setup(_: &mut impl Setup) -> Self {
    -+        unreachable!()
    -+    }
    -+    async fn teardown(self) {
    -+        unreachable!()
    -+    }
    -+}
    -+
     +#[derive(Clone)]
     +struct Fixture(Arc<Counter>);
     +

[ewienik]

Changelog for 2675812:

• refactor Statistics struct
• refactor e2etest::run to use a new Config struct instead of clap
• remove clap dependency
• introduce RunContext struct
• remove unneeded visibility for structs

[ewienik]

Changelog for bd2c639

• fix commit descriptions

range diff

 1:  ca92dc6 !  1:  9c7eef5 e2etest: refactor Backtrace struct
    @@ Metadata
      ## Commit message ##
         e2etest: refactor Backtrace struct
     
    -    This commit copies Backtrace struct form testcase.rs into a separate module.
    +    This commit copies Backtrace struct from testcase.rs into a separate module.
         This module will be used in the following commits.
     
         It is a part of redesigning e2etest framework to use group! and test! macros
 2:  1e10f50 !  2:  6594884 e2etest: refactor Statistics struct
    @@ Metadata
      ## Commit message ##
         e2etest: refactor Statistics struct
     
    -    This commit copies Statistics struct form testcase.rs into a separate module.
    +    This commit copies Statistics struct from testcase.rs into a separate module.
         This module will be used in the following commits.
     
         It is a part of redesigning e2etest framework to use group! and test! macros
 3:  9182c60 =  3:  c5c14ac e2etest: implement Fixture trait and Fixtures struct
 4:  51ae25d =  4:  2ce0e92 e2etest: introduce Run struct
 5:  3ddeace =  5:  d2af6c9 e2etest: refactor tasks for fixtures and tests
 6:  0455b73 =  6:  90fa171 e2etest: introduce Test and RunTest traits
 7:  1badb13 =  7:  e95c33b e2etest: introduce Group and RunGroup traits and refactor filtering
 8:  ac56236 =  8:  07fa5c4 e2etest: refactor running tests
 9:  0c9bfd7 =  9:  6d7e564 e2etest-macros: implement group! and test! macros
10:  350c522 = 10:  e30a0ed e2etest: integrate redisigned framework
11:  2675812 = 11:  bd2c639 e2etest: implement integration tests

[ewienik]

Changelog for daef677

• fix verification of Arc in test fn signature
• fix checking asyncness and return type without panic!

range diff

1:  3e768b9 ! 1:  9776435 e2etest-macros: implement group! and test! macros
    @@ crates/e2etest-macros/src/lib.rs (new)
     +            if let FnArg::Typed(pat_type) = arg
     +                && let Type::Path(type_path) = &*pat_type.ty
     +                && let Some(path_segment) = type_path.path.segments.first()
    ++                && path_segment.ident == "Arc"
     +                && let PathArguments::AngleBracketed(angle_bracketed) = &path_segment.arguments
     +                && let Some(generic_arg) = angle_bracketed.args.first()
     +                && let GenericArgument::Type(Type::Path(fixture_type)) = generic_arg
    @@ crates/e2etest-macros/src/lib.rs (new)
     +    let test_fixture = Ident::new(&test_fixture_name(&name_string), name.span());
     +    let test_type = Ident::new(&test_type_name(&name_string), name.span());
     +    let test_register = Ident::new(&test_register_name(&name_string), name.span());
    -+    assert!(
    -+        run.sig.asyncness.is_some(),
    -+        "Expected the test function to be async"
    -+    );
    -+    assert!(
    -+        matches!(run.sig.output, ReturnType::Default),
    -+        "Expected the test function to return ()"
    -+    );
    ++    if run.sig.asyncness.is_none() {
    ++        return Err(syn::Error::new(
    ++            run.sig.span(),
    ++            "Expected the test function to be async",
    ++        ));
    ++    }
    ++    if !matches!(run.sig.output, ReturnType::Default) {
    ++        return Err(syn::Error::new(
    ++            run.sig.output.span(),
    ++            "Expected the test function to return ()",
    ++        ));
    ++    }
     +
     +    let timeout = if let Some(timeout) = &params.timeout {
     +        quote! {
2:  c2eba19 = 2:  4bcb51a e2etest: integrate redisigned framework
3:  f4840a0 = 3:  daef677 e2etest: implement integration tests

[QuerthDP]

Leaving some comments now.
I'm still to review the last 3 commits.
Most of them are nitpicks, nothing major I suppose.

Some of them were taken from AI review. Consider if you think they're relevant. I've prefiltered them anyway, so left only those that sound reasonable. All are marked.

[knowack1]

To be honest, I haven't dived deeply into the macro code itself. However, even without looking at the implementation details, this PR provides a great interface for test case definitions, clean fixture capabilities, and solid cleanup of the existing code. It definitely deserves approval.

[ewienik]

Changelog for d30d341

• fix commit descriptions
• fix methods' visibility
• rename increment_filtered into increment_included
• add pub method docs
• remove Debug from Fixtures
• remove not used conversions
• rename file to group
• use DEFAULT_TIMEOUT

range diff

 1:  28950aa !  1:  4966697 e2etest: refactor Statistics struct
    @@ crates/e2etest/src/statistics.rs (new)
     +        }
     +    }
     +
    -+    pub(crate) fn append(&mut self, other: &Self) {
    ++    fn append(&mut self, other: &Self) {
     +        self.total += other.total;
     +        self.included += other.included;
     +        self.launched += other.launched;
    @@ crates/e2etest/src/statistics.rs (new)
     +        inner.total += count;
     +    }
     +
    -+    pub(crate) fn increment_filtered(&self, count: usize) {
    ++    pub(crate) fn increment_included(&self, count: usize) {
     +        let mut inner = self.0.lock().unwrap();
     +        inner.included += count;
     +    }
    @@ crates/e2etest/src/statistics.rs (new)
     +        inner.skipped += 1;
     +    }
     +
    ++    /// Returns true if there are no failed tests or groups.
     +    pub fn is_success(&self) -> bool {
     +        let inner = self.0.lock().unwrap();
     +        inner.failed_names.is_empty()
     +    }
     +
    ++    /// Returns number of total defined tests.
     +    pub fn total(&self) -> usize {
     +        let inner = self.0.lock().unwrap();
     +        inner.total
     +    }
     +
    ++    /// Returns number of tests included in the run after filtering.
     +    pub fn included(&self) -> usize {
     +        let inner = self.0.lock().unwrap();
     +        inner.included
     +    }
     +
    ++    /// Returns number of tests that were launched.
     +    pub fn launched(&self) -> usize {
     +        let inner = self.0.lock().unwrap();
     +        inner.launched
     +    }
     +
    ++    /// Returns number of tests that passed successfully.
     +    pub fn ok(&self) -> usize {
     +        let inner = self.0.lock().unwrap();
     +        inner.ok
     +    }
     +
    ++    /// Returns number of tests that failed.
     +    pub fn failed_tests(&self) -> usize {
     +        let inner = self.0.lock().unwrap();
     +        inner.failed_tests
     +    }
     +
    ++    /// Returns number of groups that failed.
     +    pub fn failed_groups(&self) -> usize {
     +        let inner = self.0.lock().unwrap();
     +        inner.failed_groups
     +    }
     +
    ++    /// Returns number of tests that were skipped.
     +    pub fn skipped(&self) -> usize {
     +        let inner = self.0.lock().unwrap();
     +        inner.skipped
     +    }
     +
    ++    /// Returns a list of names of failed tests and groups.
     +    pub fn failed_names(&self) -> Vec<String> {
     +        let inner = self.0.lock().unwrap();
     +        inner.failed_names.clone()
 2:  756ae1c !  2:  18cf9b6 e2etest: implement Fixture trait and Fixtures struct
    @@ Commit message
     
         This commit implements Fixtures struct as a cache for types that implement
         Fixture trait. There is permanent cache - types with Any trait can be stored
    -    there.  There is also a cache for fixtures - the user adds the type by running
    +    there. There is also a cache for fixtures - the user adds the type by running
         setup method. Calling teardown method removes all fixtures which are no longer
         in usage by anyone and call teardown on them.
     
    @@ crates/e2etest/src/fixture.rs (new)
     +#[derive(Clone)]
     +pub(crate) struct Fixtures(Arc<Mutex<Inner>>);
     +
    -+impl std::fmt::Debug for Fixtures {
    -+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    -+        f.debug_tuple("Fixtures").finish()
    -+    }
    -+}
    -+
     +impl Fixtures {
     +    pub(crate) fn new() -> Self {
     +        Self(Arc::new(Mutex::new(Inner::new())))
    @@ crates/e2etest/src/fixture.rs (new)
     +        async move { inner.lock().await.setup::<F>().await }
     +    }
     +
    -+    pub fn teardown(&self) -> impl Future<Output = ()> + Send + use<> {
    ++    pub(crate) fn teardown(&self) -> impl Future<Output = ()> + Send + use<> {
     +        let inner = Arc::clone(&self.0);
     +        async move {
     +            inner.lock().await.teardown().await;
    @@ crates/e2etest/src/fixture.rs (new)
     +/// It allows you to access to the other fixtures while setting up a fixture. It
     +/// is passed to the `setup` method of a fixture.
     +pub trait Setup: Send {
    -+    /// Set up a new fixture. If the fixture is already set up, this will do nothing.
    ++    /// Set up a new fixture. If the fixture is already set up, it will return the existing
    ++    /// fixture. Otherwise, it will set up a new fixture and return it.
     +    fn setup<F: Fixture>(&mut self) -> impl Future<Output = Arc<F>> + Send;
     +
     +    /// Get a fixture. If the fixture is not set up, this will return `None`.
 3:  dd609d1 !  3:  e05331b e2etest: introduce Run struct
    @@ Metadata
     Author: Pawel Pery <pawel.pery@scylladb.com>
     
      ## Commit message ##
    -    e2etest: introduce Run struct
    +    e2etest: introduce RunContext struct
     
    -    This commit introduces Run struct for storing context of all structs that
    -    support running tests.
    +    This commit introduces RunContext struct for storing context of all structs
    +    that support running tests.
     
      ## crates/e2etest/src/lib.rs ##
     @@
 4:  fff4ec7 =  4:  e38f832 e2etest: refactor tasks for fixtures and tests
 5:  a055591 =  5:  953f81a e2etest: introduce Test and RunTest traits
 6:  162af8f !  6:  9aaab19 e2etest: introduce Group and RunGroup traits and refactor filtering
    @@ crates/e2etest/src/filter.rs (new)
     +/// Represents the filter configuration for test execution.
     +#[derive(Clone, Debug)]
     +pub(crate) struct Filter {
    -+    /// - Key: test file name (e.g., "crud", "full_scan")
    -+    /// - Value: HashSet of specific test names within that file (empty means run all tests in file)
    ++    /// - Key: test group name (e.g., "crud", "full_scan")
    ++    /// - Value: HashSet of specific test names within that group (empty means run all tests in
    ++    ///   group)
     +    tests: Arc<HashMap<String, HashSet<String>>>,
     +    groups: Arc<HashSet<String>>,
     +}
    @@ crates/e2etest/src/filter.rs (new)
     +    }
     +
     +    /// Parse command line filters into the expected filter format for test execution.
    -+    /// Returns a HashMap where:
     +    pub(crate) fn new(filters: &[String], group: &dyn RunGroup) -> Self {
     +        let mut filter_map = HashMap::new();
     +        let mut group_set = HashSet::new();
    @@ crates/e2etest/src/filter.rs (new)
     +                    }
     +                }
     +            } else {
    -+                // Not found `::`, check for matching both file and test case name
    ++                // Not found `::`, check for matching both group and test case name
     +                let filter = FilterMatcher::new(filter);
     +
     +                for (group_name, test_name) in group
    @@ crates/e2etest/src/filter.rs (new)
     +    #[derive(Clone)]
     +    struct GroupFixture;
     +
    -+    impl From<&()> for GroupFixture {
    -+        fn from(_: &()) -> Self {
    -+            Self
    -+        }
    -+    }
    -+
     +    impl From<&GroupFixture> for GroupFixture {
     +        fn from(_: &GroupFixture) -> Self {
     +            Self
    @@ crates/e2etest/src/filter.rs (new)
     +        let test_cases = make_test_cases();
     +        let filters: Vec<String> = vec!["::".to_string()];
     +        let result = Filter::new(&filters, test_cases.as_ref());
    -+        // It should contain all available test files with empty test cases (running all)
    ++        // It should contain all available test groups with empty test cases (running all)
     +        assert_eq!(result.groups.len(), 3);
     +        assert_eq!(result.tests.len(), 3);
     +        assert!(result.groups.contains("root::crud"));
    @@ crates/e2etest/src/filter.rs (new)
     +    }
     +
     +    #[test]
    -+    fn test_file_partial_match() {
    ++    fn test_group_partial_match() {
     +        let test_cases = make_test_cases();
     +        let filters = vec!["crud".to_string()];
     +        let result = Filter::new(&filters, test_cases.as_ref());
    @@ crates/e2etest/src/filter.rs (new)
     +    }
     +
     +    #[test]
    -+    fn test_file_and_test_case_syntax() {
    ++    fn test_group_and_test_case_syntax() {
     +        let test_cases = make_test_cases();
     +        let filters = vec!["crud::simple".to_string()];
     +        let result = Filter::new(&filters, test_cases.as_ref());
    @@ crates/e2etest/src/filter.rs (new)
     +    }
     +
     +    #[test]
    -+    fn test_file_and_empty_test_case_syntax() {
    ++    fn test_group_and_empty_test_case_syntax() {
     +        let test_cases = make_test_cases();
     +        let filters = vec!["crud::".to_string()];
     +        let result = Filter::new(&filters, test_cases.as_ref());
    @@ crates/e2etest/src/filter.rs (new)
     +    }
     +
     +    #[test]
    -+    fn test_empty_file_and_test_case_syntax() {
    ++    fn test_empty_group_and_test_case_syntax() {
     +        let test_cases = make_test_cases();
     +        let filters = vec!["::simple".to_string()];
     +        let result = Filter::new(&filters, test_cases.as_ref());
    @@ crates/e2etest/src/filter.rs (new)
     +    }
     +
     +    #[test]
    -+    fn test_exact_file_match_syntax() {
    ++    fn test_exact_group_match_syntax() {
     +        let test_cases = make_overlapping_test_cases();
     +        let filters = vec!["\"crud\"::".to_string()];
     +        let result = Filter::new(&filters, test_cases.as_ref());
    @@ crates/e2etest/src/filter.rs (new)
     +    }
     +
     +    #[test]
    -+    fn test_exact_file_and_test_case_syntax() {
    ++    fn test_exact_group_and_test_case_syntax() {
     +        let test_cases = make_overlapping_test_cases();
     +        let filters = vec!["\"crud\"::\"simple_create\"".to_string()];
     +        let result = Filter::new(&filters, test_cases.as_ref());
    @@ crates/e2etest/src/group.rs (new)
     +    #[derive(Clone)]
     +    struct GroupFixture;
     +
    -+    impl From<&()> for GroupFixture {
    -+        fn from(_: &()) -> Self {
    -+            Self
    -+        }
    -+    }
    -+
     +    impl Fixture for GroupFixture {
     +        async fn setup(_: &mut impl Setup) -> Self {
     +            Self
 7:  0aa3021 !  7:  cbebcc1 e2etest: refactor running tests
    @@ Metadata
      ## Commit message ##
         e2etest: refactor running tests
     
    -    This commit moves run function into a separate module. It refactors to support
    -    running tests from single master group.
    +    This commit moves run function into a separate module. The commit refactors the
    +    code to support running tests from a single master group.
     
         It is a part of redesigning e2etest framework to use group! and test! macros
         for defining tests with the hierarchy of fixtures.
    @@ crates/e2etest/src/run.rs: impl Debug for RunContext {
     +        .with_default_timeout(default_timeout);
     +
     +    ctx.statistics.increment_total(group.test_names().len());
    -+    ctx.statistics.increment_filtered(
    ++    ctx.statistics.increment_included(
     +        group
     +            .test_names()
     +            .iter()
 8:  9776435 =  8:  16d57c6 e2etest-macros: implement group! and test! macros
 9:  4bcb51a !  9:  9264754 e2etest: integrate redisigned framework
    @@ crates/e2etest/src/lib.rs: mod run;
     +        Self {
     +            permanent_fixtures: Vec::new(),
     +            filters: Vec::new(),
    -+            default_timeout: Duration::from_secs(60),
    ++            default_timeout: DEFAULT_TIMEOUT,
              }
          }
     -
10:  daef677 = 10:  d30d341 e2etest: implement integration tests

[ewienik]

Changelog for 10d2c5d

• fix documentation for run
• fix README

range diff

1:  9264754 ! 1:  a547755 e2etest: integrate redisigned framework
    @@ crates/e2etest/Cargo.toml: license.workspace = true
      itertools.workspace = true
      tokio.workspace = true
     
    + ## crates/e2etest/README.md ##
    +@@ crates/e2etest/README.md: other testing actors - so the provided binary ought to be run in the unshared
    + environment.  In the future `e2etest` will provide the unshared environment
    + directly, without additional setup.
    + 
    ++To use macros provided by `e2etest`, you need to add `linkme` and
    ++`async-backtrace` to your `Cargo.toml` dependencies.
    ++
    ++
    + **Sample code for using `e2etest`**
    + 
    + ```rust
    +-use e2etest::TestCase;
    ++mod sample {
    ++
    + use std::net::Ipv4Addr;
    ++use std::sync::Arc;
    + use std::time::Duration;
    + 
    +-#[derive(clap::Args)]
    +-struct Args {
    +-    #[arg(short, long, default_value = "127.0.100.1")]
    ++#[derive(Clone, Copy)]
    ++pub struct FixtureCfg {
    ++    pub dns_ip: Ipv4Addr,
    ++}
    ++
    ++#[derive(Clone, Copy)]
    ++pub struct FixtureOne {
    +     dns_ip: Ipv4Addr,
    + }
    + 
    +-fn init(args: &Args) {
    ++impl e2etest::Fixture for FixtureOne {
    ++    async fn setup(setup: &mut impl e2etest::Setup) -> Self {
    ++        let cfg = setup.get::<FixtureCfg>().await.unwrap();
    ++        Self { dns_ip: cfg.dns_ip }
    ++    }
    ++
    ++    async fn teardown(self) { }
    + }
    + 
    +-#[derive(Clone)]
    +-struct Fixture {
    +-    dns_ip: Ipv4Addr,
    ++#[derive(Clone, Copy)]
    ++pub struct FixtureTwo {
    ++    octet: u8,
    + }
    + 
    +-async fn fixture(args: &Args) -> Fixture {
    +-    Fixture {
    +-        dns_ip: args.dns_ip,
    ++impl e2etest::Fixture for FixtureTwo {
    ++    async fn setup(setup: &mut impl e2etest::Setup) -> Self {
    ++        let one = setup.setup::<FixtureOne>().await;
    ++        Self { octet: one.dns_ip.octets()[2] }
    +     }
    ++
    ++    async fn teardown(self) { }
    + }
    + 
    +-async fn init_testcase(fixture: Fixture) {
    ++#[derive(Clone, Copy)]
    ++pub struct FixtureThree {
    ++    number: usize,
    ++}
    ++
    ++impl e2etest::Fixture for FixtureThree {
    ++    async fn setup(setup: &mut impl e2etest::Setup) -> Self {
    ++        let two = setup.setup::<FixtureTwo>().await;
    ++        Self { number: two.octet as usize * 1024 }
    ++    }
    ++
    ++    async fn teardown(self) { }
    ++}
    ++
    ++e2etest::group!(name = root, fixtures = (FixtureOne));
    ++
    ++e2etest::group!(name = group, fixtures = (FixtureTwo), parent = root);
    ++
    ++#[e2etest::test(group = group, timeout = Duration::from_secs(5))]
    ++async fn dns_ip_100(one: Arc<FixtureOne>, two: Arc<FixtureTwo>) {
    ++    assert_eq!(one.dns_ip, Ipv4Addr::new(127, 0, 100, 1));
    ++    assert_eq!(two.octet, 100);
    + }
    + 
    +-async fn cleanup_testcase(fixture: Fixture) {
    ++#[e2etest::test(group = group, skip = true)]
    ++async fn dns_ip_200(one: Arc<FixtureOne>) {
    ++    assert_eq!(one.dns_ip, Ipv4Addr::new(127, 0, 200, 1));
    + }
    + 
    +-async fn dns_ip(fixture: Fixture) {
    +-    assert_eq!(fixture.dns_ip, Ipv4Addr::new(127, 0, 100, 1));
    ++#[e2etest::test(group = group)]
    ++async fn number_and_octet(two: Arc<FixtureTwo>, three: Arc<FixtureThree>) {
    ++    assert_eq!(two.octet, 100);
    ++    assert_eq!(three.number, 100 * 1024);
    + }
    + 
    +-async fn register() -> Vec<(String, TestCase<Fixture>)> {
    +-    let timeout = Duration::from_secs(10);
    +-    let testcase = TestCase::empty()
    +-        .with_init(timeout, init_testcase)
    +-        .with_cleanup(timeout, cleanup_testcase)
    +-        .with_test("dns_ip", timeout, dns_ip);
    +-    vec![("simple".to_string(), testcase)]
    + }
    + 
    +-e2etest::run(["validator", "run"], init, register, fixture);
    ++tokio::runtime::Runtime::new().unwrap().block_on(async move {
    ++    use std::net::Ipv4Addr;
    ++    use std::time::Duration;
    ++
    ++    let config = e2etest::Config::default()
    ++        .with_permanent_fixture(sample::FixtureCfg { dns_ip: Ipv4Addr::new(127, 0, 100, 1) })
    ++        .with_default_timeout(Duration::from_secs(10));
    ++    let stats = e2etest::run(config, sample::root()).await;
    ++    assert!(stats.is_success());
    ++    assert_eq!(stats.total(), 3);
    ++    assert_eq!(stats.launched(), 2);
    ++    assert_eq!(stats.ok(), 2);
    ++    assert_eq!(stats.skipped(), 1);
    ++});
    + ```
    + 
    + **Sample code for script to run in the unshared environment:**
    +
      ## crates/e2etest/src/lib.rs ##
     @@
      //! See this simple example:
    @@ crates/e2etest/src/lib.rs: mod run;
      }
      
      /// Main entry point for running tests.
    -@@ crates/e2etest/src/lib.rs: where
    - /// It takes command line arguments, an initialization function,
    - /// a test registration function, and a fixture creation function.
    + ///
    +-/// It takes command line arguments, an initialization function,
    +-/// a test registration function, and a fixture creation function.
    ++/// It takes `Config` argument and a root group. Returns `Statistics` about the test run.
      #[framed]
     -pub fn run<A, F>(
     -    args: impl IntoIterator<Item = impl Into<OsString> + Clone>,
2:  d30d341 = 2:  10d2c5d e2etest: implement integration tests

[ewienik]

Changelog for 07cf545

• s/failed_test/failed_group

range diff

 1:  4966697 !  1:  64134ff e2etest: refactor Statistics struct
    @@ crates/e2etest/src/statistics.rs (new)
     +        inner.failed_names.push(failed_test.into());
     +    }
     +
    -+    pub(crate) fn record_group_failure(&self, failed_test: impl Into<String>) {
    ++    pub(crate) fn record_group_failure(&self, failed_group: impl Into<String>) {
     +        let mut inner = self.0.lock().unwrap();
     +        inner.failed_groups += 1;
    -+        inner.failed_names.push(failed_test.into());
    ++        inner.failed_names.push(failed_group.into());
     +    }
     +
     +    pub(crate) fn increment_skipped(&self) {
 2:  18cf9b6 =  2:  47a25f1 e2etest: implement Fixture trait and Fixtures struct
 3:  e05331b =  3:  feb1acf e2etest: introduce RunContext struct
 4:  e38f832 =  4:  679e423 e2etest: refactor tasks for fixtures and tests
 5:  953f81a =  5:  39c5fa9 e2etest: introduce Test and RunTest traits
 6:  9aaab19 =  6:  c101572 e2etest: introduce Group and RunGroup traits and refactor filtering
 7:  cbebcc1 =  7:  65b6b15 e2etest: refactor running tests
 8:  16d57c6 =  8:  d2762b2 e2etest-macros: implement group! and test! macros
 9:  a547755 =  9:  aae2bf5 e2etest: integrate redisigned framework
10:  10d2c5d = 10:  07cf545 e2etest: implement integration tests

[QuerthDP]

Everything looks correct. Very good contribution.

Similar to what @knowack1 already said, this PR is a great interface and long waited feature so I approve without hesitation.
I'm feeling like I understood 80% of it after long reviews, but I think this level of understanding allows me to confidently approve.