Replace cryptography with a small Python/Rust package in integration tests

[slimreaper35]

The pip/e2e_rust_extensions branch is used as the main integration test for Hermeto’s Python + Rust extension support. The scenario depends on building cryptography from source via a pinned GitHub archive URL.

Compiling cryptography’s Rust code and its large dependency graph is extremely time-consuming (by far the slowest integration test). Also, the SBOM for this test is huge, ~7,600 lines, mostly transitive Cargo crates.

Replace cryptography with a minimal package, e.g., a small maturin extension that would satisfy our test purposes.

[Harsh-2005d]

I'd like to work on this.

Should the minimal package be referenced as a local path in the test input, or packaged and referenced via a URL like the current cryptography setup?

[slimreaper35]

Should the minimal package be referenced as a local path in the test input, or packaged and referenced via a URL like the current cryptography setup?

Actually, we were thinking about having our custom "hello world" project/repo. That would most likely be hosted in our hermetoproject organization -> the integration test would consume it as a URL dependency (same as cryptography).

I was struggling to find a minimal real-world project like that. After merging hermetoproject/integration-tests#106, it would have to cover a transitive Rust git dependency as well.

[Harsh-2005d]

I went ahead and put together a minimal fixture:

It's a small maturin/pyo3 package with rand pulled via git URL, which brings in rand_core as transitive git deps , which should cover the requirements.

Will this work?

[eskultety]

I went ahead and put together a minimal fixture:

@Harsh-2005d I'd just add one requirement from my side (disclaimer: I haven't checked your fixture yet) - we'll be setting a Hermeto precedent with a solution to this issue @slimreaper35 filed, so I would first expect some discussion (I'm wondering if we should use the GH discussions feature for that) to outline how exactly we want to extrapolate this to other backends going forward, how many repositories we expect to maintain, how much automation would be needed, etc. In other words, rather than seeing an immediate solution to this issue I as a maintainer would first like to see a consensus on a sustainable approach not exhausting our (human) resources, I'd say we're already quite tired of the current integrations tests mechanics (and it's still better than what we used to have before centralizing it!!!)

[Harsh-2005d]

Understandable, happy to help where i can.

[eskultety]

Understandable, happy to help where i can.

Feel free to open a discussion topic referencing this very issue and formulating your proposal if you already have one in mind. We'll provide our insights there :) . @Harsh-2005d this is a low priority (yet very desirable nice-to-have) item, so if you're interested you can own it and bring it to a successful end. We just might be a little slower with our review comments, that said, we care about a thriving community of developers deeply we'll do our best.