RFC: 4.x releases

[totallyzen]

Context

Now that we've got a release pipeline back in action, there is lots we can do to make things nicer.

I am raising this issue in order for community to gather and talk about what should be fixed as priorities and what improvements we want to add.

Known Issues

We know there are several areas we want to improve:

• We want to thin out dependencies where possible - prefer giving a ready container but not be biased on how you should connect to the container. Example: postgres containers enforcing SqlAclhemy, as seen in fix(postgres): Remove SqlAlchemy dependency from postgres container #445
• We want to improve package typing so people have better developer experiences, as seen in Mark packages as typed #305 and the PRs it spawned so far
• Adding new containers, as there is demand for it feat(nats): Add new module #439, fix(kafka): Add redpanda testcontainer module #441, feat(new): add a new Docker Registry test container #389, etc

Note

The list above is not conclusive, please comment on this issue and show your support!

[max-pfeiffer]

@totallyzen Hello Balint, I just created a couple of new issues:

• Bug: Arangodb tests fail on machines with ARM CPU (Apple M1, M2) #449
• Bug: Kafka tests throw errors and run forever on machines with ARM CPU (Apple M1, M2) #450
• Bug: Keycloak tests fail on machines with ARM CPU (Apple M1, M2) #451
• Bug: Elasticsearch tests fail on machine with ARM CPU (Apple M1, M2) #452
• Bug: Mssql tests fail on machines with ARM CPU (Apple M1, M2) #453

The tests for these modules fail on my M2 MacBook. And most likely will fail for everyone else using these testcontainer modules on their machine with ARM CPU.

We already have an older pull request addressing that issue, but that one seems to be outdated and partially obsolet as a couple of issues already became fixed: #357

I would offer to work these issues as it is important to me to have working tests. Also anyone else using these testcontainer modules on their ARM machines would profit from it.

[max-pfeiffer]

@totallyzen PS: I just checked for an M1/M2 GitHub runner. It is actually available since January 2024.

But it comes without a Docker installation and has other quirks. I just did a proof of concept with it: because of the mentioned issues I was not able to run the tests in the pipeline.

[bearrito]

Might be more appropriate to get that in a 5.x.

AFAIK ARM runners aren't generally available yet - https://resources.github.com/devops/accelerate-your-cicd-with-arm-and-gpu-runners-in-github-actions/ . Not sure you'd want to correlate ARM with MAC OS.

In general not too difficult to get that working with github actions using a matrix - https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs#using-a-matrix-strategy

[bearrito]

1. Revise documentation after the poetry workflow change. Most of the docs refer to the previous workflow.

2. Correct usage/mental model of extras and community modules. I get the desire for keeping dependencies low. I think the downside of not including a client dependency is while you can verify that the container is running, you can't verify that the connection details you might return are valid for the eventual client. Some want a full uri, others perhaps host and port. Or maybe I didn't expose the correct port etc.

I'm the committer on #439 There is AFAIK on one supported python client to interact with NATS, so it seems natural to include it. I could take or leave any approach.

[max-pfeiffer]

Might be more appropriate to get that in a 5.x.

AFAIK ARM runners aren't generally available yet - https://resources.github.com/devops/accelerate-your-cicd-with-arm-and-gpu-runners-in-github-actions/ . Not sure you'd want to correlate ARM with MAC OS.

In general not too difficult to get that working with github actions using a matrix - https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs#using-a-matrix-strategy

Yes, I agree here. I was not aware of the general ARM runner roadmap of GitHub.

And yes, it's quite straight forward to achieve this with a matrix. I already did a little POC: max-pfeiffer#3

[totallyzen]

@max-pfeiffer thanks for the summary on mac! It helps a great deal indeed to know!

For the matrices, I think you'll find it that I've used them extensively in this project already.
We can easily add a PoC PR that piles up all the fixes for MacOS, Windows and ARM and I'm glad there are people who care about this! :)

For me I'm running on a vanilla Ubuntu and not an ARM machine so as you might imagine, testing can be kinda difficult. I also have a young child so time is limited sometimes. If you find me responding somewhat slowly, this is why (managing energy and time).

---

@bearrito Good points overall, I started on the documentation for CONTRIBUTING.md and update the issue templates so we can communicate how things fit together. I'll add all the current active members to that PR as a reviewer so you can have your say! 🙂

As for "what about auto-including clients" I think I am gravitating towards "let's add them as dev dependencies to prove the clients are working, but don't enforce it in live dependencies". This way we can show the users how we intended to use it, without enforcing the specific library (even if there is only one right now).
Deal?

---

I'll check with @kiview on giving you both some level of permissions so you can auto-run some tests yourself on PRs. Not write permissions, just so that your PRs get auto-approval for the GH runs. Same goes for @jankatins as he is also actively contributing with other things!

I think with your dedication and activity you have earned it.

[bearrito]

@totallyzen Deal. I implemented what I think should be satisfactory in a new pr at https://github.com/testcontainers/testcontainers-python/pull/462/files. That could be an overall good approach to managing clients.

[alexanderankin]

found another good fix to get merged #457 for 4.0.x

[alexanderankin]

alright im unsubscribing from notifications from this repo for half a week - its too much - so many people are so busy pushing commits and updating the project with feedback. good job everyone 👍