Convert lesson to use Podman instead of Docker

[welucas2]

I've been talking with @aturner-epcc about the need to convert this lesson to use Podman instead of Docker (#255). I've made an initial and at the moment incomplete attempt to do so here. For the time being I'll leave this in a draft state.

At the moment I think I've converted most the lesson episodes themselves with the following still outstanding (mostly things in the learners directory):

• episodes/docker-image-examples.md, the episode with examples on using containers in practice
• and the other episodes in the learners directory which follow on from the above
• the latter part of learners/setup.md which covers installation of Docker/Podman Desktop
• learners/reference.md
• instructors directory

I had to make some choices around where the line between Podman and Docker lies. For example, I've kept Dockerfile throughout as I think that's still the favoured name in most places, even with Podman, but note that Containerfile is an option. It still uses Docker Hub, but tries to cover the other free registries a little more than before. That was made a little more complex than it needed to be by Podman defaulting to Docker Hub for most push/pull operations but overriding to Quay.io for its hello image.

Apologies if there's anything slightly out of place; I initially did this for the 'combined-with-Singularity' version of the course that has been run a few times as part of ARCHER2 training, and have only now brought those changes over to the main repository.

[github-actions]

🆗 Pre-flight checks passed 😃

This pull request has been checked and contains no modified workflow files or spoofing.

It should be safe to Approve and Run the workflows that need maintainer approval.

[aturner-epcc]

Thanks @welucas2, this is a large amount of work which should benefit the lesson going forwards.

[sstevens2]

So excited to see this PR! I can go ahead and start to review it later this week if that is okay? I see that you've marked it as draft @welucas2 so let me know if I should wait.

It has been a while since @jcohen02 and I talked about this change so I can't quite remember what we decided to do going forward. This seems like a good start to me though. If I recall we were concerned about removing Docker from the name completely since folks more commonly know that name. Maybe we can change the lesson title to "Reproducible Computational Environments Using Containers: Introduction to Podman/Docker". We could also include a callout about why this lesson uses podman early on. Also perhaps a warning that the commands currently are interchange able by swappping out podman with docker if someone chooses to use Docker instead, that they may not always be.

[welucas2]

I left it in draft state as I still think there's a decent amount to do -- but if you'd like to check it out (and potentially merge in?) with those items I mentioned above still pending, I'd be happy to move it out of draft state! I don't think I'll have the time to come back to this for a little while, so just let me know what you think is best and I'll be happy to go that route.

[colinsauze]

I taught this course yesterday using Podman and the notes from this PR. I didn't find any problems with the lesson material itself but I did modify the setup instructions to direct people to install Podman Desktop. I'm not sure if I can make a PR to a PR to contribute these (do I open another PR to the downstream repo?), but the instructions are here https://github.com/NOC-OI/containers-introduction/blob/main/learners/setup.md if anybody would like to incorporate them.

[sstevens2]

@welucas2 I'll give it a full review tomorrow. Though from @colinsauze's comment it seems like it should be a good start.

Thanks @colinsauze! You can create a PR to @welucas2's fork and the branch in this PR and then it will be added to this PR when it is merged. I'm also happy to add the setup directions you linked tomorrow

[welucas2]

Thanks everyone, sounds good :) I'll remove draft status now and we can move ahead with things.

[sstevens2]

Thanks so much for these changes, @welucas2! I really appreciate your efforts to convert the lesson to podman.

I had a couple of small suggestions and then there are two action items I've noted we need to do to get this merged:

• The setup section needs to be updated (edit: completed in commit after this review concluded)
• The @carpentries-incubator/docker-introduction-maintainers need to review the language in the index.md I drafted as a suggestion around our decision to switch to from docker to podman.

[sstevens2]

Suggested change

	What happens if you use the `podman container run` command above to sun `sum.py`
	What happens if you use the `podman container run` command above to run `sum.py`

[sstevens2]

Suggested change

	Note, you can always re-tag an image later if the name needs to be updated.

[sstevens2]

Suggested change

	<!-- ::::::::::::::::::::::::::::::::::::::::: callout
	## Docker Command Line Interface (CLI) syntax
	In this lesson we use the newest Docker CLI syntax
	[introduced with the Docker Engine version 1.13](https://www.docker.com/blog/whats-new-in-docker-1-13/).
	This new syntax combines commands into groups you will most often
	want to interact with. In the help example above you can see `image` and `container`
	management commands, which can be used to interact with your images and
	containers respectively. With this new syntax you issue commands using the following
	pattern `docker [command] [subcommand] [additional options]`
	Comparing the output of two help commands above, you can
	see that the same thing can be achieved in multiple ways. For example to start a
	Docker container using the old syntax you would use `docker run`. To achieve the
	same with the new syntax, you use `docker container run` instead. Even though the old
	approach is shorter and still officially supported, the new syntax is more descriptive, less
	error-prone and is therefore recommended.
	:::::::::::::::::::::::::::::::::::::::::::::::::: -->
	::::::::::::::::::::::::::::::::::::::::: callout
	## Podman Command Line Interface (CLI) syntax
	In this lesson we use the CLI syntax
	[introduced with the Docker Engine version 1.13](https://www.docker.com/blog/whats-new-in-docker-1-13/).
	This new syntax combines commands into groups you will most often
	want to interact with. In the help example above you can see `image` and `container`
	management commands, which can be used to interact with your images and
	containers respectively. With this new syntax you issue commands using the following
	pattern `podman [command] [subcommand] [additional options]`
	Comparing the output of two help commands above, you can
	see that the same thing can be achieved in multiple ways. For example to start a
	container using the old syntax you would use `podman run`. To achieve the
	same with the new syntax, you use `podman container run` instead. Even though the old
	approach is shorter and still officially supported, the new syntax is more descriptive, less
	error-prone and is therefore recommended.
	::::::::::::::::::::::::::::::::::::::::::::::::::

[sstevens2]

Suggested change

	<!-- LocalWords: keypoints links.md endcomment systray
	-->

[sstevens2]

Suggested change

	On HPC systems it is more likely that *Singularity* rather than Podman will be the available container technology,
	though Podman is becoming more common alongside Singularity.
	If you are looking for a lesson on using Singularity containers (instead of Podman), see this lesson:
	On HPC systems it is more likely that *Singularity/Apptainer* rather than Podman will be the available container technology,
	though Podman is becoming more common alongside Singularity/Apptainer.
	If you are looking for a lesson on using Singularity/Apptainer containers (instead of Podman), see this lesson:

[sstevens2]

Suggested change

	title: 'Reproducible Computational Environments Using Containers: Introduction to Podman'
	title: 'Reproducible Computational Environments Using Containers: Introduction to Podman/Docker'

[jcohen02]

I agree with this, I think if we remove Docker from the title, there will be people who have been told that they should learn how to use Docker but have never heard of Podman and overlook this - ultimately, the content is largely agnostic to whether you use Podman or Docker from an end-user perspective (at least I believe that's correct?). In fact, I just noticed that @sstevens2 mentioned this in one of the comments. 🙂

[dme26]

I wonder whether softening the slash into "[...] Introduction to Podman and Docker" might be OK? I realise that in practice it's probably more likely to be an "or" than an "and". It would be good to also mention our general objective to include registries and the open ecosystem, but "[...] Introduction to Podman, Docker, and container registries" is starting to get long as a title...

[sstevens2]

Suggested change

[sstevens2]

Suggested change

	::::::::::::::::::::::::::::::::::::::::: callout
	:::::::::::: callout
	## Why Podman instead of Docker
	This lesson was originally written to use the tool Docker and the associated Docker Desktop.
	All the commands within this lesson can be run using `docker` instead of `podman`, though the output may vary slightly.
	The maintainers of this lesson typically advocate for researchers using open source software.
	In the past Docker, had a free tier for research/academic software though eventually removed that option leaving only exceptions for education and open source software.
	In 2024, Docker changed their terms of service and removed their exception for open source software.
	This change prompted the lesson maintainers to switch the lesson to `podman`, an open source alternative.
	::::::::::::::::::::::::
	::::::::::::::::::::::::::::::::::::::::: callout

If we decide to merge this suggestion, it should probably be reviewed by at least a couple of the other maintainers.

[sstevens2]

@carpentries-incubator/docker-introduction-maintainers

[jcohen02]

I'm suggesting a minor edit to the start of the text above:

This lesson was originally written to use the tool Docker and the associated Docker Desktop.
All the commands within this lesson can be run using `docker` instead of `podman`, though the output may vary slightly.

However, I'm not entirely sure about some of the statements above, they may well be correct but I don't have enough knowledge to be sure off-hand.

Maybe someone else can comment on this. For example, https://www.docker.com/community/open-source/ still claims that Docker provides a "free, secure and scalable platform to distribute and collaborate on containerized applications.".

[sstevens2]

Edited the suggestion to include @jcohen02's rewording

[sstevens2]

I'm a bit confused myself from looking at their websites. But in Fall (Northern Hemisphere) 2024 they removed their language about open source projects and educational uses being able to use the free personal tier of their service (including docker desktop).

I think docker engine is free to use outside of docker desktop.

Commercial use of Docker Engine obtained via Docker Desktop within larger enterprises (exceeding 250 employees OR with annual revenue surpassing $10 million USD), requires a paid subscription. Apache License, Version 2.0. See LICENSE for the full license.

from docker engine docs

[sstevens2]

I went ahead and replaced the setup directions with those referenced by @colinsauze.

[jcohen02]

Thanks for all the work here @welucas2. I'm marking this as request changes since there are some minor things to update but this is great and a really substantial piece of work that I can see must have taken lots of time.

[jcohen02]

I agree with this, I think if we remove Docker from the title, there will be people who have been told that they should learn how to use Docker but have never heard of Podman and overlook this - ultimately, the content is largely agnostic to whether you use Podman or Docker from an end-user perspective (at least I believe that's correct?). In fact, I just noticed that @sstevens2 mentioned this in one of the comments. 🙂

[jcohen02]

Suggesting the addition of a note here about the registry configuration file and the importance of using fully-qualified image names for real-world tasks to help avoid the potential for name spoofing.

Suggested change

	_While going into this in further detail is beyond the scope of this material, it's useful to note that Podman contains a registry configuration file that tells it which container registry/registries to search, and in what order, when an unqualified image name is provided. You should also be aware that using unqualified image names (i.e. providing an image name without providing the fully-qualified path to the image), can present security issues. We therefore strongly recommend always using the fully-qualified image name for any real-world work you are undertaking with containers._

[jcohen02]

I think this should refer to an "image registry" rather than a container registry (or a "container image registry")?

Suggested change

	- A container registry is an online repository of container images.
	- "Docker Hub is the most widely used container registry."
	- An image registry is an online repository of container images.
	- "Docker Hub is the most widely used image registry."

[jcohen02]

I think the docker.io prefix is just specific to Docker Hub? Docker images can of course also be hosted through other image registries.

Suggested change

	- The naming convention for Docker container images when using Podman is: `docker.io/OWNER/CONTAINER_IMAGE_NAME:TAG`
	- The naming convention for referring to container images stored in Docker Hub, when using Podman, is: `docker.io/OWNER/CONTAINER_IMAGE_NAME:TAG`

[jcohen02]

Suggested change

	The image you delete may have been formed by merging multiple underlying layers.
	The image you deleted may have been formed by merging multiple underlying layers.

[jcohen02]

Suggested change

	If you have installed the Podman Desktop GUI, you should be able to use its
	GUI to view and delete containers and container images. It will provide the same
	If you have installed the Podman Desktop GUI, you should be able to use it
	to view and delete containers and container images. It will provide the same

[jcohen02]

I'm suggesting a minor edit to the start of the text above:

This lesson was originally written to use the tool Docker and the associated Docker Desktop.
All the commands within this lesson can be run using `docker` instead of `podman`, though the output may vary slightly.

However, I'm not entirely sure about some of the statements above, they may well be correct but I don't have enough knowledge to be sure off-hand.

Maybe someone else can comment on this. For example, https://www.docker.com/community/open-source/ still claims that Docker provides a "free, secure and scalable platform to distribute and collaborate on containerized applications.".

[jcohen02]

Suggested change

	On HPC systems it is more likely that *Singularity* rather than Podman will be the available container technology,
	On HPC systems it is more likely that *Singularity* rather than Podman or Docker will be the available container technology,

[jcohen02]

I see another suggestion has been made below - maybe the "or Docker" suggested above can be integrated in the change suggested below...

[jcohen02]

A big thank you from me too, @welucas2, for all the work on this and apologies for the delay in getting to look at this. I've provided my main comments in my review that I just completed but noting one other point here:

• Re @sstevens2 comment above:

  We could also include a callout about why this lesson uses podman early on. Also perhaps a warning that the commands currently are interchange able by swappping out podman with docker if someone chooses to use Docker instead, that they may not always be.

I agree with this and can see that there's a suggestion above for some content here. I think it would also be good to make some mention of Docker in the introductory text. I wonder if something along the lines of "This session aims to introduce the use of "containers" with the goal of using them to effect reproducible computational environments. Such environments are useful for ensuring reproducible research outputs, for example. In this lesson we work with the free, open source Podman container tool. Podman is similar to, and largely compatible with, the very widely used commercial container framework, Docker. The material covered here is therefore also applicable to Docker and many commands covered here can be run in Docker simply by replacing podman with docker." might be useful.