Spark's Work

Using Apptainer (Singularity) for Reproducible Environments

An integral part of scientific computing is reproducible environments, without which experiments cannot be replicated and all published numerical results are reduced to pure coincidences.

In a previous post (see here), I have noted down ways to create and save reproducible environments in terms of packages given a language of choice. Another component of reproducibility is the programming language itself: different versions of the same language may also behave differently (see e.g. Python 2 vs Python 3).

Containers are designed to address this issue. As mentioned here by Docker, a popular container software,

A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another. A Docker container image is a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings.

While Docker is the industry standard, it is not available on my school's server due to security concerns. Instead, Apptainer (formerly known as Singularity) seems to be the go-to choice in the research community.

I have had great experiences with Apptainer so far, since it integrates with Docker images seamlessly and allows me to run whatever is convenient on my school's server (where I am not the admin and usually cannot install random stuff).

The Apptainer official website already contains good documentations, and I also found this page quite useful. To get started with a minimal example, suppose I would like to run something in Julia 1.6.0. First, pull the official docker image (similar to what's done in Docker),

apptainer pull docker://julia:1.6.0

This command will also convert the docker image to a simg file (in this case, julia-1.6.0.simg) which is actually used by Apptainer. Then, run the following to enter an interactive shell in the container.

apptainer shell julia-1.6.0.simg

After this step, everything else follows very easily, since Apptainer automatically mounts existing folders so I typically don't need a separate setup process for data access. I can then install and run stuff in this container without worrying about admin privileges. I can also save the current image and share with my collaborators, so they will be using exactly the same environment as I do to reproduce the same results. An official guide to building images is given here, which I save for future references.