• First, we learnt about Dask Clusters, which allow you to scale your computations across multiple cores or machines
• We saw how to parallelise the training of machine learning models
• How to use automated machine learning (AutoML) tools to speed up the process
• We also discussed how to implement different methods to search for the best hyperparameters using TPOT, scikit-learn, and Dask

• Today we will talk about a different topic: how to make sure your results are reproducible?
• We will discuss the importance of dependency management, virtual environments, and containers
• Replication has been a recurring theme in this course, and today we will learn how to make it easier
• That is the main reason why we use the command line, git, Quarto, Jupyter, and other tools
• So today we will discuss some of the best practices to ensure computational reproducibility
• We will also discuss how to use containers to make your code portable, reproducible, and scalable
• Let’s get started! 🚀

• You now have a project!
• Your code works great, it runs pretty fast thanks to Dask, your Quarto reports are beautiful, and your analyses (all done in the command line) are stored in a well-documented GitHub repository
• Are you done? 🤔
• Not quite! 😅
• What if you need to run your code on a different machine? Or share it with a colleague? Or run it again in a few months?
• You need to make sure your code will run in the future, and that’s where dependency management comes in

The problem

• As we have seen in this course (and in many others), libraries and packages change constantly
• New versions are released, old versions are deprecated, and not all operating systems have the required libraries installed to run your code
• Even extremely simple code can break from one version of a library to the next
• This code written in Python 2.7:

print "Hello, world!"

• will not work in Python 3.x

print "Hello world!"

  File "/var/folders/96/r1yycxlj28958p1cdynhbyzw0000gn/T/Rtmpa0OGSM/chunk-code-b08d2b78904b.txt", line 1
    print "Hello world!"
                       ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Hello world!")?

Some definitions

• Dependency management is the process of ensuring that your code will run in the future
• Dependencies are the external components necessary to run your code, such as libraries, packages, and software
• Packages have a name, a type, a version, relevant files for the package’s functionality, and potentially dependencies on other packages
• A package registry is a directory for packages and stores metadata about packages in the registry, such as CRAN, PyPI, or Conda
• Dependency management tools help you manage your dependencies, such as pip and conda

• We have already seen how important it is to ensure that your results are reproducible
• The reproducibility crisis is a term used to describe the inability of researchers to replicate the results of a study
• This affects computer science, statistics, and many other fields
• Apart from statistical issues such as p-hacking, publication bias, and low statistical power, one of the main reasons for the reproducibility crisis is the lack of proper documentation and dependency management
• While the statistical problems are a bit more complex, the latter can be easily solved with tools that we already have at our disposal

• According to a recent Nature survey, more than 70% of scholars have tried and failed to reproduce another scientist’s research, and more than half have failed to reproduce their own research 😳
• 90% of researchers believe that there is a reproducibility crisis in science

• Imagine that you have successfully done some analysis with Python, NumPy, pandas, and matplotlib installed, and you have to run a Python script
• If you are the only one working on the project, you can probably just run it in your local environment. But ask yourself:
  • What packages/libraries do I need to load?
  • What OS am I using? (Will this work on my collaborator’s system?)
  • What are the steps to reproduce my results?
• However, it is better to use a package manager to declare your dependencies
• It’s a single file describing the necessary dependencies, which can be used to install all dependencies in one step
• Store the file in the repository root (main folder)
• The file depends on the environment/package manager you want to use:
  • For conda (python and R): generate an environment.yml file
  • For pip (python only): generate a requirements.txt file

• environment.yml (for conda)
  • Used by conda to create an environment populated by specific packages and languages
  • Generate it with conda env export -f environment.yml
  • -f is a flag for “file”
  • If you would like to know more about conda environments, see this quick intro
  • Or get the full story in the conda documentation
• requirements.txt (for pip)
  • Generate it with pip freeze | grep -v 'file://' > requirements.txt
  • Install dependencies declared with pip install -r requirements.txt
  • -r is a flag for “requirements”
  • Let’s see some examples

• In conda, a build refers to a specific compiled binary version of a package
• Even when software has the same version number, it can be compiled differently for different platforms, Python versions, or with different features enabled
• A full conda package specification has four components:

numpy=1.21.5=py39h12345_0
└──┬─┘└─┬──┘ └────┬───┘└┬┘
   │    │         │     │
   │    │         │     └─ Build number (0)
   │    │         └─────── Build string (py39h12345)
   │    └───────────────── Version (1.21.5)
   └────────────────────── Package name (numpy)

• The --no-builds flag excludes build numbers from package specifications, making environment files more portable across different platforms and operating systems

• The build string (py39h12345) encodes important metadata:
  • Python version: py39, py310
  • Architecture: linux_64, osx_arm64
  • Compiler: gcc9 (GNU/Linux), clang (macOS)
  • Features: nomkl (Intel), cuda (NVIDIA GPUs)
• Why do builds exist, then?
  • Platform-specific compilation: Software must be compiled differently for different operating systems
  • Python version compatibility: Packages are compiled against specific Python versions
  • Optional features & dependencies: Different builds can enable/disable features
  • Compiler variations: Different compilers produce different binaries

• So far, so good! But what if you want to share your environment with someone else?
• You can do that by simply sharing the environment.yml file with them (or by uploading it to your repository!)
• They can then create the same environment on their machine by running the following command:

conda env create --file environment.yml

Channels:
 - defaults
Platform: osx-arm64
Collecting package metadata (repodata.json): done
Solving environment: done

Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate qtm350
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) 

• This will create a new environment called qtm350 with the same packages and versions as the original environment
• They can then activate the environment and run the code (if they have the necessary files) by running:

conda activate qtm350
python scripts/hello.py

• To delete the environment, they can run:

conda deactivate
conda env remove --name qtm350

• And you’re done! 🎉

• Containers are a way to package software in a format that can run on any system
• They are similar to virtual machines (VMs), but they are more lightweight and portable
• A virtual machine runs a full operating system, while a container runs only the necessary libraries and dependencies to run the application
• They are also one step up from virtual environments, as they package the entire computer environment, not just the dependencies and code
• That means that you can run your code on any system that has Docker installed, regardless of the operating system or the hardware
• Talk about reproducibility! 😊

• Containers are created using containerisation tools such as Docker or Singularity
• They are usually just a stripped-down version of Linux with the necessary libraries and dependencies
• Why Linux? You already know the answer by now! Because it is open-source, and it is the most widely used operating system in the world (really!)
• The Linux distribution of choice is usually our familiar Ubuntu or Alpine Linux, which is even more lightweight
• Containers are usually stored in a container registry such as Docker Hub, so you can share them with others too

• Docker is the leading containerisation platform, especially in industry
• The main entities in Docker are containers, images, and registries
• A container, as we have just seen, is an executable package of software that includes everything needed to run an application
• An image is a snapshot of a container. Images are created with the build command, and they will produce a container when started with run. More about the distinction here
• A registry is a collection of repositories from which you can pull images
• Docker has very similar syntax to Git and Linux, so Docker should feel natural (though you should still read the docs!)

• To install Docker, you can follow the instructions here
• Docker Desktop is the easiest way to get started with Docker on Windows and macOS, and it includes everything you need to run Docker on your machine
• It is a good idea to create an account too, as you can use it to store your images in Docker Hub
• After you have installed and started Docker, you can run the following commands to check if it is working:

docker info
docker run hello-world

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
478afc919002: Download complete 
Digest: sha256:305243c734571da2d100c8c8b3c3167a098cab6049c9a5b066b6021a60fcb966
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

docker -v

Docker version 28.5.2, build ecc6942

• Docker can seem a little daunting at first
• We need to create and configure a Dockerfile to build an image, which is a plain-text recipe that will build from scratch everything you need to recreate a project
• It is just a text file that you can put under version control. No extension is needed, it’s usually just called Dockerfile
• However, in most cases you do not need to start from scratch
• Docker Hub has thousands of images that you can use as a base for your project, and it is very likely that someone has already created an image similar to the one you need
• For example, as of today there are ~8,500 images of data science tools in Docker Hub
• But we will build one today, just to see how it works 😊
• Our next class will be a hands-on session on Docker, so we will have plenty of time to practice how to build containers from scratch or from existing images 😉

• A Dockerfile is a text file that contains all the commands a user could call on the command line to assemble an image
• Using docker build, we can create an automated build that executes several command-line instructions in succession
• Let’s first create a directory called docker and a file called Dockerfile inside it

mkdir docker
cd docker
touch Dockerfile

• We will use an official Python image as a base, and install some packages on top of it. Here: https://hub.docker.com/_/python
• Then we will write code that installs some Python packages and runs a Python script

• The syntax is quite straightforward:
  • FROM <image_name>:<tag> is the base image, RUN is used to run bash commands while building the image, and CMD is used to specify which commands to run when the container starts
• The full list of instructions is available here

# Use an official Python runtime as a parent image
FROM python:3

# Install libraries
RUN pip install numpy pandas matplotlib

# Add a script
RUN echo "print('Hello, QTM350!')" > hello.py

# Run the script
CMD ["python", "hello.py"]

• We build the image with the following command (-t is for “tag”, which names the image qtm350-example, and . is the current directory):

docker build -t qtm350-example .

• Dependency management is more important than people think
• It is the first step towards reproducibility and transparency
• You can use conda, pip, or pipenv to manage your dependencies in Python
• Docker offers a more comprehensive solution to the reproducibility crisis
• None of them are particularly difficult to use, and they can save you a lot of time and headaches in the future
• In the next class, we will have a hands-on session on Docker, so you can practice building containers and running them on your machine