• Set up local Dask Clusters with workers, schedulers, and dashboards
• Workers compute tasks and store results; the scheduler coordinates them
• Used Dask ML for distributed machine learning and model training
• Parallelised hyperparameter tuning with IncrementalSearchCV, HyperbandSearchCV, and TPOT
• Handled async/sync compatibility issues and cached computations
• Measured speedup and efficiency gains from parallel execution

• New topic today: how do we make sure our results are reproducible?
• Replication has been a recurring theme in this course, and that is why we use the command line, git, Quarto, and Jupyter
• Today we cover dependency management, virtual environments, and containers
• We will also learn how to use Docker to make code portable and reproducible
• Let’s get started! 🚀

• You now have a project!
• Your code works great, it runs pretty fast thanks to Dask, your Quarto reports are beautiful, and your analyses (all done in the command line) are stored in a well-documented GitHub repository 😁
• Are you done? 🤔
• Not quite! 😅
• What if you need to run your code on a different machine? Or share it with a colleague? Or run it again in a few months?
• You need to make sure your code will run in the future, and that’s where dependency management comes in

The problem

• Libraries and packages change constantly: new versions appear, old ones are deprecated, and different OSes ship different libraries
• Even simple code can break between versions. This Python 2.7 code:

print "Hello, world!"

• does not work in Python 3.x

print "Hello world!"

  File "/var/folders/96/r1yycxlj28958p1cdynhbyzw0000gn/T/Rtmpa0OGSM/chunk-code-b08d2b78904b.txt", line 1
    print "Hello world!"
                       ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Hello world!")?

Some definitions

• Dependency management: making sure your code will run in the future
• Dependencies: external libraries, packages, and software your code needs
• Packages: have a name, version, and possibly their own dependencies
• Package registry: a directory that stores packages and metadata, e.g., CRAN, PyPI, Conda
• Dependency management tools: pip, conda, and others help you track and install dependencies

• Reproducibility crisis: many researchers cannot replicate published results
• Affects CS, statistics, psychology, medicine, and other fields
• Some causes are statistical (p-hacking, publication bias, low power)
• But poor documentation and dependency management are just as common
• The good news: the computational side can be solved with tools we already have
• A Nature survey found that 70%+ of scholars failed to reproduce others’ work
• Over half could not reproduce their own results 😳

• A conda build is a specific compiled binary of a package
• The same version number can be compiled differently for different platforms, Python versions, or feature sets
• A full conda package specification:

numpy=1.21.5=py39h12345_0
└──┬─┘└─┬──┘ └────┬───┘└┬┘
   │    │         │     │
   │    │         │     └─ Build number (0)
   │    │         └─────── Build string (py39h12345)
   │    └───────────────── Version (1.21.5)
   └────────────────────── Package name (numpy)

• The --no-builds flag strips build info, making environment files portable across platforms

• The build string (py39h12345) encodes metadata:
  • Python version (py39, py310)
  • Architecture (linux_64, osx_arm64)
  • Compiler (gcc9, clang)
  • Features (nomkl, cuda)
• Why do builds exist?
  • Software is compiled differently per OS
  • Packages target specific Python versions
  • Builds can enable or disable optional features
  • Different compilers produce different binaries

• To share your environment, upload the environment.yml file to your repository
• Others can recreate the same environment by running:

conda env create --file environment.yml

Channels:
 - defaults
Platform: osx-arm64
Collecting package metadata (repodata.json): done
Solving environment: done

Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate datasci350
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) 

• This creates a datasci350 environment with the same packages and versions
• Activate the environment and run your code:

conda activate datasci350
python scripts/hello.py

• To delete the environment:

conda deactivate
conda env remove --name datasci350

• And you’re done! 🎉

• uv is a modern, very fast Python package and project manager built by Astral (the creators of Ruff)
• Written in Rust, it replaces pip, pip-tools, pipenv, poetry, virtualenv, and pyenv in a single tool!
• 10-100x faster than pip for installing packages 🏎️
• Installs Python itself, so you don’t need a separate Python installer
• Works on macOS, Linux, and Windows

# Install uv (macOS/Linux)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Or with Homebrew
brew install uv

# Check installation
uv --version

• Create a new project with uv init:

# Create a new project
uv init datasci350-project
cd datasci350-project

# Add dependencies
uv add numpy pandas matplotlib

# Run a script (uv handles the environment)
uv run python script.py

# Add development dependencies
uv add --dev pytest ruff

• uv init creates a pyproject.toml file with your project metadata and dependencies
• uv add installs packages and updates the lock file (uv.lock) automatically
• --dev flag adds packages to a separate development group (e.g., for testing and linting)
• uv run executes commands inside the project environment without needing to activate it

• Containers package software so it can run on any system
• Similar to virtual machines (VMs), but lighter: a VM runs a full OS, while a container includes only the libraries needed for the application
• One step up from virtual environments: containers package the entire computer environment, not just dependencies and code
• Your code runs on any system with Docker installed, regardless of OS or hardware

• Created with tools like Docker or Singularity
• Usually a stripped-down Linux (e.g., Ubuntu or Alpine Linux) with only the required libraries
• Why Linux? It is open-source and the most widely used server OS
• Stored in container registries like Docker Hub, so you can share them with others

• Docker is the leading containerisation platform
• Three main concepts:
  • Image: a read-only template (like a recipe) that contains the OS, libraries, code, and configuration. You build images from a Dockerfile
  • Container: a running instance of an image. You can run many containers from the same image
  • Registry: a collection of repositories from which you pull images (e.g., Docker Hub)
• Think of it this way: an image is a class, a container is an object (more here)
• The syntax is similar to Git and Linux, so it should feel familiar

• Install Docker following these instructions
• Docker Desktop is the easiest way to start on Windows and macOS
• Create an account to store images on Docker Hub
• Check your installation:

docker info
docker run hello-world

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
478afc919002: Download complete 
Digest: sha256:305243c734571da2d100c8c8b3c3167a098cab6049c9a5b066b6021a60fcb966
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

docker -v

Docker version 28.5.2, build ecc6942

• A Dockerfile is a plain-text recipe that builds everything you need to recreate a project
• It is just a text file under version control, usually called Dockerfile (no extension)
• You rarely need to start from scratch: Docker Hub has thousands of base images you can build on
• For example, there are ~9,000 images of data science tools on Docker Hub
• We will build one today just to see how it works 😊
• Next class will be a hands-on Docker session 😉

• A Dockerfile lists all the commands needed to assemble an image
• docker build runs these instructions in sequence, layer by layer
• Each instruction (e.g., RUN, COPY) adds a new layer; Docker caches layers so unchanged steps are not re-run on rebuild
• Let’s create a directory and a Dockerfile:

mkdir docker
cd docker
touch Dockerfile

• We will use an official Python image as a base and install packages on top of it

• FROM sets the base image, RUN executes commands during the build, and CMD runs when the container starts. Full reference here

# Use an official Python runtime as 
# a parent image
FROM python:3

# Install libraries
RUN pip install numpy pandas matplotlib

# Add a script
RUN echo "print('Hello, DATASCI350!')" > hello.py

# Run the script
CMD ["python", "hello.py"]

• We build the image with the following command (-t is for “tag”, which names the image datasci350-example, and . is the current directory):

docker build -t datasci350-example .

• Dependency management is more important than people think
• It is the first step towards reproducibility and transparency
• You can use conda, pip, or uv to manage your dependencies in Python
• Docker offers a more comprehensive solution to the reproducibility crisis
• None of them are difficult to use, and they can save you a lot of time and headaches in the future 😅
• In the next class, we will have a hands-on session on Docker, so you can practice building containers and running them on your machine