https://www.theregister.com/2025/11/18/linus_torvalds_vibe_coding/

• Today we will have a hands-on session where we will build a container with all the tools we have covered in this course (!) 🤓
• That includes a bash shell, a git client, a Python interpreter, a Jupyter notebook server, a Quarto document processor, a SQL database, and Jupyter tools
• We will build the container using a Dockerfile, and then run it locally
• The container will be based on the official Ubuntu image (the same we used when creating an AWS instance), and we will install all the necessary packages in it
• We will explore how to mount volumes in the container to persist data
• We will also discuss how to manage dependencies effectively within the container

• As we saw in the last lecture, there are many Docker images for data science, such as Jupyter, RStudio, and SQLite
• There are many ways to build (multiple) containers. For example:
  • Docker Compose, which allows you to define multi-container applications in a single file
  • VS Code’s Dev Containers extension, which integrates Docker with Visual Studio Code
  • Kubernetes, which is used for automating deployment, scaling, and management of containerised applications
• But to keep things simple, we will use a single container today and run it with the docker run command

• Please download Docker Desktop for your operating system here: https://www.docker.com/
• There are versions for Windows, macOS, and Linux
• You can install it like any other application, just follow the instructions of the installer
• Docker now has a native Windows application, so it should work without any issues
• More information here: https://docs.docker.com/desktop/setup/install/windows-install/

• As we saw in the last lecture, a Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image
• The Dockerfile is a recipe for building a container
• The base image is the starting point for the container, and it can be any image available on Docker Hub
• So please create a text file called Dockerfile (without any extension) in your directory
• Any editor will do to create the file! 😉
• You will need about 2 GB of free space to build the container, so make sure you have enough space on your computer
• We will start our Dockerfile with FROM, followed by the base image we want to use

# Use the official Ubuntu image as the base image
FROM ubuntu:24.04

• The : character is used to specify the version of the image. In this case, we are using Ubuntu 24.04
• You can see all tags available for the Ubuntu image at https://hub.docker.com/_/ubuntu?tab=tags
• Okay, that’s the easy part. Now let’s install all the tools we need in the container! 🛠️

• The LABEL instruction adds metadata to an image
• This is useful for documentation purposes and to provide information about the image
• You can add any key-value pairs you want, but it is a good practice to include the version, description, maintainer, and license
• You can also add a MAINTAINER instruction to specify the maintainer of the image
• For instance, you can add the following lines to your Dockerfile

# Metadata
LABEL version="1.0"
LABEL description="Container with the tools covered in QTM 350"
LABEL maintainer="Danilo Freire <danilo.freire@emory.edu>"
LABEL license="MIT"

• This will add the metadata to the image, and you can see it when you run the docker inspect command

• The RUN instruction executes any commands in a new layer on top of the current image and commits the results
• Yes, commits in Docker are like commits in Git, but instead of saving changes to a file system, they save changes to an image
• Remember apt? We will use it again to install software packages (as we did with AWS)
• The only commands we will need to use are apt-get update (only once) and apt-get install <package>
  • For example, to install git, we would run apt-get update && apt-get install git -y
• I will also use the --no-install-recommends flag to avoid installing unnecessary packages
• The -y flag is used to automatically answer yes to any prompts
• To keep the image smaller, we will also run apt-get clean and rm -rf /var/lib/apt/lists/* after installing the packages
  • This will remove the package cache and clean up the image
• Let’s add some packages to the Dockerfile 🐳

# Update and install dependencies
# Versions: https://packages.ubuntu.com/
RUN apt-get update && apt-get install -y --no-install-recommends\
    bash=5.2.21-2ubuntu4 \
    git=1:2.43.0-1ubuntu7.3 \
    sqlite3=3.45.1-1ubuntu2 \
    libsqlite3-0=3.45.1-1ubuntu2 \
    python3.12=3.12.3-1ubuntu0.8 \
    python3.12-venv=3.12.3-1ubuntu0.8 \
    python3-pip=24.0+dfsg-1ubuntu1.3 && \ 
    apt-get clean && rm -rf /var/lib/apt/lists/*

• So far, so good! 😃
• You can see that we are installing specific versions of the packages, which is a good practice to ensure reproducibility
• On Ubuntu, you can find the available versions of a package at https://packages.ubuntu.com/
• Make sure to match the version of the operating system with the version of the package (24.04 LTS or noble in this case)

• By default, Docker uses sh as the default shell, but we want to use bash instead
• We can set the default shell to bash by adding the following line to the Dockerfile

# Set the default shell to bash
SHELL ["/bin/bash", "-c"]

• The SHELL instruction sets bash as the default shell for subsequent RUN commands
• The -c flag is required because it tells bash to read commands from a string (text), not from a file or standard input
• This way, we can use bash commands in the Dockerfile without having to specify the shell every time

• Yes, it is! 😅
• I am purposely making it harder than it should be!
• But this is the most flexible way to build a container
• We are starting from a really bare-bones image (Ubuntu) and installing everything from scratch
• And I (we?) don’t have Ubuntu installed on my computer, so I can’t just run apt list --installed to see which packages are installed on my system and just copy them to the Dockerfile
• In this case, it did involve some research to find the right package names and versions
• But once you have the Dockerfile, you can reuse it as many times as you want

• We will use the RUN instructions to install a few Python libraries with pip3, such as numpy, pandas, jupyterlab, dask, and matplotlib
• You can either see the versions you already have on your computer with pip freeze > requirements.txt and then copy the versions from the file
• Ubuntu requires us to create a virtual environment to install Python packages, so we will do that too
• I will use the -m flag to run the venv module as a script and create a virtual environment in the /opt/venv directory
• ENV PATH="/opt/venv/bin:$PATH" prepends the directory /opt/venv/bin to the beginning of the existing PATH environment variable within the Docker image
• This allows us to use the Python interpreter and packages installed in the virtual environment without needing to specify the full path every time

# Create and activate virtual environment
RUN python3 -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

# Install Python dependencies in virtual environment
RUN pip install httpx==0.27.2 numpy==1.26.4 pandas==2.2.2 \
                jupyterlab==4.2.5 ipykernel==6.29.5 \
                dask==2024.5.0 matplotlib==3.9.2

• We will also install Quarto, but this time we will use wget to download the binary
• Why? Because Quarto is not available on pip or apt, so we need to download it from the official website: https://quarto.org/docs/get-started/
• wget is a command-line utility that allows you to download files from the web
• It runs on Unix-like systems, so it is available on Ubuntu, macOS, and even on Windows
• Once we download the .deb file (which is the package format for Ubuntu), we can install it with apt-get install <package> (like we did with the other packages)
• We can download any file from the web with wget, as long as we have the URL
• I use the ./ prefix to ensure that apt-get installs the local file, not a package from the repositories
• rm just removes the downloaded file after installation to keep the image size smaller

# Download and install Quarto
# Install Quarto
RUN wget https://github.com/quarto-dev/quarto-cli/releases/download/v1.6.43/quarto-1.6.43-linux-amd64.deb && \
    apt-get install -y ./quarto-1.6.43-linux-amd64.deb && \
    rm ./quarto-1.6.43-linux-amd64.deb

• Finally, we will run the Jupyter notebook server in the container
• We will have access to the Jupyter notebook server on port 8888, so we will need to expose this port with the EXPOSE instruction
• What is cool is that the new JupyterLab version comes with an embedded terminal, so we can run bash inside the JupyterLab interface and have access to all the tools we installed in the container (like git, sqlite3, and Quarto) 😉
• Let’s also create a working directory for the Jupyter notebook server, so we can persist the notebooks outside the container

# Base image
FROM ubuntu:24.04

# Metadata
LABEL version="1.0"
LABEL description="Container with the tools covered in QTM 350"
LABEL maintainer="Danilo Freire <danilo.freire@emory.edu>"
LABEL license="MIT"

# Update and install system dependencies
# We will see why we need to install libsqlite3-0 later
RUN apt-get update && apt-get install -y --no-install-recommends\
    bash=5.2.21-2ubuntu4 \
    git=1:2.43.0-1ubuntu7.2 \
    sqlite3=3.45.1-1ubuntu2 \
    libsqlite3-0=3.45.1-1ubuntu2 \
    wget=1.21.4-1ubuntu4.1 \
    nano=7.2-2ubuntu0.1 \
    python3.12=3.12.3-1ubuntu0.5 \
    python3.12-venv=3.12.3-1ubuntu0.5 \
    python3-pip=24.0+dfsg-1ubuntu1.1 && \ 
    apt-get clean && rm -rf /var/lib/apt/lists/*

# Set default shell to Bash
SHELL ["/bin/bash", "-c"]

# Create and activate virtual environment
RUN python3 -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

# Install Python dependencies in virtual environment
RUN pip install httpx==0.27.2 numpy==1.26.4 pandas==2.2.2 \
                jupyterlab==4.2.5 ipykernel==6.29.5 \
                dask==2024.5.0 matplotlib==3.9.2

# Install Quarto
RUN wget https://github.com/quarto-dev/quarto-cli/releases/download/v1.6.37/quarto-1.6.37-linux-arm64.deb && \
    # Install the local deb file (notice the "./" prefix)
    apt-get install -y ./quarto-1.6.37-linux-arm64.deb && \
    rm quarto-1.6.37-linux-arm64.deb && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Create a directory for saving files
RUN mkdir -p /workspace
WORKDIR /workspace

# Expose port for JupyterLab 
EXPOSE 8888

# Start JupyterLab
CMD . /opt/venv/bin/activate && jupyter lab --ip=0.0.0.0 --port=8888 --no-browser --allow-root

• Now that we have the Dockerfile, we can build the container with the docker build command

docker build -t qtm350-container .

• The -t flag is used to tag the image with a name, in this case qtm350-container
• The . at the end of the command specifies the build context, which is the current directory
• Now we just need to wait for the image to be built and then run it with the docker run command
• Let’s see how it goes! 🤞

• We got an error when building the container! 😕

[...]

4.585 Some packages could not be installed. This may mean that you have
4.585 requested an impossible situation or if you are using the unstable
4.585 distribution that some required packages have not yet been created
4.585 or been moved out of Incoming.
4.585 The following information may help to resolve the situation:
4.585 
4.585 The following packages have unmet dependencies:
4.650  sqlite3 : Depends: libsqlite3-0 (= 3.45.1-1ubuntu2) but 3.45.1-1ubuntu2.1 is to be installed
4.651 E: Unable to correct problems, you have held broken packages.

• The error message indicates that there are unmet dependencies for the sqlite3 package!

• But that’s fine, we know how to fix it! 😎
• We can resolve the unmet dependencies by ensuring the correct versions of the packages are installed
• So we go to https://packages.ubuntu.com/, search for libsqlite3-0 and check the available versions
• We can see that the version 3.45.1-1ubuntu2 is available, so we can just add it to the apt-get install command
• Let’s update the Dockerfile and try again

    libsqlite3-0=3.45.1-1ubuntu2 \

• Now we can build the container again with the same command as before

docker build -t qtm350-container .

• This time it should work! 🤞

• To run the container, we will use the docker run command
• We will also use the -p flag to map the port 8888 of the container to the port 8888 of the host machine
• We will also use the -v flag to mount a volume in the container, so we can persist the notebooks outside the container

docker run -p 8888:8888 -v $(pwd):/workspace qtm350-container

• The -v flag is used to mount the current directory ($(pwd)) to the /workspace directory in the container
• This way, we can save the notebooks outside the container and access them even after the container is stopped
• Now we just need to open a web browser and go to http://127.0.0.1:8888 to access the JupyterLab interface 🚀
• Jupyter will also generate a token for you to access the interface, so you will need to copy and paste the token from the terminal to the web browser (or click on the link)

• To stop the container, you can press Ctrl+C in the terminal where the container is running
• You can also run the docker ps command to see the list of running containers and then run the docker stop command with the container ID
• You can also remove the container with the docker rm command and the image with the docker rmi command
• Or you can just check Docker Desktop and stop the container from there, as well as remove or restart it
• To tag and push the container to Docker Hub, you can use the docker tag and docker push commands

docker tag qtm350-container danilofreire/qtm350-container
docker push danilofreire/qtm350-container

• To clean up your system, you can remove the container and the image with the following commands
• First, list the running containers with docker ps -a to get the container ID or name. ps is short for “process status” and the -a flag shows all containers, including those that are stopped
• Then, stop the container if it is running with docker stop <container_id_or_name>
• Next, remove the container with docker rm <container_id_or_name>
• Finally, remove the image with docker rmi qtm350-container
• You can also use docker system prune to remove all stopped containers, unused networks, images, and build cache (use the -a flag to remove all unused images too). Warning: This is a more aggressive cleanup command!

docker ps -a
docker stop <container_id_or_name>
docker rm <container_id_or_name>
docker rmi qtm350-container
# Optional: docker system prune -a

• There are many ways to run Docker containers on AWS:
• You can use the Amazon Elastic Container Service, which is a fully managed container orchestration service. More information here: https://aws.amazon.com/ecs/getting-started/
• Amazon SageMaker, which allows you to develop your machine learning models. More information here: https://docs.aws.amazon.com/sagemaker/latest/dg/docker-containers.html
• AWS Lambda. More information here: https://docs.aws.amazon.com/lambda/latest/dg/images-create.html
• And of course, you use an EC2 instance and install Docker on it, like we did a few weeks ago!