QTM 350 - Data Science Computing

Lecture 27 - Parallelising Data Analysis with Dask and AutoML

Danilo Freire

danilo.freire@emory.edu

Emory University

Hello again! 😊
How’s everything?

Brief recap of last class 📚

Parallel computing

Parallel computing is a type of computation in which many calculations or processes are carried out simultaneously
Python has several libraries that allow you to parallelise your code
- We discussed the joblib library, which is a simple and effective way to parallelise your code
- But we spent most of our time discussing dask, which is currently the de facto standard for parallel computing in Python
We also saw that parallel computing is not a panacea: it can be hard to implement and may not always lead to performance improvements
But when it works, it works great! 🚀

Today’s agenda 📅

Lecture outline

Today we will continue our discussion on parallel computing
We will focus on parallelising data analysis tasks with Dask ML
More specifically, we will discuss how to parallelise the training of machine learning models, and how to use automated machine learning (AutoML) tools to speed up the process
We will also learn about Dask Clusters, which allow you to scale your computations across multiple machines (or just one!)

Dask Clients and Clusters 🌐

What is a Dask Cluster?

Workers and schedulers

A Dask Cluster is a collection of Dask workers that can be used to parallelise your computations
In plain English, a Dask Cluster is a group of computational engines (cores, GPUs, servers, etc) that work together to solve a problem
Workers provide two functions:
- Compute tasks as directed by the scheduler, the scheduler being the brain of the cluster
- Store and serve computed results to other workers or clients
Workers are the reason why lazy evaluation speeds up computations
A simple example of workers interacting with a scheduler can help explain how lazy evaluation works:

Scheduler -> Eve: Compute a <- multiply(3, 5)!
Eve -> Scheduler: I've computed a and am holding on to it! 
Scheduler -> Frank: Compute b <- add(a, 7)!
Frank: You will need a. Eve has a.
Frank -> Eve: Please send me a.
Eve -> Frank: Sure. a is 15!
Frank -> Scheduler: I've computed b and am holding on to it!

What is a Dask Cluster?

Workers and schedulers

How do workers know what to do?
- The scheduler assigns tasks to workers based on their availability
- Workers can be on the same machine or on different machines
- Workers can be CPUs or GPUs
Dask workers save their data as a Python dictionary, which is then sent to the scheduler as a thread in the concurrent.futures library
These processes can automatically restart and scale up without any intervention from the user
The optimal number of workers depends on the size of the data and the complexity of the computations, but often the default configuration is sufficient
In an adaptive cluster, you set the minimum and maximum number of workers and let the cluster add and remove workers as needed
Dask also provides a dashboard to monitor the performance of your cluster

Virtual environment

Before we start, let’s make sure we are using the virtual environment we created in the previous class:

conda create -n venv-dask -c conda-forge python=3.10 dask-sql=2024.5.0 dask=2024.4.1 dask-ml=2024.4.4 ipykernel=6.29.3 joblib=1.3.2 numpy=1.26.4 pandas=2.2.1 scikit-learn=1.4.2 tpot=0.12.2 prophet -y

# or if you are using pip
python -m venv venv-dask
pip install dask-sql=2024.5.0 dask=2024.4.1 ipykernel=6.29.3 joblib=1.3.2 numpy=1.26.4 pandas=2.2.1 scikit-learn=1.4.2 prophet tpot=0.12.2 dask-ml=2024.4.4 -y

If you are using Anaconda, you can activate the environment with:

conda activate venv-dask

If you are using venv, you can activate the environment with:

source venv-dask/bin/activate # macOS/Linux
source venv-dask/Scripts/activate # Windows

The venv-dask environment should have all the necessary packages installed!

Setting up a Dask Client

Using dask.distributed requires that you set up a Client
This should be the first thing you do if you intend to use dask.distributed in your analysis
It offers low latency, data locality, data sharing between the workers, and is easy to set up

from dask.distributed import LocalCluster, Client

# Specify a different port for the dashboard
cluster = LocalCluster(dashboard_address=':8789') 
client = Client(cluster)

# Print the dashboard link
print(f"Dask dashboard is available at: {cluster.dashboard_link}")

Dask dashboard is available at: http://127.0.0.1:8789/status

The Client object provides a way to interact with the cluster, submit tasks, and monitor the progress of computations
You will see a screen like this in your browser:

Dask Client Dashboard

Testing the Dask Client

You can test the Dask Client by running a simple computation

import dask.array as da

# Create a random array
x = da.random.RandomState(42).random((10000, 10000), chunks=(1000, 1000))
x

	Array	Chunk
Bytes	762.94 MiB	7.63 MiB
Shape	(10000, 10000)	(1000, 1000)
Dask graph	100 chunks in 1 graph layer
Data type	float64 numpy.ndarray

# Perform a simple computation
y = (x + x.T).sum()

# Compute the result
y.compute()

99987830.48502485

The RandomState object is used to set a seed number
We can inspect the client dashboard to see how the computation was distributed across the workers

Testing the Dask Client

Dask ML 🤖

Dimensions of Scale

Addressing the Challenges

Challenge 1: Scaling Model Size

Model size: the number of parameters in a model
If your models become more complex, you need more computational resources to train them
Under this scaling challenge tasks like model training, prediction, or evaluation steps will (eventually) complete, they just take too long
You’ve become compute bound
You can continue to use your current algorithms and libraries (pandas, numpy, scikit-learn, etc), but you need to scale them up

Challenge 2: Scaling Data Size

Data size: the number of samples in your dataset
There are cases where datasets grow larger than RAM (shown along the horizontal axis)
Under this scaling challenge, even loading the data into numpy or pandas becomes impossible
You’ve become memory bound
In this case, you can use a different file format (parquet, Dask DataFrame, etc) together with algorithms that can handle large datasets

What is Dask ML?

Dask ML is built on top of Dask Arrays and DataFrames, allowing you to scale your machine learning workflows to large datasets
It provides parallel implementations of many popular machine learning algorithms and libraries, such as scikit-learn, XGBoost, LightGBM, TensorFlow, and PyTorch
You can use Dask ML to do many tasks, such as model selection, model evaluation, and, most importantly, hyperparameter tuning
And you can also use it together with automated machine learning (AutoML) tools to speed up the process
So let’s see what AutoML is and how to use it with Dask ML!

AutoML 🤖

What is AutoML?

AutoML is a set of tools that, well, automate the process of applying machine learning
The main goal of AutoML is to make machine learning more accessible to non-experts and to speed up the process of building machine learning models
Think about it as a more advanced version of scikit-learn (or a more realistic version of “vibe coding” 😂)
This allows users to focus on the problem at hand rather than the intricacies of the algorithms
There are several tools available for AutoML, such as:
All of them are great, but they have different features and capabilities
Here we will use Dask ML to parallelise the training of machine learning models with scikit-learn

What is Dask ML?

First, let’s import the necessary modules, mainly scikit-learn:

import time
import numpy as np
from dask.distributed import LocalCluster, Client
import joblib
from sklearn.datasets import load_digits
from sklearn.model_selection import RandomizedSearchCV
from sklearn.svm import SVC

The load_digits dataset is a well-known dataset in machine learning, containing 1797 8x8 pixel images of handwritten digits
The main task here is to predict the digit from the image

Dask ML and scikit-learn

Let’s estimate the model with a simple grid search
param_space: A list of settings to try out for the model
C: Controls how much to punish mistakes (regularisation, smaller values = more regularisation) to prevent overfitting
- This is in exponential notation, so np.logspace(-6, 6, 13) will create a list of 13 values between \(10^{-6}\) and \(10^6\) (!)
gamma: Defines how far the influence of a single example reaches (small points = model is less sensitive to the data)
- This is also in exponential notation, so np.logspace(-8, 8, 17) will create a list of 17 values between \(10^{-8}\) and \(10^8\) (!!)
tol: Tells the model when to stop trying to improve
class_weight: Options for handling imbalanced data
This is pretty standard scikit-learn code, but with a twist: we are using joblib.parallel_backend('dask') to parallelise the search
This will distribute the search across the workers in the Dask cluster (and we don’t have to worry about it!)
The RandomizedSearchCV object will try out 50 different combinations of hyperparameters and return the best one

Dask ML and scikit-learn

# Load the digits dataset
digits = load_digits()   

# Define the parameter space to search through
param_space = {
    'C': np.logspace(-6, 6, 13),      
    'gamma': np.logspace(-8, 8, 17),
    'tol': np.logspace(-4, -1, 4),
    'class_weight': [None, 'balanced'], 
}

# Create the model
model = SVC()

search = RandomizedSearchCV(
    model,
    param_space,
    cv=3,
    n_iter=50,
    verbose=10
)

# Perform the search using Dask
start_time = time.time()
with joblib.parallel_backend('dask'):
    search.fit(digits.data, digits.target)
end_time = time.time()

# Calculate the elapsed time
elapsed_time = end_time - start_time

# Print the best parameters
print("Best parameters found: ", search.best_params_)
print("Best score: ", search.best_score_)
print("Best estimator: ", search.best_estimator_)
print("Time taken: {:.2f} seconds".format(elapsed_time))

Fitting 3 folds for each of 50 candidates, totalling 150 fits
[CV 3/3; 2/50] START C=0.001, class_weight=None, gamma=100000000.0, tol=0.01....
[CV 1/3; 6/50] START C=0.0001, class_weight=balanced, gamma=100000.0, tol=0.001.
[CV 1/3; 5/50] START C=1e-05, class_weight=None, gamma=1e-08, tol=0.1...........
[CV 1/3; 1/50] START C=1000.0, class_weight=balanced, gamma=10000.0, tol=0.1....
[CV 2/3; 1/50] START C=1000.0, class_weight=balanced, gamma=10000.0, tol=0.1....
[CV 3/3; 4/50] START C=1000000.0, class_weight=balanced, gamma=100000000.0, tol=0.0001
[CV 3/3; 5/50] START C=1e-05, class_weight=None, gamma=1e-08, tol=0.1...........
[CV 1/3; 4/50] START C=1000000.0, class_weight=balanced, gamma=100000000.0, tol=0.0001
[CV 3/3; 2/50] END C=0.001, class_weight=None, gamma=100000000.0, tol=0.01;, score=0.102 total time=   0.1s
[CV 2/3; 4/50] START C=1000000.0, class_weight=balanced, gamma=100000000.0, tol=0.0001
[CV 1/3; 5/50] END C=1e-05, class_weight=None, gamma=1e-08, tol=0.1;, score=0.292 total time=   0.1s
[CV 2/3; 3/50] START C=0.0001, class_weight=balanced, gamma=1000.0, tol=0.01....
[CV 1/3; 1/50] END C=1000.0, class_weight=balanced, gamma=10000.0, tol=0.1;, score=0.100 total time=   0.1s
[CV 1/3; 6/50] END C=0.0001, class_weight=balanced, gamma=100000.0, tol=0.001;, score=0.098 total time=   0.2s
[CV 2/3; 5/50] START C=1e-05, class_weight=None, gamma=1e-08, tol=0.1...........
[CV 1/3; 3/50] START C=0.0001, class_weight=balanced, gamma=1000.0, tol=0.01....
[CV 2/3; 1/50] END C=1000.0, class_weight=balanced, gamma=10000.0, tol=0.1;, score=0.102 total time=   0.2s
[CV 1/3; 2/50] START C=0.001, class_weight=None, gamma=100000000.0, tol=0.01....
[CV 3/3; 4/50] END C=1000000.0, class_weight=balanced, gamma=100000000.0, tol=0.0001;, score=0.102 total time=   0.2s
[CV 1/3; 4/50] END C=1000000.0, class_weight=balanced, gamma=100000000.0, tol=0.0001;, score=0.100 total time=   0.2s
[CV 2/3; 2/50] START C=0.001, class_weight=None, gamma=100000000.0, tol=0.01....
[CV 3/3; 6/50] START C=0.0001, class_weight=balanced, gamma=100000.0, tol=0.001.
[CV 3/3; 5/50] END C=1e-05, class_weight=None, gamma=1e-08, tol=0.1;, score=0.102 total time=   0.2s
[CV 3/3; 1/50] START C=1000.0, class_weight=balanced, gamma=10000.0, tol=0.1....
[CV 2/3; 3/50] END C=0.0001, class_weight=balanced, gamma=1000.0, tol=0.01;, score=0.100 total time=   0.2s
[CV 1/3; 8/50] START C=1000000.0, class_weight=None, gamma=10.0, tol=0.01.......
[CV 2/3; 4/50] END C=1000000.0, class_weight=balanced, gamma=100000000.0, tol=0.0001;, score=0.102 total time=   0.2s
[CV 2/3; 8/50] START C=1000000.0, class_weight=None, gamma=10.0, tol=0.01.......
[CV 1/3; 2/50] END C=0.001, class_weight=None, gamma=100000000.0, tol=0.01;, score=0.100 total time=   0.2s
[CV 1/3; 3/50] END C=0.0001, class_weight=balanced, gamma=1000.0, tol=0.01;, score=0.098 total time=   0.2s
[CV 2/3; 6/50] START C=0.0001, class_weight=balanced, gamma=100000.0, tol=0.001.[CV 3/3; 7/50] START C=1.0, class_weight=None, gamma=0.1, tol=0.0001............

[CV 2/3; 5/50] END C=1e-05, class_weight=None, gamma=1e-08, tol=0.1;, score=0.102 total time=   0.2s
[CV 3/3; 3/50] START C=0.0001, class_weight=balanced, gamma=1000.0, tol=0.01....
[CV 3/3; 1/50] END C=1000.0, class_weight=balanced, gamma=10000.0, tol=0.1;, score=0.102 total time=   0.2s
[CV 3/3; 6/50] END C=0.0001, class_weight=balanced, gamma=100000.0, tol=0.001;, score=0.100 total time=   0.2s
[CV 2/3; 7/50] START C=1.0, class_weight=None, gamma=0.1, tol=0.0001............
[CV 2/3; 2/50] END C=0.001, class_weight=None, gamma=100000000.0, tol=0.01;, score=0.102 total time=   0.3s
[CV 1/3; 7/50] START C=1.0, class_weight=None, gamma=0.1, tol=0.0001............
[CV 3/3; 8/50] START C=1000000.0, class_weight=None, gamma=10.0, tol=0.01.......
[CV 2/3; 8/50] END C=1000000.0, class_weight=None, gamma=10.0, tol=0.01;, score=0.102 total time=   0.3s
[CV 2/3; 9/50] START C=1.0, class_weight=balanced, gamma=1e-07, tol=0.001.......
[CV 3/3; 7/50] END C=1.0, class_weight=None, gamma=0.1, tol=0.0001;, score=0.102 total time=   0.3s
[CV 3/3; 9/50] START C=1.0, class_weight=balanced, gamma=1e-07, tol=0.001.......
[CV 2/3; 6/50] END C=0.0001, class_weight=balanced, gamma=100000.0, tol=0.001;, score=0.100 total time=   0.3s
[CV 1/3; 10/50] START C=0.0001, class_weight=balanced, gamma=0.1, tol=0.0001....
[CV 3/3; 3/50] END C=0.0001, class_weight=balanced, gamma=1000.0, tol=0.01;, score=0.100 total time=   0.3s
[CV 1/3; 11/50] START C=100000.0, class_weight=None, gamma=1.0, tol=0.1.........
[CV 1/3; 7/50] END C=1.0, class_weight=None, gamma=0.1, tol=0.0001;, score=0.102 total time=   0.2s
[CV 3/3; 10/50] START C=0.0001, class_weight=balanced, gamma=0.1, tol=0.0001....
[CV 2/3; 7/50] END C=1.0, class_weight=None, gamma=0.1, tol=0.0001;, score=0.102 total time=   0.3s
[CV 3/3; 8/50] END C=1000000.0, class_weight=None, gamma=10.0, tol=0.01;, score=0.102 total time=   0.3s
[CV 2/3; 10/50] START C=0.0001, class_weight=balanced, gamma=0.1, tol=0.0001....
[CV 2/3; 11/50] START C=100000.0, class_weight=None, gamma=1.0, tol=0.1.........
[CV 1/3; 8/50] END C=1000000.0, class_weight=None, gamma=10.0, tol=0.01;, score=0.100 total time=   0.4s
[CV 1/3; 9/50] START C=1.0, class_weight=balanced, gamma=1e-07, tol=0.001.......
[CV 2/3; 9/50] END C=1.0, class_weight=balanced, gamma=1e-07, tol=0.001;, score=0.098 total time=   0.2s
[CV 2/3; 13/50] START C=0.001, class_weight=None, gamma=1000.0, tol=0.001.......
[CV 1/3; 11/50] END C=100000.0, class_weight=None, gamma=1.0, tol=0.1;, score=0.202 total time=   0.2s
[CV 3/3; 10/50] END C=0.0001, class_weight=balanced, gamma=0.1, tol=0.0001;, score=0.098 total time=   0.1s
[CV 3/3; 13/50] START C=0.001, class_weight=None, gamma=1000.0, tol=0.001.......
[CV 1/3; 13/50] START C=0.001, class_weight=None, gamma=1000.0, tol=0.001.......
[CV 1/3; 10/50] END C=0.0001, class_weight=balanced, gamma=0.1, tol=0.0001;, score=0.098 total time=   0.2s
[CV 1/3; 14/50] START C=0.01, class_weight=None, gamma=0.1, tol=0.001...........
[CV 2/3; 11/50] END C=100000.0, class_weight=None, gamma=1.0, tol=0.1;, score=0.102 total time=   0.2s
[CV 3/3; 9/50] END C=1.0, class_weight=balanced, gamma=1e-07, tol=0.001;, score=0.098 total time=   0.2s
[CV 2/3; 12/50] START C=1000.0, class_weight=None, gamma=1000000.0, tol=0.1.....
[CV 3/3; 11/50] START C=100000.0, class_weight=None, gamma=1.0, tol=0.1.........
[CV 1/3; 9/50] END C=1.0, class_weight=balanced, gamma=1e-07, tol=0.001;, score=0.197 total time=   0.2s
[CV 1/3; 12/50] START C=1000.0, class_weight=None, gamma=1000000.0, tol=0.1.....
[CV 2/3; 10/50] END C=0.0001, class_weight=balanced, gamma=0.1, tol=0.0001;, score=0.100 total time=   0.2s
[CV 3/3; 12/50] START C=1000.0, class_weight=None, gamma=1000000.0, tol=0.1.....
[CV 3/3; 13/50] END C=0.001, class_weight=None, gamma=1000.0, tol=0.001;, score=0.102 total time=   0.1s
[CV 3/3; 16/50] START C=0.0001, class_weight=None, gamma=1e-06, tol=0.0001......
[CV 2/3; 13/50] END C=0.001, class_weight=None, gamma=1000.0, tol=0.001;, score=0.102 total time=   0.2s
[CV 2/3; 16/50] START C=0.0001, class_weight=None, gamma=1e-06, tol=0.0001......
[CV 1/3; 13/50] END C=0.001, class_weight=None, gamma=1000.0, tol=0.001;, score=0.100 total time=   0.2s
[CV 3/3; 15/50] START C=0.1, class_weight=balanced, gamma=1000.0, tol=0.001.....
[CV 1/3; 12/50] END C=1000.0, class_weight=None, gamma=1000000.0, tol=0.1;, score=0.100 total time=   0.2s
[CV 1/3; 16/50] START C=0.0001, class_weight=None, gamma=1e-06, tol=0.0001......
[CV 3/3; 11/50] END C=100000.0, class_weight=None, gamma=1.0, tol=0.1;, score=0.102 total time=   0.2s
[CV 3/3; 14/50] START C=0.01, class_weight=None, gamma=0.1, tol=0.001...........
[CV 1/3; 14/50] END C=0.01, class_weight=None, gamma=0.1, tol=0.001;, score=0.100 total time=   0.2s
[CV 2/3; 12/50] END C=1000.0, class_weight=None, gamma=1000000.0, tol=0.1;, score=0.102 total time=   0.2s
[CV 1/3; 15/50] START C=0.1, class_weight=balanced, gamma=1000.0, tol=0.001.....
[CV 2/3; 14/50] START C=0.01, class_weight=None, gamma=0.1, tol=0.001...........
[CV 3/3; 12/50] END C=1000.0, class_weight=None, gamma=1000000.0, tol=0.1;, score=0.102 total time=   0.2s
[CV 2/3; 15/50] START C=0.1, class_weight=balanced, gamma=1000.0, tol=0.001.....
[CV 3/3; 16/50] END C=0.0001, class_weight=None, gamma=1e-06, tol=0.0001;, score=0.102 total time=   0.1s
[CV 1/3; 17/50] START C=0.01, class_weight=balanced, gamma=0.0001, tol=0.0001...
[CV 3/3; 15/50] END C=0.1, class_weight=balanced, gamma=1000.0, tol=0.001;, score=0.098 total time=   0.1s
[CV 3/3; 18/50] START C=10.0, class_weight=None, gamma=0.1, tol=0.0001..........
[CV 2/3; 16/50] END C=0.0001, class_weight=None, gamma=1e-06, tol=0.0001;, score=0.102 total time=   0.2s
[CV 2/3; 18/50] START C=10.0, class_weight=None, gamma=0.1, tol=0.0001..........
[CV 1/3; 15/50] END C=0.1, class_weight=balanced, gamma=1000.0, tol=0.001;, score=0.098 total time=   0.2s
[CV 1/3; 19/50] START C=0.01, class_weight=balanced, gamma=100.0, tol=0.1.......
[CV 3/3; 14/50] END C=0.01, class_weight=None, gamma=0.1, tol=0.001;, score=0.102 total time=   0.2s
[CV 1/3; 17/50] END C=0.01, class_weight=balanced, gamma=0.0001, tol=0.0001;, score=0.199 total time=   0.1s
[CV 3/3; 19/50] START C=0.01, class_weight=balanced, gamma=100.0, tol=0.1.......
[CV 3/3; 17/50] START C=0.01, class_weight=balanced, gamma=0.0001, tol=0.0001...
[CV 2/3; 15/50] END C=0.1, class_weight=balanced, gamma=1000.0, tol=0.001;, score=0.098 total time=   0.2s
[CV 2/3; 19/50] START C=0.01, class_weight=balanced, gamma=100.0, tol=0.1.......
[CV 2/3; 14/50] END C=0.01, class_weight=None, gamma=0.1, tol=0.001;, score=0.102 total time=   0.2s
[CV 2/3; 17/50] START C=0.01, class_weight=balanced, gamma=0.0001, tol=0.0001...
[CV 3/3; 18/50] END C=10.0, class_weight=None, gamma=0.1, tol=0.0001;, score=0.102 total time=   0.2s
[CV 1/3; 20/50] START C=0.001, class_weight=balanced, gamma=1e-06, tol=0.0001...
[CV 2/3; 18/50] END C=10.0, class_weight=None, gamma=0.1, tol=0.0001;, score=0.102 total time=   0.2s
[CV 2/3; 20/50] START C=0.001, class_weight=balanced, gamma=1e-06, tol=0.0001...
[CV 1/3; 16/50] END C=0.0001, class_weight=None, gamma=1e-06, tol=0.0001;, score=0.292 total time=   0.3s
[CV 1/3; 18/50] START C=10.0, class_weight=None, gamma=0.1, tol=0.0001..........
[CV 3/3; 19/50] END C=0.01, class_weight=balanced, gamma=100.0, tol=0.1;, score=0.098 total time=   0.1s
[CV 2/3; 21/50] START C=0.0001, class_weight=balanced, gamma=10.0, tol=0.001....
[CV 2/3; 19/50] END C=0.01, class_weight=balanced, gamma=100.0, tol=0.1;, score=0.098 total time=   0.1s
[CV 1/3; 19/50] END C=0.01, class_weight=balanced, gamma=100.0, tol=0.1;, score=0.098 total time=   0.2s
[CV 3/3; 20/50] START C=0.001, class_weight=balanced, gamma=1e-06, tol=0.0001...
[CV 1/3; 21/50] START C=0.0001, class_weight=balanced, gamma=10.0, tol=0.001....
[CV 3/3; 17/50] END C=0.01, class_weight=balanced, gamma=0.0001, tol=0.0001;, score=0.098 total time=   0.2s
[CV 2/3; 22/50] START C=1000000.0, class_weight=balanced, gamma=0.1, tol=0.1....
[CV 2/3; 17/50] END C=0.01, class_weight=balanced, gamma=0.0001, tol=0.0001;, score=0.098 total time=   0.2s
[CV 1/3; 20/50] END C=0.001, class_weight=balanced, gamma=1e-06, tol=0.0001;, score=0.197 total time=   0.2s
[CV 3/3; 21/50] START C=0.0001, class_weight=balanced, gamma=10.0, tol=0.001....
[CV 1/3; 22/50] START C=1000000.0, class_weight=balanced, gamma=0.1, tol=0.1....
[CV 2/3; 20/50] END C=0.001, class_weight=balanced, gamma=1e-06, tol=0.0001;, score=0.098 total time=   0.1s
[CV 1/3; 25/50] START C=10000.0, class_weight=None, gamma=1e-08, tol=0.001......
[CV 2/3; 21/50] END C=0.0001, class_weight=balanced, gamma=10.0, tol=0.001;, score=0.100 total time=   0.2s
[CV 1/3; 24/50] START C=100.0, class_weight=None, gamma=1e-05, tol=0.1..........
[CV 1/3; 25/50] END C=10000.0, class_weight=None, gamma=1e-08, tol=0.001;, score=0.942 total time=   0.1s
[CV 1/3; 23/50] START C=100.0, class_weight=balanced, gamma=100000000.0, tol=0.01
[CV 3/3; 20/50] END C=0.001, class_weight=balanced, gamma=1e-06, tol=0.0001;, score=0.098 total time=   0.2s
[CV 2/3; 23/50] START C=100.0, class_weight=balanced, gamma=100000000.0, tol=0.01
[CV 1/3; 24/50] END C=100.0, class_weight=None, gamma=1e-05, tol=0.1;, score=0.937 total time=   0.0s
[CV 3/3; 25/50] START C=10000.0, class_weight=None, gamma=1e-08, tol=0.001......
[CV 1/3; 21/50] END C=0.0001, class_weight=balanced, gamma=10.0, tol=0.001;, score=0.098 total time=   0.2s
[CV 2/3; 22/50] END C=1000000.0, class_weight=balanced, gamma=0.1, tol=0.1;, score=0.102 total time=   0.2s
[CV 3/3; 23/50] START C=100.0, class_weight=balanced, gamma=100000000.0, tol=0.01
[CV 3/3; 24/50] START C=100.0, class_weight=None, gamma=1e-05, tol=0.1..........
[CV 3/3; 21/50] END C=0.0001, class_weight=balanced, gamma=10.0, tol=0.001;, score=0.100 total time=   0.1s
[CV 2/3; 24/50] START C=100.0, class_weight=None, gamma=1e-05, tol=0.1..........
[CV 1/3; 18/50] END C=10.0, class_weight=None, gamma=0.1, tol=0.0001;, score=0.102 total time=   0.3s
[CV 1/3; 22/50] END C=1000000.0, class_weight=balanced, gamma=0.1, tol=0.1;, score=0.102 total time=   0.2s
[CV 1/3; 26/50] START C=1000.0, class_weight=None, gamma=1e-08, tol=0.0001......
[CV 3/3; 22/50] START C=1000000.0, class_weight=balanced, gamma=0.1, tol=0.1....
[CV 3/3; 24/50] END C=100.0, class_weight=None, gamma=1e-05, tol=0.1;, score=0.942 total time=   0.0s
[CV 1/3; 27/50] START C=100000.0, class_weight=balanced, gamma=100000000.0, tol=0.001
[CV 2/3; 24/50] END C=100.0, class_weight=None, gamma=1e-05, tol=0.1;, score=0.967 total time=   0.0s
[CV 2/3; 28/50] START C=10.0, class_weight=None, gamma=1.0, tol=0.001...........
[CV 3/3; 25/50] END C=10000.0, class_weight=None, gamma=1e-08, tol=0.001;, score=0.927 total time=   0.1s
[CV 2/3; 26/50] START C=1000.0, class_weight=None, gamma=1e-08, tol=0.0001......
[CV 1/3; 23/50] END C=100.0, class_weight=balanced, gamma=100000000.0, tol=0.01;, score=0.100 total time=   0.1s
[CV 2/3; 25/50] START C=10000.0, class_weight=None, gamma=1e-08, tol=0.001......
[CV 2/3; 23/50] END C=100.0, class_weight=balanced, gamma=100000000.0, tol=0.01;, score=0.102 total time=   0.1s
[CV 1/3; 29/50] START C=0.001, class_weight=None, gamma=1e-08, tol=0.001........
[CV 3/3; 23/50] END C=100.0, class_weight=balanced, gamma=100000000.0, tol=0.01;, score=0.102 total time=   0.2s
[CV 3/3; 26/50] START C=1000.0, class_weight=None, gamma=1e-08, tol=0.0001......
[CV 1/3; 26/50] END C=1000.0, class_weight=None, gamma=1e-08, tol=0.0001;, score=0.890 total time=   0.2s
[CV 2/3; 27/50] START C=100000.0, class_weight=balanced, gamma=100000000.0, tol=0.001
[CV 2/3; 25/50] END C=10000.0, class_weight=None, gamma=1e-08, tol=0.001;, score=0.953 total time=   0.1s
[CV 3/3; 29/50] START C=0.001, class_weight=None, gamma=1e-08, tol=0.001........
[CV 2/3; 26/50] END C=1000.0, class_weight=None, gamma=1e-08, tol=0.0001;, score=0.885 total time=   0.1s
[CV 1/3; 28/50] START C=10.0, class_weight=None, gamma=1.0, tol=0.001...........
[CV 1/3; 27/50] END C=100000.0, class_weight=balanced, gamma=100000000.0, tol=0.001;, score=0.100 total time=   0.2s
[CV 2/3; 29/50] START C=0.001, class_weight=None, gamma=1e-08, tol=0.001........
[CV 2/3; 28/50] END C=10.0, class_weight=None, gamma=1.0, tol=0.001;, score=0.102 total time=   0.2s
[CV 3/3; 28/50] START C=10.0, class_weight=None, gamma=1.0, tol=0.001...........
[CV 3/3; 22/50] END C=1000000.0, class_weight=balanced, gamma=0.1, tol=0.1;, score=0.102 total time=   0.2s
[CV 3/3; 27/50] START C=100000.0, class_weight=balanced, gamma=100000000.0, tol=0.001
[CV 1/3; 29/50] END C=0.001, class_weight=None, gamma=1e-08, tol=0.001;, score=0.292 total time=   0.2s
[CV 2/3; 31/50] START C=100000.0, class_weight=None, gamma=10000000.0, tol=0.1..
[CV 3/3; 26/50] END C=1000.0, class_weight=None, gamma=1e-08, tol=0.0001;, score=0.865 total time=   0.2s
[CV 3/3; 30/50] START C=100000.0, class_weight=None, gamma=100.0, tol=0.001.....
[CV 2/3; 27/50] END C=100000.0, class_weight=balanced, gamma=100000000.0, tol=0.001;, score=0.102 total time=   0.2s
[CV 1/3; 31/50] START C=100000.0, class_weight=None, gamma=10000000.0, tol=0.1..
[CV 3/3; 29/50] END C=0.001, class_weight=None, gamma=1e-08, tol=0.001;, score=0.102 total time=   0.2s
[CV 2/3; 30/50] START C=100000.0, class_weight=None, gamma=100.0, tol=0.001.....
[CV 2/3; 29/50] END C=0.001, class_weight=None, gamma=1e-08, tol=0.001;, score=0.102 total time=   0.2s
[CV 1/3; 28/50] END C=10.0, class_weight=None, gamma=1.0, tol=0.001;, score=0.202 total time=   0.2s
[CV 1/3; 30/50] START C=100000.0, class_weight=None, gamma=100.0, tol=0.001.....
[CV 3/3; 31/50] START C=100000.0, class_weight=None, gamma=10000000.0, tol=0.1..
[CV 3/3; 28/50] END C=10.0, class_weight=None, gamma=1.0, tol=0.001;, score=0.102 total time=   0.2s
[CV 1/3; 32/50] START C=1.0, class_weight=balanced, gamma=1e-05, tol=0.0001.....
[CV 3/3; 27/50] END C=100000.0, class_weight=balanced, gamma=100000000.0, tol=0.001;, score=0.102 total time=   0.2s
[CV 1/3; 33/50] START C=0.01, class_weight=balanced, gamma=10.0, tol=0.0001.....
[CV 2/3; 31/50] END C=100000.0, class_weight=None, gamma=10000000.0, tol=0.1;, score=0.102 total time=   0.2s
[CV 3/3; 32/50] START C=1.0, class_weight=balanced, gamma=1e-05, tol=0.0001.....
[CV 3/3; 30/50] END C=100000.0, class_weight=None, gamma=100.0, tol=0.001;, score=0.102 total time=   0.2s
[CV 2/3; 32/50] START C=1.0, class_weight=balanced, gamma=1e-05, tol=0.0001.....
[CV 1/3; 31/50] END C=100000.0, class_weight=None, gamma=10000000.0, tol=0.1;, score=0.100 total time=   0.2s
[CV 1/3; 34/50] START C=1e-06, class_weight=balanced, gamma=1000.0, tol=0.01....
[CV 1/3; 32/50] END C=1.0, class_weight=balanced, gamma=1e-05, tol=0.0001;, score=0.895 total time=   0.1s
[CV 2/3; 30/50] END C=100000.0, class_weight=None, gamma=100.0, tol=0.001;, score=0.102 total time=   0.2s
[CV 1/3; 35/50] START C=1000000.0, class_weight=None, gamma=1e-08, tol=0.1......
[CV 3/3; 33/50] START C=0.01, class_weight=balanced, gamma=10.0, tol=0.0001.....
[CV 3/3; 31/50] END C=100000.0, class_weight=None, gamma=10000000.0, tol=0.1;, score=0.102 total time=   0.2s
[CV 2/3; 33/50] START C=0.01, class_weight=balanced, gamma=10.0, tol=0.0001.....
[CV 1/3; 35/50] END C=1000000.0, class_weight=None, gamma=1e-08, tol=0.1;, score=0.938 total time=   0.0s
[CV 3/3; 36/50] START C=100.0, class_weight=None, gamma=0.0001, tol=0.01........
[CV 1/3; 30/50] END C=100000.0, class_weight=None, gamma=100.0, tol=0.001;, score=0.100 total time=   0.2s
[CV 2/3; 35/50] START C=1000000.0, class_weight=None, gamma=1e-08, tol=0.1......
[CV 2/3; 35/50] END C=1000000.0, class_weight=None, gamma=1e-08, tol=0.1;, score=0.957 total time=   0.0s
[CV 2/3; 34/50] START C=1e-06, class_weight=balanced, gamma=1000.0, tol=0.01....
[CV 3/3; 36/50] END C=100.0, class_weight=None, gamma=0.0001, tol=0.01;, score=0.952 total time=   0.0s
[CV 3/3; 37/50] START C=1000000.0, class_weight=balanced, gamma=1e-07, tol=0.1..
[CV 3/3; 32/50] END C=1.0, class_weight=balanced, gamma=1e-05, tol=0.0001;, score=0.873 total time=   0.2s
[CV 1/3; 38/50] START C=0.1, class_weight=balanced, gamma=10.0, tol=0.1.........
[CV 3/3; 37/50] END C=1000000.0, class_weight=balanced, gamma=1e-07, tol=0.1;, score=0.937 total time=   0.0s
[CV 1/3; 33/50] END C=0.01, class_weight=balanced, gamma=10.0, tol=0.0001;, score=0.098 total time=   0.2s
[CV 2/3; 36/50] START C=100.0, class_weight=None, gamma=0.0001, tol=0.01........
[CV 3/3; 34/50] START C=1e-06, class_weight=balanced, gamma=1000.0, tol=0.01....
[CV 2/3; 36/50] END C=100.0, class_weight=None, gamma=0.0001, tol=0.01;, score=0.963 total time=   0.0s
[CV 2/3; 39/50] START C=10000.0, class_weight=None, gamma=10000000.0, tol=0.001.
[CV 1/3; 34/50] END C=1e-06, class_weight=balanced, gamma=1000.0, tol=0.01;, score=0.098 total time=   0.2s
[CV 1/3; 37/50] START C=1000000.0, class_weight=balanced, gamma=1e-07, tol=0.1..
[CV 1/3; 37/50] END C=1000000.0, class_weight=balanced, gamma=1e-07, tol=0.1;, score=0.938 total time=   0.0s
[CV 1/3; 36/50] START C=100.0, class_weight=None, gamma=0.0001, tol=0.01........
[CV 2/3; 32/50] END C=1.0, class_weight=balanced, gamma=1e-05, tol=0.0001;, score=0.885 total time=   0.2s
[CV 3/3; 35/50] START C=1000000.0, class_weight=None, gamma=1e-08, tol=0.1......
[CV 2/3; 33/50] END C=0.01, class_weight=balanced, gamma=10.0, tol=0.0001;, score=0.098 total time=   0.2s
[CV 3/3; 33/50] END C=0.01, class_weight=balanced, gamma=10.0, tol=0.0001;, score=0.098 total time=   0.2s
[CV 3/3; 39/50] START C=10000.0, class_weight=None, gamma=10000000.0, tol=0.001.
[CV 1/3; 40/50] START C=100.0, class_weight=balanced, gamma=0.0001, tol=0.001...
[CV 3/3; 35/50] END C=1000000.0, class_weight=None, gamma=1e-08, tol=0.1;, score=0.937 total time=   0.0s
[CV 1/3; 39/50] START C=10000.0, class_weight=None, gamma=10000000.0, tol=0.001.
[CV 1/3; 36/50] END C=100.0, class_weight=None, gamma=0.0001, tol=0.01;, score=0.952 total time=   0.0s
[CV 3/3; 40/50] START C=100.0, class_weight=balanced, gamma=0.0001, tol=0.001...
[CV 2/3; 34/50] END C=1e-06, class_weight=balanced, gamma=1000.0, tol=0.01;, score=0.098 total time=   0.2s
[CV 2/3; 37/50] START C=1000000.0, class_weight=balanced, gamma=1e-07, tol=0.1..
[CV 1/3; 40/50] END C=100.0, class_weight=balanced, gamma=0.0001, tol=0.001;, score=0.952 total time=   0.0s
[CV 3/3; 41/50] START C=0.01, class_weight=None, gamma=0.001, tol=0.001.........
[CV 1/3; 38/50] END C=0.1, class_weight=balanced, gamma=10.0, tol=0.1;, score=0.098 total time=   0.2s
[CV 3/3; 38/50] START C=0.1, class_weight=balanced, gamma=10.0, tol=0.1.........
[CV 3/3; 34/50] END C=1e-06, class_weight=balanced, gamma=1000.0, tol=0.01;, score=0.098 total time=   0.2s
[CV 2/3; 41/50] START C=0.01, class_weight=None, gamma=0.001, tol=0.001.........
[CV 3/3; 40/50] END C=100.0, class_weight=balanced, gamma=0.0001, tol=0.001;, score=0.952 total time=   0.0s
[CV 3/3; 42/50] START C=1.0, class_weight=None, gamma=1e-06, tol=0.001..........
[CV 2/3; 37/50] END C=1000000.0, class_weight=balanced, gamma=1e-07, tol=0.1;, score=0.957 total time=   0.0s
[CV 1/3; 43/50] START C=0.001, class_weight=balanced, gamma=0.0001, tol=0.01....
[CV 2/3; 39/50] END C=10000.0, class_weight=None, gamma=10000000.0, tol=0.001;, score=0.102 total time=   0.2s
[CV 2/3; 38/50] START C=0.1, class_weight=balanced, gamma=10.0, tol=0.1.........
[CV 1/3; 39/50] END C=10000.0, class_weight=None, gamma=10000000.0, tol=0.001;, score=0.100 total time=   0.2s
[CV 3/3; 43/50] START C=0.001, class_weight=balanced, gamma=0.0001, tol=0.01....
[CV 3/3; 39/50] END C=10000.0, class_weight=None, gamma=10000000.0, tol=0.001;, score=0.102 total time=   0.2s
[CV 2/3; 44/50] START C=100000.0, class_weight=None, gamma=100000.0, tol=0.1....
[CV 3/3; 41/50] END C=0.01, class_weight=None, gamma=0.001, tol=0.001;, score=0.104 total time=   0.2s
[CV 1/3; 44/50] START C=100000.0, class_weight=None, gamma=100000.0, tol=0.1....
[CV 3/3; 38/50] END C=0.1, class_weight=balanced, gamma=10.0, tol=0.1;, score=0.098 total time=   0.2s
[CV 2/3; 43/50] START C=0.001, class_weight=balanced, gamma=0.0001, tol=0.01....
[CV 1/3; 43/50] END C=0.001, class_weight=balanced, gamma=0.0001, tol=0.01;, score=0.199 total time=   0.2s
[CV 1/3; 42/50] START C=1.0, class_weight=None, gamma=1e-06, tol=0.001..........
[CV 2/3; 41/50] END C=0.01, class_weight=None, gamma=0.001, tol=0.001;, score=0.102 total time=   0.2s
[CV 2/3; 40/50] START C=100.0, class_weight=balanced, gamma=0.0001, tol=0.001...
[CV 3/3; 42/50] END C=1.0, class_weight=None, gamma=1e-06, tol=0.001;, score=0.102 total time=   0.2s
[CV 2/3; 42/50] START C=1.0, class_weight=None, gamma=1e-06, tol=0.001..........
[CV 2/3; 38/50] END C=0.1, class_weight=balanced, gamma=10.0, tol=0.1;, score=0.098 total time=   0.2s
[CV 1/3; 41/50] START C=0.01, class_weight=None, gamma=0.001, tol=0.001.........
[CV 2/3; 40/50] END C=100.0, class_weight=balanced, gamma=0.0001, tol=0.001;, score=0.963 total time=   0.1s
[CV 2/3; 46/50] START C=1000.0, class_weight=balanced, gamma=1.0, tol=0.0001....
[CV 3/3; 43/50] END C=0.001, class_weight=balanced, gamma=0.0001, tol=0.01;, score=0.098 total time=   0.2s
[CV 3/3; 44/50] START C=100000.0, class_weight=None, gamma=100000.0, tol=0.1....
[CV 2/3; 44/50] END C=100000.0, class_weight=None, gamma=100000.0, tol=0.1;, score=0.102 total time=   0.2s
[CV 1/3; 47/50] START C=10000.0, class_weight=None, gamma=0.001, tol=0.0001.....
[CV 1/3; 44/50] END C=100000.0, class_weight=None, gamma=100000.0, tol=0.1;, score=0.100 total time=   0.2s
[CV 3/3; 46/50] START C=1000.0, class_weight=balanced, gamma=1.0, tol=0.0001....
[CV 2/3; 42/50] END C=1.0, class_weight=None, gamma=1e-06, tol=0.001;, score=0.102 total time=   0.2s
[CV 1/3; 42/50] END C=1.0, class_weight=None, gamma=1e-06, tol=0.001;, score=0.292 total time=   0.2s
[CV 3/3; 45/50] START C=1.0, class_weight=None, gamma=1000000.0, tol=0.01.......
[CV 1/3; 46/50] START C=1000.0, class_weight=balanced, gamma=1.0, tol=0.0001....
[CV 2/3; 43/50] END C=0.001, class_weight=balanced, gamma=0.0001, tol=0.01;, score=0.098 total time=   0.2s
[CV 2/3; 45/50] START C=1.0, class_weight=None, gamma=1000000.0, tol=0.01.......
[CV 1/3; 41/50] END C=0.01, class_weight=None, gamma=0.001, tol=0.001;, score=0.299 total time=   0.2s[CV 1/3; 47/50] END C=10000.0, class_weight=None, gamma=0.001, tol=0.0001;, score=0.972 total time=   0.1s
[CV 1/3; 45/50] START C=1.0, class_weight=None, gamma=1000000.0, tol=0.01.......

[CV 1/3; 48/50] START C=1.0, class_weight=None, gamma=10000.0, tol=0.1..........
[CV 2/3; 46/50] END C=1000.0, class_weight=balanced, gamma=1.0, tol=0.0001;, score=0.102 total time=   0.2s
[CV 3/3; 47/50] START C=10000.0, class_weight=None, gamma=0.001, tol=0.0001.....
[CV 3/3; 44/50] END C=100000.0, class_weight=None, gamma=100000.0, tol=0.1;, score=0.102 total time=   0.2s
[CV 2/3; 47/50] START C=10000.0, class_weight=None, gamma=0.001, tol=0.0001.....
[CV 3/3; 46/50] END C=1000.0, class_weight=balanced, gamma=1.0, tol=0.0001;, score=0.102 total time=   0.2s
[CV 1/3; 49/50] START C=100.0, class_weight=balanced, gamma=1e-08, tol=0.0001...
[CV 3/3; 47/50] END C=10000.0, class_weight=None, gamma=0.001, tol=0.0001;, score=0.975 total time=   0.1s
[CV 3/3; 50/50] START C=1000000.0, class_weight=None, gamma=100.0, tol=0.001....
[CV 2/3; 47/50] END C=10000.0, class_weight=None, gamma=0.001, tol=0.0001;, score=0.982 total time=   0.1s
[CV 2/3; 45/50] END C=1.0, class_weight=None, gamma=1000000.0, tol=0.01;, score=0.102 total time=   0.2s
[CV 3/3; 49/50] START C=100.0, class_weight=balanced, gamma=1e-08, tol=0.0001...
[CV 2/3; 50/50] START C=1000000.0, class_weight=None, gamma=100.0, tol=0.001....
[CV 3/3; 45/50] END C=1.0, class_weight=None, gamma=1000000.0, tol=0.01;, score=0.102 total time=   0.2s
[CV 2/3; 48/50] START C=1.0, class_weight=None, gamma=10000.0, tol=0.1..........
[CV 1/3; 48/50] END C=1.0, class_weight=None, gamma=10000.0, tol=0.1;, score=0.100 total time=   0.2s
[CV 1/3; 45/50] END C=1.0, class_weight=None, gamma=1000000.0, tol=0.01;, score=0.100 total time=   0.2s
[CV 1/3; 50/50] START C=1000000.0, class_weight=None, gamma=100.0, tol=0.001....
[CV 2/3; 49/50] START C=100.0, class_weight=balanced, gamma=1e-08, tol=0.0001...
[CV 1/3; 46/50] END C=1000.0, class_weight=balanced, gamma=1.0, tol=0.0001;, score=0.202 total time=   0.2s
[CV 3/3; 48/50] START C=1.0, class_weight=None, gamma=10000.0, tol=0.1..........
[CV 1/3; 49/50] END C=100.0, class_weight=balanced, gamma=1e-08, tol=0.0001;, score=0.472 total time=   0.2s
[CV 3/3; 50/50] END C=1000000.0, class_weight=None, gamma=100.0, tol=0.001;, score=0.102 total time=   0.2s
[CV 2/3; 48/50] END C=1.0, class_weight=None, gamma=10000.0, tol=0.1;, score=0.102 total time=   0.2s
[CV 2/3; 50/50] END C=1000000.0, class_weight=None, gamma=100.0, tol=0.001;, score=0.102 total time=   0.2s
[CV 3/3; 49/50] END C=100.0, class_weight=balanced, gamma=1e-08, tol=0.0001;, score=0.182 total time=   0.2s
[CV 1/3; 50/50] END C=1000000.0, class_weight=None, gamma=100.0, tol=0.001;, score=0.100 total time=   0.2s
[CV 3/3; 48/50] END C=1.0, class_weight=None, gamma=10000.0, tol=0.1;, score=0.102 total time=   0.2s
[CV 2/3; 49/50] END C=100.0, class_weight=balanced, gamma=1e-08, tol=0.0001;, score=0.197 total time=   0.2s
Best parameters found:  {'tol': 0.0001, 'gamma': 0.001, 'class_weight': None, 'C': 10000.0}
Best score:  0.9760712298274902
Best estimator:  SVC(C=10000.0, gamma=0.001, tol=0.0001)
Time taken: 4.22 seconds

Now let’s tackle the problems we discussed earlier…

Neither compute nor memory constrained

The model was trained in 4 seconds, which is pretty fast
However, the dataset is small, so we are not memory constrained
The model is not so complex, so we are not compute constrained either
So in this case we only used dask to parallelise the search, but we could have used scikit-learn alone
But what if we had a larger dataset or a more complex model?

Memory constrained, but not compute constrained

Here we have a case where the dataset is too large to fit in memory, but there is enough compute power to train the model
It makes sense to use Parquet and Dask DataFrames to load the data in chunks, but this may not be enough
A cool solution is Dask’s dask_ml Incremental class, which can train models on chunks of data
It starts training the model on many hyper-parameters on a small amount of data, and then only continues training those models that seem to be performing well
The commands is dask_ml.model_selection.IncrementalSearchCV()
There are some new variations of this method that are worth checking out. Here is the documentation
This strategy is based on scikit-learn’s partial_fit method. More information here

Incremental Search

Let’s see an example with 2 million possible hyperparameter combinations and a dataset with 100 million samples (!)
Combined, X and y will take up about 16 GB of memory
We will use the make_classification function from dask_ml.datasets to create the dataset

import time
from dask_ml.datasets import make_classification
X, y = make_classification(n_samples=100000000, n_features=20,
                           chunks=100000, random_state=0)

# Create the model
from sklearn.linear_model import SGDClassifier
model = SGDClassifier(tol=1e-3, penalty='elasticnet', random_state=0)

# Parameters we want to search through
params = {'alpha': np.logspace(-2, 1, num=1000),
          'l1_ratio': np.linspace(0, 1, num=1000),
          'average': [True, False]}

# Perform the search
from dask_ml.model_selection import IncrementalSearchCV
search = IncrementalSearchCV(model, params, random_state=0)

start_time = time.time()
search.fit(X, y, classes=[0, 1])
end_time = time.time()

# Calculate the elapsed time
elapsed_time = end_time - start_time

# Print the best parameters, best score, and the time taken
print("Best parameters found: ", search.best_params_)
print("Best score: ", search.best_score_)
print("Best estimator: ", search.best_estimator_)
print(f"Time taken: {elapsed_time:.2f} seconds")

Best parameters found:  {'l1_ratio': 0.7117117117117117, 'average': False, 'alpha': 0.02860595535175742}
Best score:  0.66976
Best estimator:  SGDClassifier(alpha=0.02860595535175742, l1_ratio=0.7117117117117117,
              penalty='elasticnet', random_state=0)
Time taken: 51.71 seconds

Incremental Search

It took less than one minute to train the model, which is pretty fast considering the size of the dataset
My computer has 16 GB of RAM, so I would not be able to load the dataset in memory
But the model runs fine, and I was using my computer normally
Now let’s see another example…

Compute constrained, but not memory constrained

Here we have a case where the model is too complex to train in a reasonable amount of time
Or the models require specialised hardware like GPU
The best class for this case is HyperbandSearchCV, which is a hyperparameter search algorithm that is based on the Hyperband algorithm
In a nutshell, this algorithm is easy to use, has strong mathematical motivation and performs remarkably well

from dask_ml.model_selection import HyperbandSearchCV
from dask_ml.datasets import make_classification
from sklearn.linear_model import SGDClassifier

X, y = make_classification(chunks=20)
est = SGDClassifier(tol=1e-3)
param_dist = {'alpha': np.logspace(-4, 0, num=1000),
              'loss': ['hinge', 'log_loss', 'modified_huber', 'squared_hinge'],
              'average': [True, False]}

start_time = time.time()
search = HyperbandSearchCV(est, param_dist)
search.fit(X, y, classes=np.unique(y))
end_time = time.time()

# Calculate the elapsed time
elapsed_time = end_time - start_time
print("Best parameters found: ", search.best_params_)
print("Best score: ", search.best_score_)
print("Best estimator: ", search.best_estimator_)
print(f"Time taken: {elapsed_time:.2f} seconds")

Best parameters found:  {'loss': 'modified_huber', 'average': False, 'alpha': 0.07427982482564918}
Best score:  0.85
Best estimator:  SGDClassifier(alpha=0.07427982482564918, loss='modified_huber')
Time taken: 1.62 seconds

Compute and memory constrained

This is the worst-case scenario, where you have a large dataset and a complex model
In this case, you can use a combination of the strategies we discussed earlier
- Use Dask DataFrames to load the data in chunks
- Use the HyperbandSearchCV class to search for the best hyperparameters
Apart from this, you can always use cloud computing services like AWS, Google Cloud, Azure…
… or pray to the machine learning gods 😂

More Dask with AutoML 🤖

TPOT + Dask

Here we will use TPOT, which is a Python library that automatically creates and optimises machine learning pipelines using genetic programming
TPOT is built on top of scikit-learn and uses a similar syntax
There are several articles about how to optimise AutoML algorithms and quickly find the best model for your data, so I won’t go into too much detail here (but you can check out this article for more information)
You should have TPOT installed in your venv-dask environment, but if you don’t, you can install it with:

conda install -c conda-forge tpot=0.12.2

I’m using Python 3.10.18 for this example (as suggested in the previous lecture), but your mileage may vary

Using TPOT

Let’s see an example using the load_digits dataset

import time
from tpot import TPOTClassifier
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

# Load the digits dataset
digits = load_digits()

# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(digits.data, digits.target,
                                                    train_size=0.75, test_size=0.25)

# Create the TPOTClassifier object
start_time = time.time()
tpot = TPOTClassifier(generations=5, population_size=20, verbosity=2, random_state=42)

# Fit the model
tpot.fit(X_train, y_train)
end_time = time.time()
elapsed_time = end_time - start_time

# Print the score
print(tpot.score(X_test, y_test))
print(f"Time taken: {elapsed_time:.2f} seconds")

Generation 1 - Current best internal CV score: 0.9821836706595072
                                                                             
Generation 2 - Current best internal CV score: 0.9821836706595072
                                                                             
Generation 3 - Current best internal CV score: 0.9821836706595072
                                                                             
Generation 4 - Current best internal CV score: 0.9821836706595072
                                                                              
Generation 5 - Current best internal CV score: 0.9829299187663499
                                                                              
Best pipeline: KNeighborsClassifier(Normalizer(input_matrix, norm=l1), n_neighbors=2, p=2, weights=distance)
0.9911111111111112
Time taken: 240.45 seconds

Using TPOT with Dask

Now let’s see an example using Dask
It is really easy to use Dask with TPOT
I mean it! Just add use_dask=True to the TPOTClassifier object and you are good to go 😊

import time
from tpot import TPOTClassifier
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

# Load the digits dataset
digits = load_digits()

# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(digits.data, digits.target,
                                                    train_size=0.75, test_size=0.25)

# Create the TPOTClassifier object
start_time = time.time()
tpot = TPOTClassifier(generations=5, population_size=20, 
                      verbosity=2, random_state=42, use_dask=True)

# Fit the model
tpot.fit(X_train, y_train)
end_time = time.time()
elapsed_time = end_time - start_time

# Print the score
print(tpot.score(X_test, y_test))
print(f"Time taken: {elapsed_time:.2f} seconds")

Generation 5 - Current best internal CV score: 0.985895635412364
                                                                              
Best pipeline: KNeighborsClassifier(GaussianNB(input_matrix), n_neighbors=2, p=2, weights=distance)
0.9844444444444445
Time taken: 32.35 seconds

Time series forecasting with Prophet

Let’s see another example using the Prophet library
Prophet is a forecasting tool that is open source and maintained by Facebook
It is designed for analyzing time series data that display patterns on different time scales
It is particularly good for data that has multiple seasonality with changing trends
The library is built on top of pystan, which is a Python interface to Stan, a probabilistic programming language
If you installed the venv-dask environment, you should already have Prophet installed

Large datasets are not the only type of scaling challenge teams run into
In this example we will focus once more on model complexity
In the words of Sean Taylor and Ben Letham, the authors of the Prophet library:
- “In most realistic settings, a large number of forecasts will be created, necessitating efficient, automated means of evaluating and comparing them, as well as detecting when they are likely to be performing poorly. When hundreds or even thousands of forecasts are made, it becomes important to let machines do the hard work of model evaluation and comparison while efficiently using human feedback to fix performance problems.”

Using Prophet

We will use Prophet and Dask together to parallise the diagnostics stage of research
It does not attempt to parallise the training of the model itself (which is actually quite fast to begin with)

import pandas as pd
from prophet import Prophet

We will values represent the log daily page views for Peyton Manning’s wikipedia page

df = pd.read_csv(
    'https://raw.githubusercontent.com/facebook/prophet/master/examples/example_wp_log_peyton_manning.csv',
    parse_dates=['ds']
)
df.head()

	ds	y
0	2007-12-10	9.590761
1	2007-12-11	8.519590
2	2007-12-12	8.183677
3	2007-12-13	8.072467
4	2007-12-14	7.893572

Using Prophet

Let’s plot the data and fit the model
No Dask here, just Prophet

df.plot(x='ds', y='y')

m = Prophet(daily_seasonality=False)
m.fit(df)

Using Prophet

And we can make a forecast. Again, no Dask here

future = m.make_future_dataframe(periods=365)
forecast = m.predict(future)
m.plot(forecast)

Using Prophet with Dask

Now let’s use Dask to parallelise the diagnostics stage of the research
Prophet includes a prophet.diagnostics.cross_validation function method, which uses simulated historical forecasts to provide some idea of a model’s quality
This is done by selecting cutoff points in the history, and for each of them fitting the model using data only up to that cutoff point
We can then compare the forecasted values to the actual values

Using Prophet with Dask

Let’s then use Dask to parallelise the diagnostics stage of the research

import prophet.diagnostics
df_cv = prophet.diagnostics.cross_validation(
    m, initial="730 days", period="180 days", horizon="365 days",
    parallel="dask"
)
df_cv.head()

Conclusion 🎉

Summary

Today we discussed how to parallelise data analysis tasks with Dask and AutoML
We learned about Dask Clusters and how to set up a Dask Client
We discussed the types of problems data scientists face when scaling their models and datasets
- No constraints
- Compute constrained
- Memory constrained
- Compute and memory constrained
We also saw how AutoML tools like TPOT can be used to speed up the process of model selection and hyperparameter tuning
And we discussed how to use Prophet and Dask together to parallelise the diagnostics stage of research
I hope you enjoyed the lecture and learned something new today! 😊

Next class

Next class we will discuss how to deal with environments and, mainly, how to use containers to manage your projects
We will discuss the basics of Docker and how to use it to create and manage containers
We will also discuss how to use Docker to create reproducible environments for your projects
I hope to see you there! 🚀

QTM 350 - Data Science Computing

Hello again! 😊 How’s everything?

Brief recap of last class 📚

Parallel computing

Today’s agenda 📅

Lecture outline

Dask Clients and Clusters 🌐

What is a Dask Cluster?

Workers and schedulers

What is a Dask Cluster?

Workers and schedulers

Virtual environment

Setting up a Dask Client

Dask Client Dashboard

Dask Client Dashboard

Testing the Dask Client

Testing the Dask Client

Testing the Dask Client

Dask ML 🤖

Dimensions of Scale

Addressing the Challenges

Challenge 1: Scaling Model Size

Challenge 2: Scaling Data Size

What is Dask ML?

AutoML 🤖

What is AutoML?

What is Dask ML?

Dask ML and scikit-learn

Dask ML and scikit-learn

Now let’s tackle the problems we discussed earlier…

Neither compute nor memory constrained

Memory constrained, but not compute constrained

Incremental Search

Incremental Search

Compute constrained, but not memory constrained

Compute and memory constrained

More Dask with AutoML 🤖

TPOT + Dask

Using TPOT

Using TPOT with Dask

Time series forecasting with Prophet

Using Prophet

Using Prophet

Using Prophet

Using Prophet with Dask

Using Prophet with Dask

Conclusion 🎉

Summary

Next class

Thank you! 🙏

See you next time! 🚀

Hello again! 😊
How’s everything?