• Parallel computing is a type of computation in which many calculations or processes are carried out simultaneously
• Python has several libraries that allow you to parallelise your code
  • We discussed the joblib library, which is a simple and effective way to parallelise your code
  • But we spent most of our time discussing dask, which is currently the de facto standard for parallel computing in Python
• We also saw that parallel computing is not a panacea: it can be hard to implement and may not always lead to performance improvements
• But when it works, it works great! 🚀

• Today we will continue our discussion on parallel computing
• We will focus on parallelising data analysis tasks with Dask ML
• More specifically, we will discuss how to parallelise the training of machine learning models, and how to use automated machine learning (AutoML) tools to speed up the process
• We will also learn about Dask Clusters, which allow you to scale your computations across multiple machines (or just one!)

• A Dask Cluster is a collection of Dask workers that can be used to parallelise your computations

• In plain English, a Dask Cluster is a group of computational engines (cores, GPUs, servers, etc) that work together to solve a problem

• Workers provide two functions:

  • Compute tasks as directed by the scheduler, the scheduler being the brain of the cluster
  • Store and serve computed results to other workers or clients

• Workers are the reason why lazy evaluation speeds up computations

• A simple example of workers interacting with a scheduler can help explain how lazy evaluation works:

Scheduler -> Eve: Compute a <- multiply(3, 5)!
Eve -> Scheduler: I've computed a and am holding on to it! 
Scheduler -> Frank: Compute b <- add(a, 7)!
Frank: You will need a. Eve has a.
Frank -> Eve: Please send me a.
Eve -> Frank: Sure. a is 15!
Frank -> Scheduler: I've computed b and am holding on to it!

• How do workers know what to do?
  • The scheduler assigns tasks to workers based on their availability
  • Workers can be on the same machine or on different machines
  • Workers can be CPUs or GPUs
• Dask workers save their data as a Python dictionary, which is then sent to the scheduler as a thread in the concurrent.futures library
• These processes can automatically restart and scale up without any intervention from the user
• The optimal number of workers depends on the size of the data and the complexity of the computations, but often the default configuration is sufficient
• In an adaptive cluster, you set the minimum and maximum number of workers and let the cluster add and remove workers as needed

Challenge 1: Scaling Model Size

• Model size: the number of parameters in a model
• If your models become more complex, you need more computational resources to train them
• Under this scaling challenge tasks like model training, prediction, or evaluation steps will (eventually) complete, they just take too long
• You’ve become compute bound
• You can continue to use your current algorithms and libraries (pandas, numpy, scikit-learn, etc), but you need to scale them up

Challenge 2: Scaling Data Size

• Data size: the number of samples in your dataset
• There are cases where datasets grow larger than RAM (shown along the horizontal axis)
• Under this scaling challenge, even loading the data into numpy or pandas becomes impossible
• You’ve become memory bound
• In this case, you can use a different file format (parquet, Dask DataFrame, etc) together with algorithms that can handle large datasets

• Dask ML is a scalable machine learning library built on top of Dask
• It provides parallel implementations of many popular machine learning algorithms and libraries, such as scikit-learn, XGBoost, LightGBM, TensorFlow, and PyTorch
• Dask ML is built on top of Dask Arrays and DataFrames, allowing you to scale your machine learning workflows to large datasets
• You can use Dask ML to do many tasks, such as model selection, model evaluation, and, most importantly, hyperparameter tuning
• And you can also use it together with automated machine learning (AutoML) tools to speed up the process
• The main advantage of using Dask ML + AutoML together is that you can quickly move from data preprocessing to model evaluation without having to worry about the details of the machine learning algorithms
• In a sense, this is a tool to democratise machine learning

• First, let’s install Dask ML:

!pip install dask-ml

• And then import the necessary modules, mainly scikit-learn:

import numpy as np
from dask.distributed import LocalCluster, Client
import joblib
from sklearn.datasets import load_digits
from sklearn.model_selection import RandomizedSearchCV
from sklearn.svm import SVC

• The load_digits dataset is a well-known dataset in machine learning, containing 1797 8x8 pixel images of handwritten digits
• The main task here is to predict the digit from the image

• Let’s estimate the model with a simple grid search
• param_space: A list of settings to try out for the model
• C: Controls how much to punish mistakes
• gamma: Defines how far the influence of a single example reaches
• tol: Tells the model when to stop trying to improve
• class_weight: Options for handling imbalanced data
• This is pretty standard scikit-learn code, but with a twist: we are using joblib.parallel_backend('dask') to parallelise the search
• This will distribute the search across the workers in the Dask cluster (and we don’t have to worry about it!)
• The RandomizedSearchCV object will try out 50 different combinations of hyperparameters and return the best one

Neither compute nor memory constrained

• The model was trained in 2.68 seconds, which is pretty fast
• The dataset is small, so we are not memory constrained
• The model is not complex, so we are not compute constrained
• So in this case we only used dask to parallelise the search, but we could have used scikit-learn alone
• But what if we had a larger dataset or a more complex model?

Memory constrained, but not compute constrained

• Here we have a case where the dataset is too large to fit in memory, but there is enough compute power to train the model
• It makes sense to use Parquet and Dask DataFrames to load the data in chunks, but this may not be enough
• A cool solution is Dask’s dask_ml Incremental class, which can train models on chunks of data
• It starts training the model on many hyper-parameters on a small amount of data, and then only continues training those models that seem to be performing well
• The commands is dask_ml.model_selection.IncrementalSearchCV()
• There are some new variations of this method that are worth checking out. Here is the documentation
• This strategy is based on scikit-learn’s partial_fit method. More information here

• Let’s see an example with 2 million possible hyperparameter combinations and a dataset with 100 million samples (!)
• Combined, X and y will take up about 16 GB of memory
• We will use the make_classification function from dask_ml.datasets to create the dataset

import time
from dask_ml.datasets import make_classification
X, y = make_classification(n_samples=100000000, n_features=20,
                           chunks=100000, random_state=0)

# Create the model
from sklearn.linear_model import SGDClassifier
model = SGDClassifier(tol=1e-3, penalty='elasticnet', random_state=0)

# Parameters we want to search through
params = {'alpha': np.logspace(-2, 1, num=1000),
          'l1_ratio': np.linspace(0, 1, num=1000),
          'average': [True, False]}

# Perform the search
from dask_ml.model_selection import IncrementalSearchCV
search = IncrementalSearchCV(model, params, random_state=0)

start_time = time.time()
search.fit(X, y, classes=[0, 1])
end_time = time.time()

# Calculate the elapsed time
elapsed_time = end_time - start_time

# Print the best parameters, best score, and the time taken
print("Best parameters found: ", search.best_params_)
print("Best score: ", search.best_score_)
print("Best estimator: ", search.best_estimator_)
print(f"Time taken: {elapsed_time:.2f} seconds")

Best parameters found:  {'l1_ratio': 0.7117117117117117, 'average': False, 'alpha': 0.02860595535175742}
Best score:  0.66976
Best estimator:  SGDClassifier(alpha=0.02860595535175742, l1_ratio=0.7117117117117117,
              penalty='elasticnet', random_state=0)
Time taken: 51.71 seconds

• It took less than one minute to train the model, which is pretty fast considering the size of the dataset
• My computer has 16 GB of RAM, so I would not be able to load the dataset in memory
• But the model runs fine, and I was using my computer normally
• Now let’s see another example…

Compute constrained, but not memory constrained

• Here we have a case where the model is too complex to train in a reasonable amount of time
• Or the models require specialised hardware like GPU
• The best class for this case is HyperbandSearchCV, which is a hyperparameter search algorithm that is based on the Hyperband algorithm
• In a nutshell, this algorithm is easy to use, has strong mathematical motivation and performs remarkably well

from dask_ml.model_selection import HyperbandSearchCV
from dask_ml.datasets import make_classification
from sklearn.linear_model import SGDClassifier

X, y = make_classification(chunks=20)
est = SGDClassifier(tol=1e-3)
param_dist = {'alpha': np.logspace(-4, 0, num=1000),
              'loss': ['hinge', 'log_loss', 'modified_huber', 'squared_hinge'],
              'average': [True, False]}

start_time = time.time()
search = HyperbandSearchCV(est, param_dist)
search.fit(X, y, classes=np.unique(y))
end_time = time.time()

# Calculate the elapsed time
elapsed_time = end_time - start_time
print("Best parameters found: ", search.best_params_)
print("Best score: ", search.best_score_)
print("Best estimator: ", search.best_estimator_)
print(f"Time taken: {elapsed_time:.2f} seconds")

Best parameters found:  {'loss': 'modified_huber', 'average': False, 'alpha': 0.07427982482564918}
Best score:  0.85
Best estimator:  SGDClassifier(alpha=0.07427982482564918, loss='modified_huber')
Time taken: 1.62 seconds

Compute and memory constrained

• This is the worst-case scenario, where you have a large dataset and a complex model
• In this case, you can use a combination of the strategies we discussed earlier
  • Use Dask DataFrames to load the data in chunks
  • Use the HyperbandSearchCV class to search for the best hyperparameters
• Apart from this, you can always use cloud computing services like AWS, GCP, or Azure…
• … or pray to the machine learning gods 😂

• In the last part of this lecture, we will discuss automated machine learning (AutoML)
• AutoML is the process of automating the process of applying machine learning to real-world problems
• The main goal, which I personally support, is to move quickly from data preprocessing to model evaluation without having to worry about the details of the machine learning algorithms
• In my experience, the faster we can get the model running, the more time we have to think about the problem and the data
• AutoML tools can be used to automate the process of hyperparameter tuning, model selection, feature engineering, and model evaluation
• There are several tools available for AutoML, such as Auto-sklearn, H2O.ai, and TPOT

• Here we will use TPOT, which is a Python library that automatically creates and optimises machine learning pipelines using genetic programming
• TPOT is built on top of scikit-learn and uses a similar syntax
• There are several articles about how to optimise AutoML algorithms and quickly find the best model for your data, so I won’t go into too much detail here (but you can check out this article for more information)
• First, let’s install TPOT:

!pip install tpot

• I’m using Python 3.9.6 for this example, so your mileage may vary

• Let’s see another example using the Prophet library
• Prophet is a forecasting tool that is open source and maintained by Facebook
• It is designed for analyzing time series data that display patterns on different time scales
• It is particularly good for data that has multiple seasonality with changing trends
• The library is built on top of pystan, which is a Python interface to Stan, a probabilistic programming language
• First, let’s install the library:

!pip install prophet

• Large datasets are not the only type of scaling challenge teams run into
• In this example we will focus on the third type of scaling challenge: model complexity
• In the words of Sean Taylor and Ben Letham, the authors of the Prophet library:
  • “In most realistic settings, a large number of forecasts will be created, necessitating efficient, automated means of evaluating and comparing them, as well as detecting when they are likely to be performing poorly. When hundreds or even thousands of forecasts are made, it becomes important to let machines do the hard work of model evaluation and comparison while efficiently using human feedback to fix performance problems.”

• Next class we will discuss how to deal with environments and, mainly, how to use containers to manage your projects
• We will discuss the basics of Docker and how to use it to create and manage containers
• We will also discuss how to use Docker to create reproducible environments for your projects
• I hope to see you there! 🚀