• Typical programs operate lines sequentially:

# Import packages
import numpy as np

# Define an array of numbers
foo = np.array([0, 1, 2, 3, 4, 5])

# Define a function that squares numbers
def bar(x):
    return x * x

# Loop over each element and perform an action on it
for element in foo:

        # Print the result of bar
        print(bar(element))

0
1
4
9
16
25

• Say we have a number of input files, like .jpg images, that we want to perform the same actions on, like rotate by 180 degrees and convert to a different format
• We can define a function that takes a file as input and performs these actions, then map it on a list of files

# Import python image library functions
from PIL import Image

from matplotlib.pyplot import imshow
%matplotlib inline

• Let’s read an image file and display it:

im = Image.open( './data/kings_cross.jpg' )
# Display image
im 

• Parallel computing can be much faster than serial computing
• These problems are essentially independent and share no information between them
• The joblib module makes it simple to run these steps together with a single command
• This workflow is limited to running on a single computer (or compute node) since there is no mechanism to communicate outside

• Install joblib and numpy if you haven’t done so yet
  • !pip install joblib numpy
• Compare the time of the serial and parallel versions of the following function:

def square(x):
    return x**2

# Create a large array to process
numbers = np.arange(1000000)

# Sequential version
%timeit [square(x) for x in numbers]

• Then try the parallel version:

from joblib import Parallel, delayed

# Parallel version
%timeit Parallel(n_jobs=4)(delayed(square)(x) for x in numbers)

• What did you find? Is the parallel version faster?
• Appendix 01

• Dask arrays support most of the NumPy functions
• They are lazy, that is, they do not compute anything until you ask for it

a[:10, 5] # first 10 rows of the 6th column

• We use the .compute() method to compute the result

a[:10, 5].compute()

array([-1.01717283, -0.38495348,  0.4420003 ,  0.42915814, -0.22395892,
       -0.17129497,  0.84975211, -0.45828641, -1.58461724,  0.12297606])

• Dask Array supports many common NumPy operations including:
• Basic arithmetic and scalar mathematics (+, *, exp, log, etc)
• Reductions along axes (sum(), mean(), std())
• Tensor operations and matrix multiplication (tensordot)
• Array slicing and basic indexing
• Axis reordering and transposition

a.sum().compute()

505.74604200460675

• Let’s compare a similar operation in NumPy:

size = 100000000
np_arr = np.random.random(size)
%timeit np_result = np.sqrt(np_arr) + np.sin(np_arr)

542 ms ± 17.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

• And in Dask:

da_arr = da.random.random(size, chunks='auto') 
%timeit da_result = da.sqrt(da_arr) + da.sin(da_arr)

878 μs ± 22.3 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

• Since the Dask dataframe we are using has a time series structure, we can use the resample method to aggregate the data by a time period
• Let’s resample the data by 1 hour and calculate the mean of the x and y columns

df[["x", "y"]].resample("1h").mean().head(3)

• We can also use the rolling method to calculate a rolling mean of the data

df[["x", "y"]].rolling(window="24h").mean().head()

• Install dask if you haven’t done so yet
  • !pip install dask
• Create a Dask array with 1 million random numbers and calculate the mean of the square root of the array
• Compare the time it takes to run the same operation in NumPy
• Appendix 02

• dask-sql is a library that allows you to run SQL queries on Dask DataFrames
• It is built on top of Apache Calcite, a SQL interpreter
• It is still under development, but it already has many features
• You can install it with pip install dask-sql
• A dask_sql.Context is the Python equivalent to a SQL database,
• It serves as an interface to register all tables and functions used in SQL queries, as well as execute the queries themselves
• A single Context is created and used for the duration of a Python script or notebook

from dask_sql import Context
c = Context()

• Once a Context has been created, there are many ways to register tables in it
• The simplest way to do this is through the create_table method
• Supported input types include Pandas DataFrames, Dask DataFrames, remote datasets (like on Amazon S3), and more

# Register and persist a dask table
from dask.datasets import timeseries
ddf = timeseries()
c.create_table("dask", ddf, persist=True)

• Now we can run SQL queries on the timeseries table

c.sql("SELECT AVG(x) FROM dask").compute()

• You can of course combine all three libraries!

# Perform a multi-column sort
res = c.sql("""
    SELECT * FROM dask ORDER BY name ASC, id DESC, x ASC
""")

# Now do some follow groupby aggregations
res.groupby("name").agg({"x": "sum", "y": "mean"}).compute()

• .csv is very common in data science (and for good reasons)
• Pandas reads and writes .csv files very well, but it is not the best option for large files
• It may need several gigabytes of memory to read a large file, as it reads the entire file into memory
• Dask provides a much more efficient way to read and write .csv files
• Let’s split our dataset

df = dask.datasets.timeseries()
df

import os
import datetime

if not os.path.exists('data'):
    os.mkdir('data')

def name(i):
    """ Provide date for filename given index

    Examples
    --------
    >>> name(0)
    '2000-01-01'
    >>> name(10)
    '2000-01-11'
    """
    return str(datetime.date(2000, 1, 1) + i * datetime.timedelta(days=1))

df.to_csv('data/*.csv', name_function=name);

• We now have many CSV files in our data directory, one for each day in the month of January 2000
• Each CSV file holds timeseries data for that day
• We can read all of them as one logical dataframe using the dd.read_csv function

df = dd.read_csv('data/2000-*-*.csv')
df

• Let’s do a simple computation

%timeit df.groupby('name').x.mean().compute()

1.34 s ± 451 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

• Although .csv files are nice, new formats like Parquet are gaining popularity
• Data are stored by column rather than by row, allowing for efficient querying of specific columns without reading entire rows
• It can be used with various programming languages and data processing frameworks
• Achieves up to 75% reduction in file size compared to uncompressed formats

df.to_parquet('data/2000-01.parquet', engine='pyarrow')

• Now we can read the parquet file

df = dd.read_parquet('data/2000-01.parquet', columns=['name', 'x'], engine='pyarrow')
df

• Let’s do the same computation as before

%timeit df.groupby('name').x.mean().compute()

163 ms ± 1.17 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

• We can even visualise the computation graph

@dask.delayed
def generate_data(size):
    return np.random.rand(size)

@dask.delayed
def transform_data(data):
    return np.sqrt(data) + np.sin(data)

@dask.delayed
def aggregate_data(data):
    return {
        'mean': np.mean(data),
        'std': np.std(data),
        'max': np.max(data)
    }

# Compare execution
sizes = [1000000, 2000000, 3000000]

# Dask execution
dask_results = []
for size in sizes:
    data = generate_data(size)
    transformed = transform_data(data)
    stats = aggregate_data(transformed)
    dask_results.append(stats)

%timeit dask.compute(*dask_results)

37.9 ms ± 349 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)

dask.visualize(*dask_results)

• Parallel computing can be a powerful tool, but it is not always the best solution
• So here are some times from the creators of Dask themselves:
• Start small: NumPy, pandas, Scikit-learn may have faster functions for what you’re trying to do. Sometimes just by changing the data format to Parquet you can get a huge speedup
• Sampling: If you have a large dataset, try working with a sample of the data first. Do you really need all those TBs of data?
• Load data with Dask: If you have a large dataset, load it with Dask from the start. It’s much easier to scale up than to scale down
• Use .compute and .persist sparingly: These functions can be expensive, so use them only when you need to
• Break up computations into many pieces: This will allow Dask to parallelise the computation faster
• More tips can be found here

• Here is the solution to the exercise:

def square(x):
    return x**2

# Create a large array to process
numbers = np.arange(1000000)

# Sequential version
%timeit [square(x) for x in numbers]

60.3 ms ± 683 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)

from joblib import Parallel, delayed

# Parallel version
%timeit Parallel(n_jobs=4)(delayed(square)(x) for x in numbers)

2.39 s ± 33.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Back to exercise

• Create a Dask array with 1 million random numbers and calculate the mean of the square root of the array
• Compare the time it takes to run the same operation in NumPy

size = 1000000
np_arr = np.random.random(size)
%timeit np_result = np.sqrt(np_arr).mean()

451 μs ± 60.5 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

da_arr = da.random.random(size, chunks=100000)
%timeit da_result = da.sqrt(da_arr).mean().compute()

3.72 ms ± 24 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Back to exercise