Slides

Dask.array

# NumPy code
import numpy as np
x = np.random.random((1000, 1000))
u, s, v = np.linalg.svd(x.dot(x.T))

# Dask.array code
import dask.array as da
x = da.random.random((100000, 100000), chunks=(1000, 1000))
u, s, v = da.linalg.svd(x.dot(x.T))

Dask.DataFrame

import pandas as pd
df = pd.read_csv('myfile.csv', parse_dates=['timestamp'])
df.groupby(df.timestamp.dt.hour).value.mean()

import dask.dataframe as dd
df = dd.read_csv('hdfs://myfiles.*.csv', parse_dates=['timestamp'])
df.groupby(df.timestamp.dt.hour).value.mean().compute()

But many problems aren't just big arrays and dataframes

The world is messy. Algorithms can be complex.

Fine Grained Code

results = {}

for a in A:
    for b in B:
        if a < b:
            results[a, b] = f(a, b)
        else:
            results[a, b] = g(a, b)

.

Parallelizable, but not a list, dataframe, or array

Fine Grained Code

from dask import delayed, compute

results = {}

for a in A:
    for b in B:
        if a < b:
            results[a, b] = delayed(f)(a, b)  # lazily construct graph
        else:
            results[a, b] = delayed(g)(a, b)  # without structure

results = compute(results)  # trigger all computation

Parallelizable, but not a list, dataframe, or array

Dask APIs Produce Task Graphs

Dask Schedulers Execute Task Graphs

1D-Array

>>> np.ones((15,))
array([ 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

>>> x = da.ones((15,), chunks=(5,))

1D-Array

x = da.ones((15,), chunks=(5,))
x.sum()

ND-Array - Sum

x = da.ones((15, 15), chunks=(5, 5))
x.sum(axis=0)

ND-Array - Transpose

x = da.ones((15, 15), chunks=(5, 5))
x + x.T

ND-Array - Matrix Multiply

x = da.ones((15, 15), chunks=(5, 5))
x.dot(x.T + 1)

ND-Array - Compound Operations

x = da.ones((15, 15), chunks=(5, 5))
x.dot(x.T + 1) - x.mean()

ND-Array - Compound Operations

import dask.array as da
x = da.ones((15, 15), chunks=(5, 5))
y = (x.dot(x.T + 1) - x.mean()).std()

Brief and Incomplete Summary of Parallelism Options

Embarrassingly parallel systems (multiprocessing, joblib)
Big Data collections (MapReduce, Spark, Flink, Database)
Task schedulers (Airflow, Luigi, Celery, Make)

map

output = map(func, data)  # Sequential

pool = multiprocessing.Pool()
output = pool.map(func, data)  # Parallel

Pros
- Easy to install and use in the common case
- Lightweight dependency
Cons
- Data interchange cost
- Not able to handle complex computations

Big Data collections

from pyspark import SparkContext
sc = SparkContext('...')

rdd = sc.parallelize(data)
rdd.map(json.loads).filter(...).groupBy(...).count()

df = spark.read_json(...)
df.groupBy('name').aggregate({'value': 'sum'})

Pros
- Larger set of operations
- Scales nicely on clusters
- Well trusted by enterprise
Cons
- Heavyweight and JVM focused
- Not able to handle complex computations

Setup Distributed Network

$ dask-scheduler
Scheduler listening at tcp://192.168.1.100:8786

$ dask-worker tcp://192.168.1.100:8786
$ dask-worker tcp://192.168.1.100:8786

>>> from dask.distributed import Client
>>> client = Client('tcp://192.168.1.100:8786')

Run locally on your laptop

from dask.distributed import Client
client = Client()  # set up local scheduler and workers

Deploy on cluster resource manager

dask-kubernetes
dask-marathon
dask-yarn
dask-drmaa
dask-ssh
dask-ec2
...

Dask is easy to use and adopt

You already know the API

You already have the dependencies installed

Dask supports Pythonic APIs

NumPy/Pandas/SKLearn protocols and APIs
PEP-3148 concurrent.futures
PEP-492 async/await
Joblib (sklearn's parallelism library)

Dask is lightweight

Single-machine scheduler runs on stdlib only
Distributed scheduler is a Tornado TCP application
Pure Python 2.7+ or 3.4+

Dask overview

High level APIs
- Dask.array = dask + numpy
- Dask.dataframe = dask + pandas
- Machine learning, lists, real time, and others
- Parallelizes custom systems
Low level task scheduler
- Determines when and where to call Python functions
- Works with any Python functions on any Python objects
- Handles data dependencies, locality, data movement, etc..

Example Domain: Machine Learning

Integrate with Scikit-Learn

Create API compatible components like model selection

pipe = Pipeline(steps=[('pca', PCA()),
                       ('logistic', LogisticRegression)])
grid = GridSearchCV(pipe, parameter_grid)

Hijack SKlearn's internal parallel library, joblib

Implement algorithms with dask.array

eXbeta = da.exp(X.dot(beta))
gradient = X.T.dot(eXbeta / (eXbeta + 1) - y)
...

Collaborate with other distributed systems
- Pre-process with dask.dataframe
- Deploy other distributed services (xgboost, tensorflow, ...)
- Pass data from Dask and train with other service
Build custom systems with dask.delayed, concurrent.futures

Example: Convex optimization algorithms with Dask array

for k in range(max_steps):
    Xbeta = X.dot(beta_hat)
    func = ((y - Xbeta)**2).sum()
    gradient = 2 * X.T.dot(Xbeta - y)

    ## Update
    obeta = beta_hat
    beta_hat = beta_hat - step_size * gradient
    new_func = ((y - X.dot(beta_hat))**2).sum()
    beta_hat, func, new_func = dask.compute(beta_hat, func, new_func) # new

    ## Check for convergence
    change = np.absolute(beta_hat - obeta).max()

    if change < tolerance:
        break

Example: Dask + XGBoost

df = dd.read_csv('...')    # load and clean with dask.dataframe
training_data = df[[...]]  # prepare training data and labels
labels = df['clicked']

import dask_xgboost
params = {'objective': 'binary:logistic', 'eta': 0.01, 'max_depth': 16}
bst = dask_xgboost.train(client, params, training_data, labels)  # hand off

>>> bst
<xgboost.core.Booster at 0x7fa1c18c4c18>

Launches XGBoost workers alongside Dask workers
Passes data off and lets XGBoost handle what it does best

dask and xgboost in same workers

Easy to build in ~ 15 hours. Collaboration with Tianqi Chen, Olivier Grisel. dmlc/xgboost #2032

Example: Dask + XGBoost

df = dd.read_csv('...')    # load and clean with dask.dataframe
training_data = df[[...]]  # prepare training data and labels
labels = df['clicked']

import dask_xgboost
params = {'objective': 'binary:logistic', 'eta': 0.01, 'max_depth': 16}
bst = dask_xgboost.train(client, params, training_data, labels)  # hand off

>>> bst
<xgboost.core.Booster at 0x7fa1c18c4c18>

Launches XGBoost workers alongside Dask workers
Passes data off and lets XGBoost handle what it does best

dask and xgboost in same workers

Easy to build in ~ 15 hours. Collaboration with Tianqi Chen, Olivier Grisel. dmlc/xgboost #2032

Example: Gradient-free search

from dask_patternsearch import search

def f(x):
    return x.dot(x)

x0 = np.array([1, 2, 3])
best, results = search(f, x0, stepsize, stopratio=1e-4)

Directs computation based on recent results
Fully live and asynchronous
Algorithm scales to very many dimensions

Dask pattern search example

Implementation by Erik Welch (Continuum)

Thank you -- Questions?

Dask scales the existing Python ecosystem
Dask enables custom applications
Docs: dask.pydata.org -- Github: github.com/dask

You can set it up right now during questions:

 $ conda install dask            # Install with conda
 $ pip install dask[complete]    # or with pip

 >>> from dask.distributed import Client
 >>> client = Client()  # starts "cluster" on your laptop

 >>> futures = client.map(lambda x: x + 1, range(1000))
 >>> total = client.submit(sum, futures)
 >>> total.result()

Comparison to Spark

Reasons to prefer Spark
- More established
- All-in-one framework for clusters
- SQL support and high-level optimizations
- Complements existing JVM infrastructure
Reasons to prefer Dask
- Grows out of existing Python stack
- Familiar to Python users and applications
- Supports more complex computations
- Integrates nicely into existing systems, lightweight

Map	Shuffle	Reduce

Dask: Parallel Computing in Python

Python has a strong analytics stack (NumPy, Pandas)

that is mostly restricted to a single core and RAM

How do we parallelize an existing analytics stack?

of thousands of packages

each with custom algorithms

Sckit-Image: general image analysis

Scikit-Allel: Specialized genomics

Need a parallel computing library

... that is flexible enough

... and light enough

... to parallelize a disparate ecosystem

Outline

Dask.array

Dask.DataFrame

But many problems aren't just big arrays and dataframes

The world is messy. Algorithms can be complex.

Fine Grained Code

Parallelizable, but not a list, dataframe, or array

Fine Grained Code

Parallelizable, but not a list, dataframe, or array

Dask APIs Produce Task Graphs

Dask Schedulers Execute Task Graphs

1D-Array

1D-Array

ND-Array - Sum

ND-Array - Transpose

ND-Array - Matrix Multiply

ND-Array - Compound Operations

ND-Array - Compound Operations

Dask APIs Produce Task Graphs

Dask Schedulers Execute Task Graphs

Dask.array/dataframe/delayed author task graphs

Now we need to run them efficiently

Dask.array/dataframe/delayed author task graphs

Now we need to run them efficiently

Brief and Incomplete Summary of Parallelism Options

map

Big Data collections

This is what I mean by complex

Spark does the following well

Task Schedulers (Airflow, Luigi, Make, ...)

Want a task scheduler (like Airflow, Luigi)

Built for computational loads (like Spark, Flink)

Single Machine Scheduler

Distributed Scheduler

Distributed Network

Setup Distributed Network

Run locally on your laptop

Deploy on cluster resource manager

Distributed Scheduler

Distributed Scheduler

Distributed Scheduler

Distributed Scheduler

Distributed Scheduler

Distributed Scheduler

Distributed Scheduler

Distributed Scheduler

Distributed Scheduler

Distributed Scheduler

Distributed Scheduler

Distributed Scheduler

Distributed Scheduler

Distributed Scheduler

Distributed Scheduler

Dask is easy to use and adopt

You already know the API

You already have the dependencies installed

Dask supports Pythonic APIs

Dask is lightweight

Dask overview

Example Domain: Machine Learning

Example: Convex optimization algorithms with Dask array

Example: Convex optimization algorithms with Dask array

Example: Dask + XGBoost

Example: Dask + XGBoost

Example: Gradient-free search

Engagement

Github issues in the last 48 hours

Thank you -- Questions?