GPUs are neat

.

First, an example

Same code

Different Hardware

Different performance profile

work by Peter Entschev blog.dask.org/2019/06/27/single-gpu-cupy-benchmarks

SVD Example

import dask.array
.

rs = dask.array.random.RandomState()
x = rs.random((1000000, 1000), chunks=(10000, 1000))

u, s, v = np.linalg.svd(x)
u, s, v = dask.compute(u, s, v)

A complex algorithm built on the Numpy API

SVD Example

import dask.array
import cupy

rs = dask.array.random.RandomState(RandomState=cupy.random.RandomState)
x = rs.random((1000000, 1000), chunks=(10000, 1000))

u, s, v = np.linalg.svd(x)
u, s, v = dask.compute(u, s, v)

Works today on GPU arrays too

GPUs are good for other things too

>>> import pandas, cudf

>>> %time pandas.read_csv("nyc-taxi-2015-01.csv")
Wall time: 29.2s

>>> %time cudf.read_csv("nyc-taxi-2015-01.csv")
Wall time: 2.12s

$ du -h nyc-taxi-2015-01.csv
1.9G

GPUs are good for other things too

>>> import umap, cuml

>>> %time umap.UMAP(n_neighbors=5, init="spectral").fit_transform(cpu_data)
Wall time: 1min 49s

>>> %time cuml.UMAP(n_neighbors=5, init="spectral").fit_transform(cpu_data)
Wall time: 19.5s

$ du -h my_data
400 MB

We don't have to write CUDA to use a GPU

Rapids, CuPy, Numba provide familiar SciPy APIs around GPUs

.

We need to think about standards

The SciPy ecosystem is expanding

Lots of new faces (welcome!)

• :)
• :-)
• ;)
• B-)

The SciPy ecosystem is expanding

Lots of new domains

• Genomics
• Cyber security
• Genomics
• Text processing

The SciPy ecosystem is expanding

Lots of new technologies

• Multi-core CPUs
• Distributed Clusters
• Accelerators (GPUs, TPUs, ...)
• ARM

The SciPy ecosystem is expanding

Lots of new institutional partners

• Philanthropic foundations
• Fresh startups
• Large technology companies
• .

This is great!

...

This is great!

But also scary!

The ecosystem works because everything fits together

Integrating new components takes care

Otherwise we risk fracturing the ecosystem

History

• 1990's Numarray and Numeric: fractured community
• 2000: Numpy unites the ecosystem
• 2000s: The golden age of compatibility
• 2009: Rise of Pandas, Numpy not able to grow fast enough
• 2010s: Rise of PyData/SciPy, garner attention
• 2015: Deep learning frameworks, large tech companies arrive

• 2019: Numpy, Tensorflow, PyTorch, CuPy, Jax, Sparse, Dask, ...

  Fractured community

History

• 1990's Numarray and Numeric: fractured community
• .
• .
• We're back in the 90's, but with more time pressure
• .
• .

• 2019: Numpy, Tensorflow, PyTorch, CuPy, Jax, Sparse, Dask, ...

  Fractured community

Need to organize to facilitate growth

Carefully, but also quickly

.

Need to organize to facilitate growth

Carefully, but also quickly

Standards enable inter-project coordination

Protocol examples: __iter__

Implement the __iter__ protocol to operate in for loops

class MyObject:
    def __iter__(self):

for x in MyObject():
    ...

Protocol example: __array__

Implement the __array__ protocol to convert to a NumPy array

import matplotlib.pyplot as plt

x = np.array(...)

plt.plot(x)

Protocol example: __array__

Implement the __array__ protocol to convert to a NumPy array

import matplotlib.pyplot as plt

df = pandas.read_csv('myfile.csv')

plt.plot(df.balance)

Protocol example: __array__

Implement the __array__ protocol to convert to a NumPy array

# pandas/core/series.py

class Series:
    def __array__(self):
        return np.array(self._pointer_to_data)

Protocol example: __array__

Implement the __array__ protocol to convert to a NumPy array

import h5py

x = numpy.ones(10000)

h5py.File('myfile.h5')['x'] = x

Protocol example: __array__

Implement the __array__ protocol to convert to a NumPy array

import h5py

df = pandas.read_csv('myfile.csv')

h5py.File('myfile.h5')['x'] = df.balance  # <-- mediate two non-numpy libraries

Protocol example: __array__

Implement the __array__ protocol to convert to a NumPy array

import h5py

df = dask.dataframe.read_csv('myfile.csv')

h5py.File('myfile.h5')['x'] = df.balance  # <-- mediate two non-numpy libraries

Protocol example: __array__

Implement the __array__ protocol to convert to a NumPy array

import h5py

x = dask.array.random.random(1000000)

h5py.File('myfile.h5')['x'] = x

Protocol example: __array__

Implement the __array__ protocol to convert to a NumPy array

import h5py, zarr

x = zarr.create_dataset(...)

h5py.File('myfile.h5')['x'] = x

Protocol example: __array__

Implement the __array__ protocol to convert to a NumPy array

import h5py, zarr

x = h5py.File('myfile.h5')['x']

zarr.create_dataset(...)['x'] = x

Standards let us mix and match technologies

Enabling creative solutions that maintainers didn't foresee

Protocol example: fit/predict

Implement fit/transform/predict to work with Scikit-Learn

from sklearn import ...
from sklearn.cluster import DBScan
pipeline = Pipeline(..., DBScan(), ...)

pipeline.fit(X, y)

Protocol example: fit/predict

Implement fit/transform/predict to work with Scikit-Learn

from sklearn import ...
from umap import UMAP
pipeline = Pipeline(..., UMAP(), ...)

pipeline.fit(X, y)

Protocol example: fit/predict

Implement fit/transform/predict to work with Scikit-Learn

from sklearn import ...
from cuml import UMAP
pipeline = Pipeline(..., UMAP(), ...)

pipeline.fit(X, y)

Protocol example: .ipynb

• Classic Jupyter notebook
• Jupyter Lab
• Nteract
• .
• .
• .

Protocol example: .ipynb

• Classic Jupyter notebook
• Jupyter Lab
• Nteract
• GitHub rendering
• Netflix infrastructure
• .

Protocol example: .ipynb

• Classic Jupyter notebook
• Jupyter Lab
• Nteract
• GitHub rendering
• Netflix infrastructure
• Voilà

Protocol example: __array_function__

Implement __array_function__ to use NumPy functions

import numpy

x = numpy.random.random((10000, 10000))

u, s, v = np.linalg.svd(x)

Protocol example: __array_function__

Implement __array_function__ to use NumPy functions

import dask.array

x = dask.array.random.random((10000, 10000))

u, s, v = np.linalg.svd(x)

Protocol example: __array_function__

Implement __array_function__ to use NumPy functions

import cupy

x = cupy.random.random((10000, 10000))

u, s, v = np.linalg.svd(x)

Protocol example: __array_function__

Implement __array_function__ to use NumPy functions

import cupy, xarray

x = cupy.random.random((10000, 10000))

d = xarray.DataArary(x)

Standards level the playing field for technologies

New technologies can quickly compete

For example an OpenCL Numpy implementation could gain traction quickly

Protocol example: __array_function__

Implement __array_function__ to use NumPy functions

import cupy

x = cupy.random.random((10000, 10000))

u, s, v = np.linalg.svd(x)

Protocol example: __array_function__

Implement __array_function__ to use NumPy functions

import clpy

x = clpy.random.random((10000, 10000))

u, s, v = np.linalg.svd(x)

Protocol example: __array_function__

Implement __array_function__ to use NumPy functions

import clpy, xarray

x = clpy.random.random((10000, 10000))

d = xarray.DataArary(x)

Standards enforce consistency for users

Quickly mix and match to explore new functionality

"Let's just throw sparse arrays in Dask and see what happens"

Standards enforce consistency for users

Quickly mix and match to explore new functionality

"Let's just throw GPU dataframes in Dask and see what happens"

Example API Consumer: Dask Array

Example API Consumer: Dask Array

Example API Consumer: Dask Array

Example API Consumer: Dask DataFrame

Works with either Pandas or RAPIDS cuDF for GPUs

Standards level the playing field for technologies

New developers can quickly engage

PhD students need more opportunities to distract them from research

Pandas and Extension Arrays: internal

Pandas has many custom array-like internal types

• Datetime with timezones
• Categoricals
• Sparse
• Missing values

>>> import pandas as pd
.

>>> pd.Series(["Healthy", "No change", "No change"]).astype("category")
0      Healthy
1    No change
2    No change
dtype: category
Categories (2, object): [Healthy, No change]
.

.

Pandas and Extension Arrays: external

Externalizing that API enables external packages to be Pandas native

• CyberPandas
• GeoPandas
• Text
• Decimal

>>> import pandas as pd
>>> from cyberpandas import IPArray  # <--- External library

>>> arr = IPArray([0, 1, 2, 3])
>>> s = pd.Series(arr)               # <--- Native integration
0    0.0.0.0
1    0.0.0.1
2    0.0.0.2
3    0.0.0.3
dtype: ip                            # <--- Neat!

What would previously be a community fork is now native and integrated

(this works with Dask DataFrame too)

>>> import pandas as pd
>>> from cyberpandas import IPArray

>>> arr = IPArray([0, 1, 2, 3])
>>> s = pd.Series(arr)
0    0.0.0.0
1    0.0.0.1
2    0.0.0.2
3    0.0.0.3
dtype: ip

>>> ds = dask.dataframe.from_pandas(s, npartitions=2)
Dask Series Structure:
npartitions=2
0     ip
2    ...
3    ...
dtype: ip                            # <--- Downstream benefits
Dask Name: from_pandas, 2 tasks

Core projects define clear contracts

Directing external ecosystem growth

What should we do about it?

• Users: Experiment new technologies, share experiences

• Developers:: Build on standards, not closed systems

  Build new things!

• Core Maintainers: Build extension points