Custom Parallel Workflows dask graphs <3 for loops
tl;dr: We motivate the expansion of parallel programming beyond big collections. We discuss the usability custom of dask graphs.
Recent Parallel Work Focuses on Big Collections
Parallel databases, Spark, and Dask collections all provide large distributed
collections that handle parallel computation for you. You put data into the
collection, program with a small set of operations like
the collections handle the parallel processing. This idea has become so
popular that there are now a dozen projects promising big and friendly Pandas
This is good. These collections provide usable, high-level interfaces for a large class of common problems.
However, many workloads are too complex for these collections. Workloads might be complex either because they come from sophisticated algorithms (as we saw in a recent post on SVD) or because they come from the real world, where problems tend to be messy.
In these cases I tend to see people do two things
- Fall back to
MPIor some other explicit form of parallelism
- Perform mental gymnastics to fit their problem into Spark using a clever choice of keys. These cases often fail to acheive much speedup.
Direct Dask Graphs
Historically I’ve recommended the manual construction of dask graphs in these cases. Manual construction of dask graphs lets you specify fairly arbitrary workloads that then use the dask schedulers to execute in parallel. The dask docs hold the following example of a simple data processing pipeline:
Feedback from users is that this is interesting and powerful but that programming directly in dictionaries is not inutitive, doesn’t integrate well with IDEs, and is prone to error.
To create the same custom parallel workloads using normal-ish Python code we
use the dask.do function.
do function turns any normal Python function into a delayed version that
adds to a dask graph. The
do function lets us rewrite the computation above
The explicit function calls here don’t perform work directly; instead they build up a dask graph which we can then execute in parallel with our choice of scheduler.
Example: Nested Cross Validation
I sat down with a Machine learning student, Gabriel
Krummenacher and worked to parallelize a
small code to do nested cross validation. Below is a comparison of a
sequential implementation that has been parallelized using
You can safely skip reading this code in depth. The take-away is that it’s somewhat involved but that the addition of parallelism is light.
The parallel version runs about four times faster on my notebook. Disclaimer: The sequential version presented here is just a downgraded version of the parallel code, hence why they look so similar. This is available on github.
So the result of our normal imperative-style for-loop code is a fully parallelizable dask graph. We visualize that graph below.
Is this a useful interface? It would be great if people could try this out
and generate feedback on
For more information on
dask.do see the
dask imperative documentation.
blog comments powered by Disqus