EFA's SRD transport moves CUDA buffers ~40x faster than tuned TCP, ~13x faster through a shuffle, and ~2x faster using PDS-H Q9 as a benchmark. Why is there a performance difference between SRD and TCP, and why does that difference shrink as workload complexity grows?
cuDF-Polars can scale a familiar Polars query across multiple GPUs with only a few configuration changes
Pinned memory changes both the cost of each transfer and the behavior of the pipeline. It pays an upfront allocation cost, but can reduce spill overhead, lower memory pressure, and improve end-to-end runtime
Spilling makes larger-than-VRAM GPU analytics possible, but memory movement is not free. For longer-running workflows or repeated queries, using pinned host memory can materially reduce memory movement overhead and decrease overall execution time.
Building software for enterprises does not just mean more error checking (though it definitely does include that!), it means understanding a bit more about operations and IT. Each enterprise/organization has their own idea on how networking, security, authentication, and authorization are implemented; the variety can throw a big wrench in the reasonable assumptions often made in software tools deployed in those environments. It can be challenging to generalize these environments even with experiential knowledge (painful scars) and they can be even more challenging to test.
This past summer I had the opportunity to work with Min Ragan-Kelley and Matthew Rocklin on delivering a tutorial at the scientific computing conference, SciPy 2016, in Austin, Texas. We set out to teach folks generally about parallel computing in the context of data analysis and not necessarily about any one tool. That is, focusing on core concepts rather than a specific framework. There is something strangely visceral when you are first learning about distributed computation and different hostnames pop up when executing a simple map across the cluster; and to that end, we wanted to give students access to a cluster capable of doing significant work -- something more than a toy. The tutorial was well received and all the content is publicly available:
In my previous post, I described different scenarios for bootstrapping Python on a multi-node cluster. I offered a general solution using Anaconda for cluster management and solution using a custom conda env deployed with Knit.
In the past few months we've seen a number of posts about Dask. For those unfamiliar with it, Dask is an out-of-core parallel framework for data analysis. Some of the more recent examples (1, 2, 3) have demonstrated Dask's distributed capabilities -- leveraging not just multi-core architectures, but also multi-node clusters. We need a way to launch Dask workers on many machines in our cluster. In a small cluster we might do this by manually SSH-ing into many machines, using a job scheduler like SGE, or using the dec2 tool to provision and bootstrap on ec2. However, for larger clusters this approach breaks down, especially when the cluster is simultaneously running many parallel frameworks like Hadoop, Spark, Impala, etc. In this case we typically use a cluster resource manager like YARN to start and stop jobs on the cluster and to manage their execution environments. In this post, I demonstrate a toy example using the YARN resource manager.
Things generally worked out