Dryad: Distributed data-parallel programs from sequential building blocks

Authors: 
Isard, M; Budiu, M; Yu, Y; A Birrell, D
Author: 
Isard, M
Budiu, M
Yu, Y
Fetterly, D
Birrell, A

Dryad is a general-purpose distributed execution engine for
coarse-grain data-parallel applications. A Dryad applica-
tion combines computational “vertices” with communica-
tion “channels” to form a dataflow graph. Dryad runs the
application by executing the vertices of this graph on a set of
available computers, communicating as appropriate through
files, TCP pipes, and shared-memory FIFOs.
The vertices provided by the application developer are
quite simple and are usually written as sequential programs
with no thread creation or locking. Concurrency arises from
Dryad scheduling vertices to run simultaneously on multi-
ple computers, or on multiple CPU cores within a computer.
The application can discover the size and placement of data
at run time, and modify the graph as the computation pro-
gresses to make efficient use of the available resources.
Dryad is designed to scale from powerful multi-core sin-
gle computers, through small clusters of computers, to data
centers with thousands of computers. The Dryad execution
engine handles all the difficult problems of creating a large
distributed, concurrent application: scheduling the use of
computers and their CPUs, recovering from communication
or computer failures, and transporting data between ver-
tices.

Year: 
2007
Venue: 
EuroSys 2007
URL: 
http://research.microsoft.com/en-us/projects/dryad/eurosys07.pdf
Citations: 
0
Citations range: 
n/a
AttachmentSize
Isard2007DryadDistributeddataparallelprogramsfromsequential.pdf508.06 KB