I/O

The library operates in three different modes when reading and writing a dataset. When reading a non-distributed dataset in a parallel environment, the dataset will be by default read by the master thread and sent to all slave threads. You can also define the read as being exclusively local (imagine a local temporary file). When reading a distributed dataset, the read will just return the portion of the given read request that it locally owns. This requires you always to check the amount of data that is returned. In the next section I will provide an example.

When writing a distributed dataset, the library will by default assume that you are writing only the portion of the given window that it actually owns. When writing a non-distributed dataset you can again specify that a write is exclusively local, but otherwise only the master thread will write data, and it is the programmer's job to make sure they have collected the dataset.