This package implements classes for delayed array operations, mirroring the Bioconductor package of the same name. It allows BiocPy-based packages to easily inteoperate with delayed arrays from the Bioconductor ecosystem, with focus on serialization to/from file with chihaya/rds2py and entry into tatami-compatible C++ libraries via mattress.
Note
These classes follow a functional paradigm for accessing or setting properties, with further details discussed in functional paradigm section.
Installation
This package is published to PyPI and can be installed via the usual methods:
pip install delayedarray
Quick start
We can create a DelayedArray from any object that respects the seed contract, i.e., has the shape/dtype properties and supports NumPy slicing. For example, a typical NumPy array qualifies:
Each operation just returns a DelayedArray with an increasing stack of delayed operations, without evaluating anything or making any copies. Check out the documentation for more information.
Extracting data
A DelayedArray is typically used by iteratively extracting blocks into memory for further calculations. This “block processing” strategy improves memory efficiency by only realizing the delayed operations for a subset of the data. For example, to iterate over the rows:
grid = delayedarray.chunk_grid(d)dims = (*range(len(d.shape)),)for block in grid.iterate(dimensions=dims): subsets = (*(range(s, e) for s, e in block),) current = delayedarray.extract_dense_array(d, subsets)
Each call to extract_dense_array() yields a NumPy array containing the the specified rows and columns. If the DelayedArray might contain masked values, a NumPy MaskedArray is returned instead; this can be determined by checking whether is_masked(d) returns True.
The above iteration can be simplified with the apply_over_dimension() function, which handles the block coordinate calculations for us. We could also use the apply_over_blocks() function to iterate over arbitrary block shapes, which may be more efficient if the best dimension for iteration is not known.
# To iterate over a single dimension:delayedarray.apply_over_dimension( d, dimension=0, fun=some_user_supplied_function, block_size=block_size,)# To iterate over arbitrary blocks.delayedarray.apply_over_blocks( d, fun=another_user_supplied_function, block_shape=(20, 100),)
Handling sparse data
If the DelayedArray contains sparse data, is_sparse(d) will return True. This allows callers to instead use the extract_sparse_array() function for block processing:
if delayedarray.is_sparse(d): current = delayedarray.extract_sparse_array(d, (*block_coords,))
This returns a SparseNdarray consisting of a tree of sparse vectors for the specified block. Users can retrieve the sparse vectors by inspecting the contents property of the SparseNdarray:
In the one-dimensional case, this is a tuple of two 1-dimensional NumPy arrays storing data about the non-zero elements. The first array contains sorted indices while the secon array contains the associated values. If is_masked(d) returns True, the values will be represented as NumPy MaskedArray objects.
For the two-dimensional case, this is a list of such tuples, with one tuple per column. This is roughly analogous to a compressed sparse column matrix. An entry of the list may also be None, indicating that no non-zero elements are present in that column.
For higher-dimensionals, the tree is a nested list of lists of tuples. Each nesting level corresponds to a dimension; the outermost level contains elements of the last dimension, the next nesting level contains elements of the second-last dimension, and so on, with the indices in the tuple referring to the first dimension. Any list element may be None indicating that the corresponding element of the dimension has no non-zero elements.
In all cases, it is possible for contents to be None, indicating that there are no non-zero elements in the entire array.
The apply_over_* functions can also be instructed to iteratively extract blocks as SparseNdarray objects. This only occurs if the input array is sparse (as specified by is_sparse).
# To iterate over a single dimension:delayedarray.apply_over_dimension( d, dimension=0, fun=some_user_supplied_function, block_size=block_size, allow_sparse=True,)
Other coercions
A DelayedArray can be converted to a (possibly masked) NumPy array with the to_dense_array() function. Similarly, sparse DelayedArrays can be converted to SparseNdarrays with the to_sparse_array() function.
res =Noneif delayedarray.is_sparse(d): res = delayedarray.to_sparse_array(d)else: res = delayedarray.to_dense_array(d)res
More simply, users can just call numpy.array() to realize the delayed operations into a standard NumPy array for consumption. Note that this discards any masking information so should not be called if is_masked() returns True.
import randomsimple = numpy.array([random.random() for _ inrange(50)])type(simple)
numpy.ndarray
Users can also call delayedarray.create_dask_array(), to obtain a dask array that contains the delayed operations:
# Note: requires installation as 'delayedarray[dask]'.dasky = delayedarray.create_dask_array(simple)type(dasky)
dask.array.core.Array
Interoperability with BiocPy packages
The general idea is that DelayedArrays should be a drop-in replacement for NumPy arrays, at least for BiocPy applications. So, for example, we can use a DelayedArray inside a SummarizedExperiment:
One of the main goals of the DelayedArray package is to make it easier for Bioconductor developers to inspect the delayed operations. (See the developer notes for some comments on dask.) For example, we can pull out the “seed” object underlying our DelayedArray instance: