wobbegong.core package

Submodules

wobbegong.core.utils module

wobbegong.core.utils.compress_and_write(f, data_bytes, compression='zlib')

Use zlib to compress and write data to file.

Parameters:

• f – File writer object.

• data_bytes (bytes) – Data to write (in bytes).

• compression (Literal['lz4', 'zlib']) – Compression method to use, either ‘lz4’ or ‘zlib’ (default).

Return type:

int

Returns:

Length of the compressed data written.

wobbegong.core.utils.dump_list_of_vectors(columns, types, filepath, compression='zlib')

Write a list of vectors to disk.

Parameters:

• columns (list) – List of column vectors (arrays) to write.

• types (list[str]) – List identifying the type for each column.

• filepath (str) – Path to write the data.

• compression (Literal['lz4', 'zlib']) – Compression method to use, either ‘lz4’ or ‘zlib’ (default).

Return type:

list[int]

Returns:

List of compressed sizes for each column.

wobbegong.core.utils.dump_matrix(x, filepath, type_str, chunk_size=10000, num_threads=1, compression='zlib')

Uses mattress to initialize the matrix and iterate over rows.

Calculates stats and writes compressed rows.

Parameters:

• x – Matrix object. Refer to the mattress package for all supported matrix input types.

• filepath (str) – Path to write.

• type_str (str) – Wobbegong type string of the matrix data.

• chunk_size (int) – Number of rows to read per chunk. Defaults to 10000.

• num_threads (int) – Number of threads for stats calculation. Defaults to 1.

• compression (Literal['lz4', 'zlib']) – Compression method to use, either ‘lz4’ or ‘zlib’ (default).

Return type:

dict

Returns:

Dictionary containing matrix statistics and file byte offsets.

wobbegong.core.utils.get_byte_order()

Return byte order of the machine.

Return type:

str

Returns:

‘little_endian’ or ‘big_endian’.

wobbegong.core.wobbegongify module

wobbegong.core.wobbegongify.wobbegongify(x, path, compression='zlib')

wobbegong.core.wobbegongify.wobbegongify(x, path, compression='zlib')

wobbegong.core.wobbegongify.wobbegongify(x, path, compression='zlib')

wobbegong.core.wobbegongify.wobbegongify(x, path, compression='zlib')

wobbegong.core.wobbegongify.wobbegongify(x, path, compression='zlib')

wobbegong.core.wobbegongify.wobbegongify(x, path, compression='zlib')

wobbegong.core.wobbegongify.wobbegongify(x, path, compression='zlib')

Convert an object to the wobbegong format.

Parameters:

• x (Any) – Object to save to disk.

• path (str) – Path to store object.

• compression (Literal['lz4', 'zlib']) – Compression method to use, either ‘lz4’ or ‘zlib’ (default).

Return type:

None

Module contents