utils.compression_helper¶
compression_helper¶
Please Reference ding/ding/utils/compression_helper.py for usage.
get_data_compressor¶
- Overview:
Get the data compressor according to the input name
- Arguments:
name(
str): Name of the compressor, support['lz4', 'zlib', 'none']
- Return:
(
Callable): Corresponding data_compressor, taking input data returning compressed data.
- Example:
>>> compress_fn = get_data_compressor('lz4') >>> compressed_data = compressed(input_data)
dummy_compressor¶
- Overview:
Return input data.
zlib_data_compressor¶
- Overview:
Takes the input compressed data and return the compressed original data (zlib compressor) in binary format.
- Examples:
>>> zlib_data_compressor("Hello") b'x\x9ck`\x99\xca\xc9\x00\x01=\xac\x1e\xa999\xf9S\xf4\x00%L\x04j'
lz4_data_compressor¶
- Overview:
Return the compressed original data (lz4 compressor).The compressor outputs in binary format.
- Examples:
>>> lz4.block.compress(pickle.dumps("Hello")) b'\x14\x00\x00\x00R\x80\x04\x95\t\x00\x01\x00\x90\x8c\x05Hello\x94.'
get_data_decompressor¶
- Overview:
Get the data decompressor according to the input name
- Arguments:
name(
str): Name of the decompressor, support['lz4', 'zlib', 'none']
Note
For all the decompressors, the input of a bytes-like object is required.
- Returns:
(
Callable): Corresponding data_decompressor.
- Examples:
>>> decompress_fn = get_data_decompressor('lz4') >>> origin_data = compressed(compressed_data)
dummy_decompressor¶
- Overview:
Return input data.
zlib_data_decompressor¶
- Overview:
Return the decompressed original data (zlib compressor).
lz4_data_decompressor¶
- Overview:
Return the decompressed original data (lz4 compressor).