utils.file_helper¶
file_helper¶
Please Reference ding/ding/utils/file_helper.py for usage.
read_from_ceph¶
- Overview:
Read file from ceph
- Arguments:
path (
str
): File path in ceph, start with"s3://"
- Returns:
(
data
): Deserialized data
read_from_redis¶
- Overview:
Read file from redis
- Arguments:
path (
str
): Dile path in redis, could be a string key
- Returns:
(
data
): Deserialized data
read_from_rediscluster¶
- Overview:
Read file from rediscluster
- Arguments:
path (
str
): Dile path in rediscluster, could be a string key
- Returns:
(
data
): Deserialized data
read_from_mc¶
- Overview:
Read file from memcache, file must be saved by torch.save()
- Arguments:
path (
str
): File path in local system
- Returns:
(
data
): Deserialized data
read_from_file¶
- Overview:
Read file from local file system
- Arguments:
path (
str
): File path in local file system
- Returns:
(
data
): Deserialized data
read_from_path¶
- Overview:
Read file from ceph
- Arguments:
path (
str
): File path in ceph, start with"s3://"
, or use local file system
- Returns:
(
data
): Deserialized data
save_file_ceph¶
- Overview:
Save pickle dumped data file to ceph
- Arguments:
path (
str
): File path in ceph, start with"s3://"
, use file system when notdata (
Any
): Could be dict, list or tensor etc.
save_file_redis¶
- Overview:
Save pickle dumped data file to redis
- Arguments:
path (
str
): File path (could be a string key) in redisdata (
Any
): Could be dict, list or tensor etc.
save_file_rediscluster¶
- Overview:
Save pickle dumped data file to rediscluster
- Arguments:
path (
str
): File path (could be a string key) in redisdata (
Any
): Could be dict, list or tensor etc.
read_file¶
- Overview:
Read file from path
- Arguments:
path (
str
): The path of file to readfs_type (
str
orNone
): The file system type, support{'normal', 'ceph'}
use_lock (
bool
): Whetheruse_lock
is in local normal file system
save_file¶
- Overview:
Save data to file of path
- Arguments:
path (
str
): The path of file to save todata (
object
): The data to savefs_type (
str
orNone
): The file system type, support{'normal', 'ceph'}
use_lock (
bool
): Whetheruse_lock
is in local normal file system
remove_file¶
- Overview:
Remove file
- Arguments:
path (
str
): The path of file you want to removefs_type (
str
orNone
): The file system type, support{'normal', 'ceph'}