未验证 提交 504db4f5 编写于 作者: G Ghost Screaming 提交者: GitHub

Fix paddle save for multi-processing (#49657)

* Fix bug of reduce_sum op. When input.numel() > INT32_MAX, its result
is wrong.

* Remove climits.

* Fix bug of paddle.save. It may cause bug for saving sharded optimizer
state_dict() in parallel.
上级 b7d44eb9
...@@ -777,7 +777,7 @@ def save(obj, path, protocol=4, **configs): ...@@ -777,7 +777,7 @@ def save(obj, path, protocol=4, **configs):
# 2. save object # 2. save object
dirname = os.path.dirname(path) dirname = os.path.dirname(path)
if dirname and not os.path.exists(dirname): if dirname and not os.path.exists(dirname):
os.makedirs(dirname) os.makedirs(dirname, exist_ok=True)
elif not _is_memory_buffer(path): elif not _is_memory_buffer(path):
raise ValueError( raise ValueError(
"only supports saving objects to file and `BytesIO`, but got {}".format( "only supports saving objects to file and `BytesIO`, but got {}".format(
...@@ -853,7 +853,7 @@ def _legacy_save(obj, path, protocol=2): ...@@ -853,7 +853,7 @@ def _legacy_save(obj, path, protocol=2):
# 2. save object # 2. save object
dirname = os.path.dirname(path) dirname = os.path.dirname(path)
if dirname and not os.path.exists(dirname): if dirname and not os.path.exists(dirname):
os.makedirs(dirname) os.makedirs(dirname, exist_ok=True)
if isinstance(obj, dict): if isinstance(obj, dict):
saved_obj = _build_saved_state_dict(obj) saved_obj = _build_saved_state_dict(obj)
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册