# Traverse different rank programs and traverse each op of them,
# Traverse different rank programs and traverse each op of them,
# instantiate communication by process_mapping.
# instantiate communication by process_mapping.
all_process_groups=get_all_process_groups()
all_process_groups=get_all_process_groups()
cur_rank=self._cur_rank
# NOTE: After the implementation of the unified dynamic and static communication group initialization mode in the future, the initialization logic of full mode will be removed because port occupation error may occur.