Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
曾经的那一瞬间
Models
提交
ac97d775
M
Models
项目概览
曾经的那一瞬间
/
Models
12 个月 前同步成功
通知
1
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
M
Models
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
ac97d775
编写于
5月 27, 2022
作者:
S
Scott Zhu
提交者:
A. Unique TensorFlower
5月 27, 2022
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Prepare for upcoming keras initializer change.
PiperOrigin-RevId: 451481056
上级
b58f478c
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
11 addition
and
8 deletion
+11
-8
official/projects/roformer/roformer_encoder.py
official/projects/roformer/roformer_encoder.py
+6
-5
official/projects/roformer/roformer_encoder_block.py
official/projects/roformer/roformer_encoder_block.py
+5
-3
未找到文件。
official/projects/roformer/roformer_encoder.py
浏览文件 @
ac97d775
...
...
@@ -19,6 +19,7 @@ import collections
from
absl
import
logging
import
tensorflow
as
tf
from
official.modeling
import
tf_utils
from
official.nlp.modeling
import
layers
from
official.projects.roformer
import
roformer_encoder_block
...
...
@@ -115,7 +116,7 @@ class RoformerEncoder(tf.keras.Model):
embedding_layer_inst
=
layers
.
on_device_embedding
.
OnDeviceEmbedding
(
vocab_size
=
vocab_size
,
embedding_width
=
embedding_width
,
initializer
=
initializer
,
initializer
=
tf_utils
.
clone_initializer
(
initializer
)
,
name
=
'word_embeddings'
)
else
:
embedding_layer_inst
=
embedding_layer
...
...
@@ -125,7 +126,7 @@ class RoformerEncoder(tf.keras.Model):
type_embedding_layer
=
layers
.
on_device_embedding
.
OnDeviceEmbedding
(
vocab_size
=
type_vocab_size
,
embedding_width
=
embedding_width
,
initializer
=
initializer
,
initializer
=
tf_utils
.
clone_initializer
(
initializer
)
,
use_one_hot
=
True
,
name
=
'type_embeddings'
)
type_embeddings
=
type_embedding_layer
(
type_ids
)
...
...
@@ -146,7 +147,7 @@ class RoformerEncoder(tf.keras.Model):
'...x,xy->...y'
,
output_shape
=
hidden_size
,
bias_axes
=
'y'
,
kernel_initializer
=
initializer
,
kernel_initializer
=
tf_utils
.
clone_initializer
(
initializer
)
,
name
=
'embedding_projection'
)
embeddings
=
embedding_projection
(
embeddings
)
else
:
...
...
@@ -171,7 +172,7 @@ class RoformerEncoder(tf.keras.Model):
attention_dropout
=
attention_dropout
,
norm_first
=
norm_first
,
output_range
=
transformer_output_range
,
kernel_initializer
=
initializer
,
kernel_initializer
=
tf_utils
.
clone_initializer
(
initializer
)
,
name
=
'roformer/layer_%d'
%
i
)
transformer_layers
.
append
(
layer
)
data
=
layer
([
data
,
attention_mask
])
...
...
@@ -185,7 +186,7 @@ class RoformerEncoder(tf.keras.Model):
pooler_layer
=
tf
.
keras
.
layers
.
Dense
(
units
=
hidden_size
,
activation
=
'tanh'
,
kernel_initializer
=
initializer
,
kernel_initializer
=
tf_utils
.
clone_initializer
(
initializer
)
,
name
=
'pooler_transform'
)
cls_output
=
pooler_layer
(
first_token_tensor
)
...
...
official/projects/roformer/roformer_encoder_block.py
浏览文件 @
ac97d775
...
...
@@ -15,6 +15,7 @@
"""Roformer TransformerEncoder block layer."""
import
tensorflow
as
tf
from
official.modeling
import
tf_utils
from
official.projects.roformer
import
roformer_attention
...
...
@@ -111,7 +112,8 @@ class RoformerEncoderBlock(tf.keras.layers.Layer):
self
.
_attention_initializer
=
tf
.
keras
.
initializers
.
get
(
attention_initializer
)
else
:
self
.
_attention_initializer
=
self
.
_kernel_initializer
self
.
_attention_initializer
=
tf_utils
.
clone_initializer
(
self
.
_kernel_initializer
)
self
.
_attention_axes
=
attention_axes
def
build
(
self
,
input_shape
):
...
...
@@ -164,7 +166,7 @@ class RoformerEncoderBlock(tf.keras.layers.Layer):
einsum_equation
,
output_shape
=
(
None
,
self
.
_inner_dim
),
bias_axes
=
"d"
,
kernel_initializer
=
self
.
_kernel_initializer
,
kernel_initializer
=
tf_utils
.
clone_initializer
(
self
.
_kernel_initializer
)
,
name
=
"intermediate"
,
**
common_kwargs
)
policy
=
tf
.
keras
.
mixed_precision
.
global_policy
()
...
...
@@ -182,7 +184,7 @@ class RoformerEncoderBlock(tf.keras.layers.Layer):
output_shape
=
(
None
,
hidden_size
),
bias_axes
=
"d"
,
name
=
"output"
,
kernel_initializer
=
self
.
_kernel_initializer
,
kernel_initializer
=
tf_utils
.
clone_initializer
(
self
.
_kernel_initializer
)
,
**
common_kwargs
)
self
.
_output_dropout
=
tf
.
keras
.
layers
.
Dropout
(
rate
=
self
.
_output_dropout
)
# Use float32 in layernorm for numeric stability.
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录