Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
Greenplum
Annotated Deep Learning Paper Implementations
提交
7d3c6fcd
A
Annotated Deep Learning Paper Implementations
项目概览
Greenplum
/
Annotated Deep Learning Paper Implementations
大约 1 年 前同步成功
通知
6
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
A
Annotated Deep Learning Paper Implementations
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
7d3c6fcd
编写于
12月 28, 2020
作者:
V
Varuna Jayasiri
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
lstm chunk
上级
814fd565
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
8 addition
and
8 deletion
+8
-8
labml_nn/hypernetworks/hyper_lstm.py
labml_nn/hypernetworks/hyper_lstm.py
+2
-1
labml_nn/lstm/__init__.py
labml_nn/lstm/__init__.py
+6
-7
未找到文件。
labml_nn/hypernetworks/hyper_lstm.py
浏览文件 @
7d3c6fcd
...
...
@@ -31,6 +31,7 @@ class HyperLSTMCell(Module):
self
.
w_x
=
nn
.
ParameterList
([
nn
.
Parameter
(
torch
.
zeros
(
hidden_size
,
input_size
))
for
_
in
range
(
4
)])
self
.
layer_norm
=
nn
.
ModuleList
([
nn
.
LayerNorm
(
hidden_size
)
for
_
in
range
(
4
)])
self
.
layer_norm_c
=
nn
.
LayerNorm
(
hidden_size
)
def
__call__
(
self
,
x
:
torch
.
Tensor
,
h
:
torch
.
Tensor
,
c
:
torch
.
Tensor
,
...
...
@@ -69,7 +70,7 @@ class HyperLSTMCell(Module):
c_next
=
f
*
c
+
i
*
g
# $$h_t = o_t \odot \tanh(c_t)$$
h_next
=
o
*
torch
.
tanh
(
c_next
)
h_next
=
o
*
torch
.
tanh
(
self
.
layer_norm_c
(
c_next
)
)
return
h_next
,
c_next
,
rhn_h
,
rhn_c
...
...
labml_nn/lstm/__init__.py
浏览文件 @
7d3c6fcd
...
...
@@ -55,8 +55,6 @@ class LSTMCell(Module):
def
__init__
(
self
,
input_size
:
int
,
hidden_size
:
int
):
super
().
__init__
()
self
.
hidden_size
=
hidden_size
# These are the linear layer to transform the `input` and `hidden` vectors.
# One of them doesn't need a bias since we add the transformations.
...
...
@@ -68,17 +66,18 @@ class LSTMCell(Module):
def
__call__
(
self
,
x
:
torch
.
Tensor
,
h
:
torch
.
Tensor
,
c
:
torch
.
Tensor
):
# We compute the linear transformations for $i_t$, $f_t$, $g_t$ and $o_t$
# using the same linear layers.
# Each layer produces an output of 4 times the `hidden_size` and we split them later
ifgo
=
self
.
hidden_lin
(
h
)
+
self
.
input_lin
(
x
)
# Each layer produces an output of 4 times the `hidden_size` and we split them
ifgo
=
ifgo
.
chunk
(
4
,
dim
=-
1
)
# $$i_t = \sigma\big(lin_{xi}(x_t) + lin_{hi}(h_{t-1})\big)$$
i
=
torch
.
sigmoid
(
ifgo
[
:,
:
self
.
hidden_size
])
i
=
torch
.
sigmoid
(
ifgo
[
0
])
# $$f_t = \sigma\big(lin_{xf}(x_t) + lin_{hf}(h_{t-1})\big)$$
f
=
torch
.
sigmoid
(
ifgo
[
:,
self
.
hidden_size
:
self
.
hidden_size
*
2
])
f
=
torch
.
sigmoid
(
ifgo
[
1
])
# $$g_t = \tanh\big(lin_{xg}(x_t) + lin_{hg}(h_{t-1})\big)$$
g
=
torch
.
tanh
(
ifgo
[
:,
self
.
hidden_size
*
2
:
self
.
hidden_size
*
3
])
g
=
torch
.
tanh
(
ifgo
[
2
])
# $$o_t = \sigma\big(lin_{xo}(x_t) + lin_{ho}(h_{t-1})\big)$$
o
=
torch
.
sigmoid
(
ifgo
[
:,
self
.
hidden_size
*
3
:
self
.
hidden_size
*
4
])
o
=
torch
.
sigmoid
(
ifgo
[
3
])
# $$c_t = f_t \odot c_{t-1} + i_t \odot g_t$$
c_next
=
f
*
c
+
i
*
g
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录