This trains a DDPM based model on CelebA HQ dataset. You can find the download instruction in this discussion on fast.ai. Save the images inside data/celebA
folder.
In simple terms, we get an image from data and add noise step by step. Then We train a model to predict that noise at each step and use the model to generate images.
The following definitions and derivations show how this works. For details please refer to the paper.
@@ -297,7 +298,6 @@ M834 80h400000v40h-400000z">This file implements the loss calculation and a basic sampling method that we use to generate images during training.
In simple terms, we get an image from data and add noise step by step. Then We train a model to predict that noise at each step and use the model to generate images.
-
diff --git a/docs/sitemap.xml b/docs/sitemap.xml
index d4c4d3649682b6cf296306fa48edce38891f9030..2cf57f05f55b163fd72df0edb074f9786c13c971 100644
--- a/docs/sitemap.xml
+++ b/docs/sitemap.xml
@@ -372,7 +372,7 @@
https://nn.labml.ai/diffusion/ddpm/unet.html
- 2021-10-24T16:30:00+00:00
+ 2022-06-09T16:30:00+00:00
1.00
@@ -400,7 +400,7 @@
https://nn.labml.ai/diffusion/ddpm/evaluate.html
- 2021-10-24T16:30:00+00:00
+ 2022-06-09T16:30:00+00:00
1.00
diff --git a/labml_nn/activations/fta/__init__.py b/labml_nn/activations/fta/__init__.py
index fd24407e335d35345d52156094ecaad1ea61f090..664d21964f80c972bc53be4867b735cd1435ad39 100644
--- a/labml_nn/activations/fta/__init__.py
+++ b/labml_nn/activations/fta/__init__.py
@@ -8,6 +8,9 @@ summary: >
# Fuzzy Tiling Activations (FTA)
+[](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/activations/fta/experiment.ipynb)
+[](https://www.comet.ml/labml/fta/69be11f83693407f82a86dcbb232bcfe?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&viewId=rlJOpXDGtL8zbkcX66R77P5me&xAxis=step)
+
This is a [PyTorch](https://pytorch.org) implementation/tutorial of
[Fuzzy Tiling Activations: A Simple Approach to Learning Sparse Representations Online](https://papers.labml.ai/paper/aca66d8edc8911eba3db37f65e372566).
@@ -54,9 +57,6 @@ FTA uses this to create soft boundaries between bins.
$$\phi_\eta(z) = 1 - I_{\eta,+} \big( \max(\mathbf{c} - z, 0) + \max(z - \delta - \mathbf{c}, 0) \big)$$
[Here's a simple experiment](experiment.html) that uses FTA in a transformer.
-
-[](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/activations/fta/experiment.ipynb)
-[](https://www.comet.ml/labml/fta/69be11f83693407f82a86dcbb232bcfe?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&viewId=rlJOpXDGtL8zbkcX66R77P5me&xAxis=step)
"""
import torch
diff --git a/labml_nn/activations/fta/experiment.py b/labml_nn/activations/fta/experiment.py
index be740a029781ea53aaf9e28f30b03a2f6de27a17..f5f5378950930bf8ad7d0b806260aee7ed90518b 100644
--- a/labml_nn/activations/fta/experiment.py
+++ b/labml_nn/activations/fta/experiment.py
@@ -7,6 +7,9 @@ summary: >
# [Fuzzy Tiling Activation](index.html) Experiment
+[](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/activations/fta/experiment.ipynb)
+[](https://www.comet.ml/labml/fta/69be11f83693407f82a86dcbb232bcfe?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&viewId=rlJOpXDGtL8zbkcX66R77P5me&xAxis=step)
+
Here we train a transformer that uses [Fuzzy Tiling Activation](index.html) in the
[Feed-Forward Network](../../transformers/feed_forward.html).
We use it for a language model and train it on Tiny Shakespeare dataset
@@ -14,9 +17,6 @@ for demonstration.
However, this is probably not the ideal task for FTA, and we
believe FTA is more suitable for modeling data with continuous variables.
-
-[](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/activations/fta/experiment.ipynb)
-[](https://www.comet.ml/labml/fta/69be11f83693407f82a86dcbb232bcfe?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&viewId=rlJOpXDGtL8zbkcX66R77P5me&xAxis=step)
"""
import copy
diff --git a/labml_nn/capsule_networks/__init__.py b/labml_nn/capsule_networks/__init__.py
index 145052d9f448cf448ed1bdaa1775dfbec8b80d6c..1c0836e37e9276d220bdf1ae6e9e67d0f489d986 100644
--- a/labml_nn/capsule_networks/__init__.py
+++ b/labml_nn/capsule_networks/__init__.py
@@ -28,7 +28,6 @@ Here's a notebook for training a Capsule Network on MNIST dataset.
[](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/capsule_networks/mnist.ipynb)
[](https://app.labml.ai/run/e7c08e08586711ebb3e30242ac1c0002)
-[](https://www.comet.ml/labml/capsule-networks/reports/capsule-networks)
"""
import torch.nn as nn
diff --git a/labml_nn/capsule_networks/mnist.py b/labml_nn/capsule_networks/mnist.py
index 24504e8c44fcfb361d287e7c9e0431205ea6525a..f9eb475f8fa72e4016c2f79f6de6808e8d4718cb 100644
--- a/labml_nn/capsule_networks/mnist.py
+++ b/labml_nn/capsule_networks/mnist.py
@@ -10,8 +10,6 @@ This is an annotated PyTorch code to classify MNIST digits with PyTorch.
This paper implements the experiment described in paper
[Dynamic Routing Between Capsules](https://papers.labml.ai/paper/1710.09829).
-
-[](https://www.comet.ml/labml/capsule-networks/reports/capsule-networks)
"""
from typing import Any
diff --git a/labml_nn/diffusion/ddpm/__init__.py b/labml_nn/diffusion/ddpm/__init__.py
index de1f24acb866e6ddce9d455f56610bca1a036ea0..e4fc9684b483882184d4b9978e4edc25276026d4 100644
--- a/labml_nn/diffusion/ddpm/__init__.py
+++ b/labml_nn/diffusion/ddpm/__init__.py
@@ -8,6 +8,9 @@ summary: >
# Denoising Diffusion Probabilistic Models (DDPM)
+[](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb)
+[](https://www.comet.ml/labml/diffuse/view/FknjSiKWotr8fgZerpC1sV1cy/panels)
+
This is a [PyTorch](https://pytorch.org) implementation/tutorial of the paper
[Denoising Diffusion Probabilistic Models](https://papers.labml.ai/paper/2006.11239).
@@ -156,9 +159,6 @@ training.
Here is the [UNet model](unet.html) that gives $\textcolor{lightgreen}{\epsilon_\theta}(x_t, t)$ and
[training code](experiment.html).
[This file](evaluate.html) can generate samples and interpolations from a trained model.
-
-[](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb)
-[](https://www.comet.ml/labml/diffuse/1260757bcd6148e084ad3a46c38ac5c4?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step)
"""
from typing import Tuple, Optional
diff --git a/labml_nn/diffusion/ddpm/experiment.ipynb b/labml_nn/diffusion/ddpm/experiment.ipynb
index ea7b9ffdfb1b1355b00a84df64e499950b49a5c5..0126ff4f081fb3f10bdeb6a4e9e4d410569eaae9 100644
--- a/labml_nn/diffusion/ddpm/experiment.ipynb
+++ b/labml_nn/diffusion/ddpm/experiment.ipynb
@@ -11,7 +11,7 @@
"source": [
"[](https://github.com/labmlai/annotated_deep_learning_paper_implementations)\n",
"[](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb)\n",
- "[](https://www.comet.ml/labml/diffuse/1260757bcd6148e084ad3a46c38ac5c4?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step)\n",
+ "[](https://www.comet.ml/labml/diffuse/view/FknjSiKWotr8fgZerpC1sV1cy/panels)\n",
"\n",
"## [Denoising Diffusion Probabilistic Models (DDPM)](https://nn.labml.ai/diffusion/ddpm/index.html)\n",
"\n",
@@ -201,7 +201,11 @@
},
{
"cell_type": "markdown",
- "metadata": {},
+ "metadata": {
+ "pycharm": {
+ "name": "#%% md\n"
+ }
+ },
"source": [
"Initializ"
]
@@ -209,7 +213,11 @@
{
"cell_type": "code",
"execution_count": null,
- "metadata": {},
+ "metadata": {
+ "pycharm": {
+ "name": "#%%\n"
+ }
+ },
"outputs": [],
"source": [
"configs.init()"
@@ -282,7 +290,11 @@
{
"cell_type": "code",
"execution_count": null,
- "metadata": {},
+ "metadata": {
+ "pycharm": {
+ "name": "#%%\n"
+ }
+ },
"outputs": [],
"source": []
}
@@ -314,4 +326,4 @@
},
"nbformat": 4,
"nbformat_minor": 4
-}
+}
\ No newline at end of file
diff --git a/labml_nn/diffusion/ddpm/experiment.py b/labml_nn/diffusion/ddpm/experiment.py
index e34d710f6dc9d551ce52cd37a784db24590d65c1..e2681cea370e695b812c8c03fca3d83698f3937d 100644
--- a/labml_nn/diffusion/ddpm/experiment.py
+++ b/labml_nn/diffusion/ddpm/experiment.py
@@ -8,15 +8,15 @@ summary: >
# [Denoising Diffusion Probabilistic Models (DDPM)](index.html) training
+[](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb)
+[](https://www.comet.ml/labml/diffuse/view/FknjSiKWotr8fgZerpC1sV1cy/panels)
+
This trains a DDPM based model on CelebA HQ dataset. You can find the download instruction in this
[discussion on fast.ai](https://forums.fast.ai/t/download-celeba-hq-dataset/45873/3).
Save the images inside [`data/celebA` folder](#dataset_path).
The paper had used a exponential moving average of the model with a decay of $0.9999$. We have skipped this for
simplicity.
-
-[](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb)
-[](https://www.comet.ml/labml/diffuse/1260757bcd6148e084ad3a46c38ac5c4?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step)
"""
from typing import List
diff --git a/labml_nn/diffusion/ddpm/readme.md b/labml_nn/diffusion/ddpm/readme.md
index 378b5af8ed5c5c7414231b117c0e178404853fb9..ff92474fca54518e8fbb35db0cec81452c66a40d 100644
--- a/labml_nn/diffusion/ddpm/readme.md
+++ b/labml_nn/diffusion/ddpm/readme.md
@@ -1,5 +1,8 @@
# [Denoising Diffusion Probabilistic Models (DDPM)](https://nn.labml.ai/diffusion/ddpm/index.html)
+[](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb)
+[](https://www.comet.ml/labml/diffuse/view/FknjSiKWotr8fgZerpC1sV1cy/panels)
+
This is a [PyTorch](https://pytorch.org) implementation/tutorial of the paper
[Denoising Diffusion Probabilistic Models](https://papers.labml.ai/paper/2006.11239).
@@ -11,5 +14,3 @@ Here is the [UNet model](https://nn.labml.ai/diffusion/ddpm/unet.html) that pred
[training code](https://nn.labml.ai/diffusion/ddpm/experiment.html).
[This file](https://nn.labml.ai/diffusion/ddpm/evaluate.html) can generate samples and interpolations
from a trained model.
-
-[](https://app.labml.ai/run/a44333ea251411ec8007d1a1762ed686)
diff --git a/labml_nn/normalization/deep_norm/__init__.py b/labml_nn/normalization/deep_norm/__init__.py
index bc7842fb11351cd05876972a84a566185729cc65..2222d2d4253e6198117905b9d492d7290764282d 100644
--- a/labml_nn/normalization/deep_norm/__init__.py
+++ b/labml_nn/normalization/deep_norm/__init__.py
@@ -7,6 +7,9 @@ summary: >
# DeepNorm
+[](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/deep_norm/experiment.ipynb)
+[](https://www.comet.ml/labml/deep-norm/61d817f80ff143c8825fba4aacd431d4?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step)
+
This is a [PyTorch](https://pytorch.org) implementation of
the DeepNorm from the paper
[DeepNet: Scaling Transformers to 1,000 Layers](https://papers.labml.ai/paper/2203.00555).
@@ -66,10 +69,6 @@ Where $N$ is the number of layers in the encoder and $M$ is the number of layers
Refer to [the paper](https://papers.labml.ai/paper/2203.00555) for derivation.
[Here is an experiment implementation](experiment.html) that uses DeepNorm.
-
-[](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/deep_norm/experiment.ipynb)
-[](https://app.labml.ai/run/ec8e4dacb7f311ec8d1cd37d50b05c3d)
-[](https://www.comet.ml/labml/deep-norm/61d817f80ff143c8825fba4aacd431d4?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step)
"""
from typing import Union, List
diff --git a/labml_nn/normalization/deep_norm/experiment.ipynb b/labml_nn/normalization/deep_norm/experiment.ipynb
index df5f443c9033eeb5a297c6765bbf16349c385934..9dd84b4f2bf0eeeb41cbecbcced7643b2094ca3f 100644
--- a/labml_nn/normalization/deep_norm/experiment.ipynb
+++ b/labml_nn/normalization/deep_norm/experiment.ipynb
@@ -11,7 +11,6 @@
"source": [
"[](https://github.com/labmlai/annotated_deep_learning_paper_implementations)\n",
"[](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/deep_norm/experiment.ipynb)\n",
- "[](https://app.labml.ai/run/ec8e4dacb7f311ec8d1cd37d50b05c3d)\n",
"[](https://www.comet.ml/labml/deep-norm/61d817f80ff143c8825fba4aacd431d4?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step)\n",
"\n",
"## DeepNorm\n",
diff --git a/labml_nn/normalization/deep_norm/experiment.py b/labml_nn/normalization/deep_norm/experiment.py
index ade34a972cde1b6461b5c03149bb8c3c1042361d..845397bf2b248ffd8ac19cbb1ea7933e5401f17e 100644
--- a/labml_nn/normalization/deep_norm/experiment.py
+++ b/labml_nn/normalization/deep_norm/experiment.py
@@ -8,7 +8,6 @@ summary: >
# [DeepNorm](index.html) Experiment
[](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/deep_norm/experiment.ipynb)
-[](https://app.labml.ai/run/ec8e4dacb7f311ec8d1cd37d50b05c3d)
[](https://www.comet.ml/labml/deep-norm/61d817f80ff143c8825fba4aacd431d4?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step)
"""
