<p><ahref="https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb"><imgalt="Open In Colab"src="https://colab.research.google.com/assets/colab-badge.svg"></a><ahref="https://www.comet.ml/labml/diffuse/view/FknjSiKWotr8fgZerpC1sV1cy/panels"><imgalt="Open In Comet"src="https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model"></a></p>
<p>This trains a DDPM based model on CelebA HQ dataset. You can find the download instruction in this <ahref="https://forums.fast.ai/t/download-celeba-hq-dataset/45873/3">discussion on fast.ai</a>. Save the images inside <ahref="#dataset_path"><codeclass="highlight"><span></span><spanclass="n">data</span><spanclass="o">/</span><spanclass="n">celebA</span></code>
folder</a>.</p>
<p>The paper had used a exponential moving average of the model with a decay of <spanclass="katex"><spanaria-hidden="true"class="katex-html"><spanclass="base"><spanclass="strut"style="height:0.64444em;vertical-align:0em;"></span><spanclass="mord">0.9999</span></span></span></span>. We have skipped this for simplicity.</p>
<p><ahref="https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb"><imgalt="Open In Colab"src="https://colab.research.google.com/assets/colab-badge.svg"></a><ahref="https://www.comet.ml/labml/diffuse/1260757bcd6148e084ad3a46c38ac5c4?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step"><imgalt="Open In Comet"src="https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model"></a></p>
<p><ahref="https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb"><imgalt="Open In Colab"src="https://colab.research.google.com/assets/colab-badge.svg"></a><ahref="https://www.comet.ml/labml/diffuse/view/FknjSiKWotr8fgZerpC1sV1cy/panels"><imgalt="Open In Comet"src="https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model"></a></p>
<p>This is a <ahref="https://pytorch.org">PyTorch</a> implementation/tutorial of the paper <ahref="https://papers.labml.ai/paper/2006.11239">Denoising Diffusion Probabilistic Models</a>.</p>
<p>In simple terms, we get an image from data and add noise step by step. Then We train a model to predict that noise at each step and use the model to generate images.</p>
<p>The following definitions and derivations show how this works. For details please refer to <ahref="https://papers.labml.ai/paper/2006.11239">the paper</a>.</p>
<p>This file implements the loss calculation and a basic sampling method that we use to generate images during training.</p>
<p>Here is the <ahref="unet.html">UNet model</a> that gives <spanclass="katex"><spanaria-hidden="true"class="katex-html"><spanclass="base"><spanclass="strut"style="height:1em;vertical-align:-0.25em;"></span><spanclass="mord"><spanclass="mord coloredeq eqbg"style=""><spanclass="mord"style=""><spanclass="mord coloredeq eqbo"style=""><spanclass="mord mathnormal"style="">ϵ</span></span><spanclass="msupsub"><spanclass="vlist-t vlist-t2"><spanclass="vlist-r"><spanclass="vlist"style="height:0.33610799999999996em;"><spanstyle="top:-2.5500000000000003em;margin-right:0.05em;"><spanclass="pstrut"style="height:2.7em;"></span><spanclass="sizing reset-size6 size3 mtight"style=""><spanclass="mord mathnormal mtight"style="margin-right:0.02778em">θ</span></span></span></span><spanclass="vlist-s"></span></span><spanclass="vlist-r"><spanclass="vlist"style="height:0.15em;"><span></span></span></span></span></span></span></span><spanclass="mord coloredeq eqbm"style=""><spanclass="mopen"style="">(</span><spanclass="mord coloredeq eqbv"style=""><spanclass="mord"style=""><spanclass="mord mathnormal"style="">x</span><spanclass="msupsub"><spanclass="vlist-t vlist-t2"><spanclass="vlist-r"><spanclass="vlist"style="height:0.2805559999999999em;"><spanstyle="top:-2.5500000000000003em;margin-left:0em;margin-right:0.05em;"><spanclass="pstrut"style="height:2.7em;"></span><spanclass="sizing reset-size6 size3 mtight"style=""><spanclass="mord mtight coloredeq eqbx"style=""><spanclass="mord mathnormal mtight"style="">t</span></span></span></span></span><spanclass="vlist-s"></span></span><spanclass="vlist-r"><spanclass="vlist"style="height:0.15em;"><span></span></span></span></span></span></span></span><spanclass="mpunct"style="">,</span><spanclass="mspace"style="margin-right:0.16666666666666666em"></span><spanclass="mord coloredeq eqbx"style=""><spanclass="mord mathnormal"style="">t</span></span><spanclass="mclose"style="">)</span></span></span></span></span></span> and <ahref="experiment.html">training code</a>. <ahref="evaluate.html">This file</a> can generate samples and interpolations from a trained model.</p>
<p><ahref="https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb"><imgalt="Open In Colab"src="https://colab.research.google.com/assets/colab-badge.svg"></a><ahref="https://www.comet.ml/labml/diffuse/1260757bcd6148e084ad3a46c38ac5c4?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step"><imgalt="Open In Comet"src="https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model"></a></p>
<p><ahref="https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb"><imgalt="Open In Colab"src="https://colab.research.google.com/assets/colab-badge.svg"></a><ahref="https://www.comet.ml/labml/diffuse/view/FknjSiKWotr8fgZerpC1sV1cy/panels"><imgalt="Open In Comet"src="https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model"></a></p>
<p>This is a <ahref="https://pytorch.org">PyTorch</a> implementation/tutorial of the paper <ahref="https://papers.labml.ai/paper/2006.11239">Denoising Diffusion Probabilistic Models</a>.</p>
<p>In simple terms, we get an image from data and add noise step by step. Then We train a model to predict that noise at each step and use the model to generate images.</p>
<p>Here is the <ahref="https://nn.labml.ai/diffusion/ddpm/unet.html">UNet model</a> that predicts the noise and <ahref="https://nn.labml.ai/diffusion/ddpm/experiment.html">training code</a>. <ahref="https://nn.labml.ai/diffusion/ddpm/evaluate.html">This file</a> can generate samples and interpolations from a trained model.</p>
<p>Here is the <ahref="https://nn.labml.ai/diffusion/ddpm/unet.html">UNet model</a> that predicts the noise and <ahref="https://nn.labml.ai/diffusion/ddpm/experiment.html">training code</a>. <ahref="https://nn.labml.ai/diffusion/ddpm/evaluate.html">This file</a> can generate samples and interpolations from a trained model. </p>
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/activations/fta/experiment.ipynb)
[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model)](https://www.comet.ml/labml/fta/69be11f83693407f82a86dcbb232bcfe?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&viewId=rlJOpXDGtL8zbkcX66R77P5me&xAxis=step)
This is a [PyTorch](https://pytorch.org) implementation/tutorial of
[Fuzzy Tiling Activations: A Simple Approach to Learning Sparse Representations Online](https://papers.labml.ai/paper/aca66d8edc8911eba3db37f65e372566).
...
...
@@ -54,9 +57,6 @@ FTA uses this to create soft boundaries between bins.
[Here's a simple experiment](experiment.html) that uses FTA in a transformer.
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/activations/fta/experiment.ipynb)
[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model)](https://www.comet.ml/labml/fta/69be11f83693407f82a86dcbb232bcfe?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&viewId=rlJOpXDGtL8zbkcX66R77P5me&xAxis=step)
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/activations/fta/experiment.ipynb)
[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model)](https://www.comet.ml/labml/fta/69be11f83693407f82a86dcbb232bcfe?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&viewId=rlJOpXDGtL8zbkcX66R77P5me&xAxis=step)
Here we train a transformer that uses [Fuzzy Tiling Activation](index.html) in the
We use it for a language model and train it on Tiny Shakespeare dataset
...
...
@@ -14,9 +17,6 @@ for demonstration.
However, this is probably not the ideal task for FTA, and we
believe FTA is more suitable for modeling data with continuous variables.
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/activations/fta/experiment.ipynb)
[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model)](https://www.comet.ml/labml/fta/69be11f83693407f82a86dcbb232bcfe?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&viewId=rlJOpXDGtL8zbkcX66R77P5me&xAxis=step)
@@ -28,7 +28,6 @@ Here's a notebook for training a Capsule Network on MNIST dataset.
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/capsule_networks/mnist.ipynb)
[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model)](https://www.comet.ml/labml/capsule-networks/reports/capsule-networks)
@@ -10,8 +10,6 @@ This is an annotated PyTorch code to classify MNIST digits with PyTorch.
This paper implements the experiment described in paper
[Dynamic Routing Between Capsules](https://papers.labml.ai/paper/1710.09829).
[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=mnist)](https://www.comet.ml/labml/capsule-networks/reports/capsule-networks)
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb)
[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model)](https://www.comet.ml/labml/diffuse/view/FknjSiKWotr8fgZerpC1sV1cy/panels)
This is a [PyTorch](https://pytorch.org) implementation/tutorial of the paper
Here is the [UNet model](unet.html) that gives $\textcolor{lightgreen}{\epsilon_\theta}(x_t, t)$ and
[training code](experiment.html).
[This file](evaluate.html) can generate samples and interpolations from a trained model.
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb)
[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model)](https://www.comet.ml/labml/diffuse/1260757bcd6148e084ad3a46c38ac5c4?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step)
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb)\n",
"[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model)](https://www.comet.ml/labml/diffuse/1260757bcd6148e084ad3a46c38ac5c4?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step)\n",
"[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model)](https://www.comet.ml/labml/diffuse/view/FknjSiKWotr8fgZerpC1sV1cy/panels)\n",
# [Denoising Diffusion Probabilistic Models (DDPM)](index.html) training
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb)
[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model)](https://www.comet.ml/labml/diffuse/view/FknjSiKWotr8fgZerpC1sV1cy/panels)
This trains a DDPM based model on CelebA HQ dataset. You can find the download instruction in this
[discussion on fast.ai](https://forums.fast.ai/t/download-celeba-hq-dataset/45873/3).
Save the images inside [`data/celebA` folder](#dataset_path).
The paper had used a exponential moving average of the model with a decay of $0.9999$. We have skipped this for
simplicity.
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb)
[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model)](https://www.comet.ml/labml/diffuse/1260757bcd6148e084ad3a46c38ac5c4?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step)
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb)
[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model)](https://www.comet.ml/labml/diffuse/view/FknjSiKWotr8fgZerpC1sV1cy/panels)
This is a [PyTorch](https://pytorch.org) implementation/tutorial of the paper
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/deep_norm/experiment.ipynb)
[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=deep_norm&file=model)](https://www.comet.ml/labml/deep-norm/61d817f80ff143c8825fba4aacd431d4?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step)
This is a [PyTorch](https://pytorch.org) implementation of
the DeepNorm from the paper
[DeepNet: Scaling Transformers to 1,000 Layers](https://papers.labml.ai/paper/2203.00555).
...
...
@@ -66,10 +69,6 @@ Where $N$ is the number of layers in the encoder and $M$ is the number of layers
Refer to [the paper](https://papers.labml.ai/paper/2203.00555) for derivation.
[Here is an experiment implementation](experiment.html) that uses DeepNorm.
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/deep_norm/experiment.ipynb)
[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=deep_norm&file=model)](https://www.comet.ml/labml/deep-norm/61d817f80ff143c8825fba4aacd431d4?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step)
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/deep_norm/experiment.ipynb)\n",
"[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=deep_norm&file=colab)](https://www.comet.ml/labml/deep-norm/61d817f80ff143c8825fba4aacd431d4?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step)\n",
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/deep_norm/experiment.ipynb)
[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=deep_norm&file=experiment)](https://www.comet.ml/labml/deep-norm/61d817f80ff143c8825fba4aacd431d4?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step)