Skip to content

A
Annotated Deep Learning Paper Implementations

Greenplum / Annotated Deep Learning Paper Implementations
10 个月 前同步成功

通知 6
Star 0
Fork 0
A
Annotated Deep Learning Paper Implementations

前往新版Gitcode，体验更适合开发者的 AI 搜索 >>

未验证 提交 f8abd6a2 编写于 作者: V Varuna Jayasiri 提交者： GitHub
浏览文件
操作

Conv mixer (#100)

上级 217b56a1
展开全部 隐藏空白更改
内联 并排
Showing with 1413 addition and 16 deletion
docs/conv_mixer/experiment.html 0 → 100644
浏览文件 @ f8abd6a2
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="content-type" content="text/html;charset=utf-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1.0"/>
<meta name="description" content="Train ConvMixer on CIFAR 10"/>
<meta name="twitter:card" content="summary"/>
<meta name="twitter:image:src" content="https://avatars1.githubusercontent.com/u/64068543?s=400&amp;v=4"/>
<meta name="twitter:title" content="Train ConvMixer on CIFAR 10"/>
<meta name="twitter:description" content="Train ConvMixer on CIFAR 10"/>
<meta name="twitter:site" content="@labmlai"/>
<meta name="twitter:creator" content="@labmlai"/>
<meta property="og:url" content="https://nn.labml.ai/conv_mixer/experiment.html"/>
<meta property="og:title" content="Train ConvMixer on CIFAR 10"/>
<meta property="og:image" content="https://avatars1.githubusercontent.com/u/64068543?s=400&amp;v=4"/>
<meta property="og:site_name" content="LabML Neural Networks"/>
<meta property="og:type" content="object"/>
<meta property="og:title" content="Train ConvMixer on CIFAR 10"/>
<meta property="og:description" content="Train ConvMixer on CIFAR 10"/>
<title>Train ConvMixer on CIFAR 10</title>
<link rel="shortcut icon" href="/icon.png"/>
<link rel="stylesheet" href="../pylit.css">
<link rel="canonical" href="https://nn.labml.ai/conv_mixer/experiment.html"/>
<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=G-4V3HC8HBLH"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag() {
dataLayer.push(arguments);
}
gtag('js', new Date());
gtag('config', 'G-4V3HC8HBLH');
</script>
</head>
<body>
<div id='container'>
<div id="background"></div>
<div class='section'>
<div class='docs'>
<p>
<a class="parent" href="/">home</a>
<a class="parent" href="index.html">conv_mixer</a>
</p>
<p>
<a href="https://github.com/labmlai/annotated_deep_learning_paper_implementations/tree/master/labml_nn/conv_mixer/experiment.py">
<img alt="Github"
src="https://img.shields.io/github/stars/labmlai/annotated_deep_learning_paper_implementations?style=social"
style="max-width:100%;"/></a>
<a href="https://twitter.com/labmlai"
rel="nofollow">
<img alt="Twitter"
src="https://img.shields.io/twitter/follow/labmlai?style=social"
style="max-width:100%;"/></a>
</p>
</div>
</div>
<div class='section' id='section-0'>
<div class='docs doc-strings'>
<div class='section-link'>
<a href='#section-0'>#</a>
</div>
<h1>Train a <a href="index.html">ConvMixer</a> on CIFAR 10</h1>
<p>This script trains a ConvMixer on CIFAR 10 dataset.</p>
<p>This is not an attempt to reproduce the results of the paper.
The paper uses image augmentations
present in <a href="https://github.com/rwightman/pytorch-image-models">PyTorch Image Models (timm)</a>
for training. We haven&rsquo;t done this for simplicity - which causes our validation accuracy to drop.</p>
<p><a href="https://app.labml.ai/run/0fc344da2cd011ecb0bc3fdb2e774a3d"><img alt="View Run" src="https://img.shields.io/badge/labml-experiment-brightgreen" /></a></p>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">20</span><span></span><span class="kn">from</span> <span class="nn">labml</span> <span class="kn">import</span> <span class="n">experiment</span>
<span class="lineno">21</span><span class="kn">from</span> <span class="nn">labml.configs</span> <span class="kn">import</span> <span class="n">option</span>
<span class="lineno">22</span><span class="kn">from</span> <span class="nn">labml_nn.experiments.cifar10</span> <span class="kn">import</span> <span class="n">CIFAR10Configs</span></pre></div>
</div>
</div>
<div class='section' id='section-1'>
<div class='docs doc-strings'>
<div class='section-link'>
<a href='#section-1'>#</a>
</div>
<h2>Configurations</h2>
<p>We use <a href="../experiments/cifar10.html"><code>CIFAR10Configs</code></a> which defines all the
dataset related configurations, optimizer, and a training loop.</p>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">25</span><span class="k">class</span> <span class="nc">Configs</span><span class="p">(</span><span class="n">CIFAR10Configs</span><span class="p">):</span></pre></div>
</div>
</div>
<div class='section' id='section-2'>
<div class='docs'>
<div class='section-link'>
<a href='#section-2'>#</a>
</div>
<p>Size of a patch, $p$</p>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">34</span> <span class="n">patch_size</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">2</span></pre></div>
</div>
</div>
<div class='section' id='section-3'>
<div class='docs'>
<div class='section-link'>
<a href='#section-3'>#</a>
</div>
<p>Number of channels in patch embeddings, $h$</p>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">36</span> <span class="n">d_model</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">256</span></pre></div>
</div>
</div>
<div class='section' id='section-4'>
<div class='docs'>
<div class='section-link'>
<a href='#section-4'>#</a>
</div>
<p>Number of <a href="#ConvMixerLayer">ConvMixer layers</a> or depth, $d$</p>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">38</span> <span class="n">n_layers</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">8</span></pre></div>
</div>
</div>
<div class='section' id='section-5'>
<div class='docs'>
<div class='section-link'>
<a href='#section-5'>#</a>
</div>
<p>Kernel size of the depth-wise convolution, $k$</p>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">40</span> <span class="n">kernel_size</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">7</span></pre></div>
</div>
</div>
<div class='section' id='section-6'>
<div class='docs'>
<div class='section-link'>
<a href='#section-6'>#</a>
</div>
<p>Number of classes in the task</p>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">42</span> <span class="n">n_classes</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">10</span></pre></div>
</div>
</div>
<div class='section' id='section-7'>
<div class='docs doc-strings'>
<div class='section-link'>
<a href='#section-7'>#</a>
</div>
<h3>Create model</h3>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">45</span><span class="nd">@option</span><span class="p">(</span><span class="n">Configs</span><span class="o">.</span><span class="n">model</span><span class="p">)</span>
<span class="lineno">46</span><span class="k">def</span> <span class="nf">_conv_mixer</span><span class="p">(</span><span class="n">c</span><span class="p">:</span> <span class="n">Configs</span><span class="p">):</span></pre></div>
</div>
</div>
<div class='section' id='section-8'>
<div class='docs'>
<div class='section-link'>
<a href='#section-8'>#</a>
</div>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">50</span> <span class="kn">from</span> <span class="nn">labml_nn.conv_mixer</span> <span class="kn">import</span> <span class="n">ConvMixerLayer</span><span class="p">,</span> <span class="n">ConvMixer</span><span class="p">,</span> <span class="n">ClassificationHead</span><span class="p">,</span> <span class="n">PatchEmbeddings</span></pre></div>
</div>
</div>
<div class='section' id='section-9'>
<div class='docs'>
<div class='section-link'>
<a href='#section-9'>#</a>
</div>
<p>Create ConvMixer</p>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">53</span> <span class="k">return</span> <span class="n">ConvMixer</span><span class="p">(</span><span class="n">ConvMixerLayer</span><span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">d_model</span><span class="p">,</span> <span class="n">c</span><span class="o">.</span><span class="n">kernel_size</span><span class="p">),</span> <span class="n">c</span><span class="o">.</span><span class="n">n_layers</span><span class="p">,</span>
<span class="lineno">54</span> <span class="n">PatchEmbeddings</span><span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">d_model</span><span class="p">,</span> <span class="n">c</span><span class="o">.</span><span class="n">patch_size</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span>
<span class="lineno">55</span> <span class="n">ClassificationHead</span><span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">d_model</span><span class="p">,</span> <span class="n">c</span><span class="o">.</span><span class="n">n_classes</span><span class="p">))</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">device</span><span class="p">)</span></pre></div>
</div>
</div>
<div class='section' id='section-10'>
<div class='docs'>
<div class='section-link'>
<a href='#section-10'>#</a>
</div>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">58</span><span class="k">def</span> <span class="nf">main</span><span class="p">():</span></pre></div>
</div>
</div>
<div class='section' id='section-11'>
<div class='docs'>
<div class='section-link'>
<a href='#section-11'>#</a>
</div>
<p>Create experiment</p>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">60</span> <span class="n">experiment</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">&#39;ConvMixer&#39;</span><span class="p">,</span> <span class="n">comment</span><span class="o">=</span><span class="s1">&#39;cifar10&#39;</span><span class="p">)</span></pre></div>
</div>
</div>
<div class='section' id='section-12'>
<div class='docs'>
<div class='section-link'>
<a href='#section-12'>#</a>
</div>
<p>Create configurations</p>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">62</span> <span class="n">conf</span> <span class="o">=</span> <span class="n">Configs</span><span class="p">()</span></pre></div>
</div>
</div>
<div class='section' id='section-13'>
<div class='docs'>
<div class='section-link'>
<a href='#section-13'>#</a>
</div>
<p>Load configurations</p>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">64</span> <span class="n">experiment</span><span class="o">.</span><span class="n">configs</span><span class="p">(</span><span class="n">conf</span><span class="p">,</span> <span class="p">{</span></pre></div>
</div>
</div>
<div class='section' id='section-14'>
<div class='docs'>
<div class='section-link'>
<a href='#section-14'>#</a>
</div>
<p>Optimizer</p>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">66</span> <span class="s1">&#39;optimizer.optimizer&#39;</span><span class="p">:</span> <span class="s1">&#39;Adam&#39;</span><span class="p">,</span>
<span class="lineno">67</span> <span class="s1">&#39;optimizer.learning_rate&#39;</span><span class="p">:</span> <span class="mf">2.5e-4</span><span class="p">,</span></pre></div>
</div>
</div>
<div class='section' id='section-15'>
<div class='docs'>
<div class='section-link'>
<a href='#section-15'>#</a>
</div>
<p>Training epochs and batch size</p>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">70</span> <span class="s1">&#39;epochs&#39;</span><span class="p">:</span> <span class="mi">150</span><span class="p">,</span>
<span class="lineno">71</span> <span class="s1">&#39;train_batch_size&#39;</span><span class="p">:</span> <span class="mi">64</span><span class="p">,</span></pre></div>
</div>
</div>
<div class='section' id='section-16'>
<div class='docs'>
<div class='section-link'>
<a href='#section-16'>#</a>
</div>
<p>Simple image augmentations</p>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">74</span> <span class="s1">&#39;train_dataset&#39;</span><span class="p">:</span> <span class="s1">&#39;cifar10_train_augmented&#39;</span><span class="p">,</span></pre></div>
</div>
</div>
<div class='section' id='section-17'>
<div class='docs'>
<div class='section-link'>
<a href='#section-17'>#</a>
</div>
<p>Do not augment images for validation</p>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">76</span> <span class="s1">&#39;valid_dataset&#39;</span><span class="p">:</span> <span class="s1">&#39;cifar10_valid_no_augment&#39;</span><span class="p">,</span>
<span class="lineno">77</span> <span class="p">})</span></pre></div>
</div>
</div>
<div class='section' id='section-18'>
<div class='docs'>
<div class='section-link'>
<a href='#section-18'>#</a>
</div>
<p>Set model for saving/loading</p>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">79</span> <span class="n">experiment</span><span class="o">.</span><span class="n">add_pytorch_models</span><span class="p">({</span><span class="s1">&#39;model&#39;</span><span class="p">:</span> <span class="n">conf</span><span class="o">.</span><span class="n">model</span><span class="p">})</span></pre></div>
</div>
</div>
<div class='section' id='section-19'>
<div class='docs'>
<div class='section-link'>
<a href='#section-19'>#</a>
</div>
<p>Start the experiment and run the training loop</p>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">81</span> <span class="k">with</span> <span class="n">experiment</span><span class="o">.</span><span class="n">start</span><span class="p">():</span>
<span class="lineno">82</span> <span class="n">conf</span><span class="o">.</span><span class="n">run</span><span class="p">()</span></pre></div>
</div>
</div>
<div class='section' id='section-20'>
<div class='docs'>
<div class='section-link'>
<a href='#section-20'>#</a>
</div>
</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">86</span><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">&#39;__main__&#39;</span><span class="p">:</span>
<span class="lineno">87</span> <span class="n">main</span><span class="p">()</span></pre></div>
</div>
</div>
<div class='footer'>
<a href="https://papers.labml.ai">Trending Research Papers</a>
<a href="https://labml.ai">labml.ai</a>
</div>
</div>
<script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.4/MathJax.js?config=TeX-AMS_HTML">
</script>
<!-- MathJax configuration -->
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
tex2jax: {
inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'] ],
processEscapes: true,
processEnvironments: true
},
// Center justify equations in code and markdown cells. Elsewhere
// we use CSS to left justify single line equations in code cells.
displayAlign: 'center',
"HTML-CSS": { fonts: ["TeX"] }
});
</script>
<script>
function handleImages() {
var images = document.querySelectorAll('p>img')
console.log(images);
for (var i = 0; i < images.length; ++i) {
handleImage(images[i])
}
}
function handleImage(img) {
img.parentElement.style.textAlign = 'center'
var modal = document.createElement('div')
modal.id = 'modal'
var modalContent = document.createElement('div')
modal.appendChild(modalContent)
var modalImage = document.createElement('img')
modalContent.appendChild(modalImage)
var span = document.createElement('span')
span.classList.add('close')
span.textContent = 'x'
modal.appendChild(span)
img.onclick = function () {
console.log('clicked')
document.body.appendChild(modal)
modalImage.src = img.src
}
span.onclick = function () {
document.body.removeChild(modal)
}
}
handleImages()
</script>
</body>
</html>
\ No newline at end of file
docs/conv_mixer/index.html 0 → 100644
浏览文件 @ f8abd6a2
此差异已折叠。
docs/diffusion/ddpm/index.html
浏览文件 @ f8abd6a2
......@@ -599,18 +599,6 @@ q(x_t|x_0) &= \mathcal{N} \Big(x_t; \sqrt{\bar\alpha_t} x_0, (1-\bar\alpha_t) \m
<div class="highlight"><pre><span class="lineno">287</span> <span class="k">return</span> <span class="n">F</span><span class="o">.</span><span class="n">mse_loss</span><span class="p">(</span><span class="n">noise</span><span class="p">,</span> <span class="n">eps_theta</span><span class="p">)</span></pre></div>
</div>
</div>
<div class='section' id='section-33'>
<div class='docs'>
<div class='section-link'>
<a href='#section-33'>#</a>
</div>
<h2>Here&rsquo;s our Twitter thread with a summary</h2>
<p><blockquote class="twitter-tweet"><p lang="en" dir="ltr">Annotated <a href="https://twitter.com/PyTorch?ref_src=twsrc%5Etfw">@PyTorch</a> implementation of &quot;Denoising Diffusion Probabilistic Models&quot; by <a href="https://twitter.com/hojonathanho?ref_src=twsrc%5Etfw">@hojonathanho</a> <a href="https://twitter.com/ajayj_?ref_src=twsrc%5Etfw">@ajayj_</a> <a href="https://twitter.com/pabbeel?ref_src=twsrc%5Etfw">@pabbeel</a> <a href="https://twitter.com/berkeley_ai?ref_src=twsrc%5Etfw">@berkeley_ai</a><br><br>📝 Annotated code <a href="https://t.co/IxJMNQxJMa">https://t.co/IxJMNQxJMa</a><br>🖥 Github <a href="https://t.co/he5yIZZlB2">https://t.co/he5yIZZlB2</a><br>📎 Paper <a href="https://t.co/FjpamUVhLI">https://t.co/FjpamUVhLI</a><br><br>🧵👇 <a href="https://t.co/5SIZud6OnH">pic.twitter.com/5SIZud6OnH</a></p>&mdash; labml.ai (@labmlai) <a href="https://twitter.com/labmlai/status/1446676487361290240?ref_src=twsrc%5Etfw">October 9, 2021</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></p>
</div>
<div class='code'>
<div class="highlight"><pre></pre></div>
</div>
</div>
<div class='footer'>
<a href="https://papers.labml.ai">Trending Research Papers</a>
<a href="https://labml.ai">labml.ai</a>
......
docs/index.html
浏览文件 @ f8abd6a2
......@@ -102,6 +102,7 @@ implementations.</p>
<h4>✨ <a href="lstm/index.html">LSTM</a></h4>
<h4>✨ <a href="hypernetworks/hyper_lstm.html">HyperNetworks - HyperLSTM</a></h4>
<h4>✨ <a href="resnet/index.html">ResNet</a></h4>
<h4>✨ <a href="conv_mixer/index.html">ConvMixer</a></h4>
<h4>✨ <a href="capsule_networks/index.html">Capsule Networks</a></h4>
<h4>✨ <a href="gan/index.html">Generative Adversarial Networks</a></h4>
<ul>
......
docs/sitemap.xml
浏览文件 @ f8abd6a2
......@@ -344,7 +344,7 @@
<url>
<loc>https://nn.labml.ai/diffusion/ddpm/index.html</loc>
<lastmod>2021-10-08T16:30:00+00:00</lastmod>
<lastmod>2021-10-09T16:30:00+00:00</lastmod>
<priority>1.00</priority>
</url>
......@@ -881,6 +881,20 @@
</url>
<url>
<loc>https://nn.labml.ai/conv_mixer/index.html</loc>
<lastmod>2021-10-14T16:30:00+00:00</lastmod>
<priority>1.00</priority>
</url>
<url>
<loc>https://nn.labml.ai/conv_mixer/experiment.html</loc>
<lastmod>2021-10-14T16:30:00+00:00</lastmod>
<priority>1.00</priority>
</url>
<url>
<loc>https://nn.labml.ai/uncertainty/evidence/index.html</loc>
<lastmod>2021-08-21T16:30:00+00:00</lastmod>
......
labml_nn/__init__.py
浏览文件 @ f8abd6a2
......@@ -42,6 +42,8 @@ implementations.
#### ✨ [ResNet](resnet/index.html)
#### ✨ [ConvMixer](conv_mixer/index.html)
#### ✨ [Capsule Networks](capsule_networks/index.html)
#### ✨ [Generative Adversarial Networks](gan/index.html)
......
labml_nn/conv_mixer/__init__.py 0 → 100644
浏览文件 @ f8abd6a2
"""
---
title: Patches Are All You Need? (ConvMixer)
summary: >
A PyTorch implementation/tutorial of the paper
"Patches Are All You Need?"
---
# Patches Are All You Need? (ConvMixer)
This is a [PyTorch](https://pytorch.org) implementation of the paper
[Patches Are All You Need?](https://papers.labml.ai/paper/TVHS5Y4dNvM).
![ConvMixer diagram from the paper](conv_mixer.png)
ConvMixer is Similar to [MLP-Mixer](../transformers/mlp_mixer/index.html).
MLP-Mixer separates mixing of spatial and channel dimensions, by applying a MLP across spatial dimension
and then an MLP across the channel dimension
(spatial MLP replaces the [ViT](../transformers/vit/index.html) attention
and channel MLP is the [FFN](../transformers/feed_forward.html) of ViT).
ConvMixer use a $1 \times 1$ convolution for channel mixing and a
depth-wise convolution for spatial mixing.
Since it's a convolution instead of a full MLP across the space, it mixes only the nearby batches in
contrast to ViT or MLP-Mixer.
Also the MLP-mixer uses MLPs of two layers for each mixing and ConvMixer uses a single layer for each mixing.
The paper recommends removing the residual connection across the channel mixing (point-wise convolution),
and having only a residual connection over the spatial mixing (depth-wise convolution).
They also use [Batch normalization](../normalization/batch_norm/index.html) instead
of [Layer normalization)(../normalization/layer_norm/index.html).
Here's [an experiment](experiment.html) that trains ConvMixer on CIFAR-10.
[![View Run](https://img.shields.io/badge/labml-experiment-brightgreen)](https://app.labml.ai/run/0fc344da2cd011ecb0bc3fdb2e774a3d)
"""
import torch
from torch import nn
from labml_helpers.module import Module
from labml_nn.utils import clone_module_list
class ConvMixerLayer(Module):
"""
<a id="ConvMixerLayer">
## ConvMixer layer
</a>
This is a single ConvMixer layer. The model will have a series of these.
"""
def __init__(self, d_model: int, kernel_size: int):
"""
* `d_model` is the number of channels in patch embeddings, $h$
* `kernel_size` is the size of the kernel of spatial convolution, $k$
"""
super().__init__()
# Depth-wise convolution is separate convolution for each channel.
# We do this with a convolution layer with the number of groups equal to the number of channels.
# So that each channel is it's own group.
self.depth_wise_conv = nn.Conv2d(d_model, d_model,
kernel_size=kernel_size,
groups=d_model,
padding=(kernel_size - 1) // 2)
# Activation after depth-wise convolution
self.act1 = nn.GELU()
# Normalization after depth-wise convolution
self.norm1 = nn.BatchNorm2d(d_model)
# Point-wise convolution is a $1 \times 1$ convolution.
# i.e. a linear transformation of patch embeddings
self.point_wise_conv = nn.Conv2d(d_model, d_model, kernel_size=1)
# Activation after point-wise convolution
self.act2 = nn.GELU()
# Normalization after point-wise convolution
self.norm2 = nn.BatchNorm2d(d_model)
def forward(self, x: torch.Tensor):
# For the residual connection around the depth-wise convolution
residual = x
# Depth-wise convolution, activation and normalization
x = self.depth_wise_conv(x)
x = self.act1(x)
x = self.norm1(x)
# Add residual connection
x += residual
# Point-wise convolution, activation and normalization
x = self.point_wise_conv(x)
x = self.act2(x)
x = self.norm2(x)
#
return x
class PatchEmbeddings(Module):
"""
<a id="PatchEmbeddings">
## Get patch embeddings
</a>
This splits the image into patches of size $p \times p$ and gives an embedding for each patch.
"""
def __init__(self, d_model: int, patch_size: int, in_channels: int):
"""
* `d_model` is the number of channels in patch embeddings $h$
* `patch_size` is the size of the patch, $p$
* `in_channels` is the number of channels in the input image (3 for rgb)
"""
super().__init__()
# We create a convolution layer with a kernel size and and stride length equal to patch size.
# This is equivalent to splitting the image into patches and doing a linear
# transformation on each patch.
self.conv = nn.Conv2d(in_channels, d_model, kernel_size=patch_size, stride=patch_size)
# Activation function
self.act = nn.GELU()
# Batch normalization
self.norm = nn.BatchNorm2d(d_model)
def forward(self, x: torch.Tensor):
"""
* `x` is the input image of shape `[batch_size, channels, height, width]`
"""
# Apply convolution layer
x = self.conv(x)
# Activation and normalization
x = self.act(x)
x = self.norm(x)
#
return x
class ClassificationHead(Module):
"""
<a id="ClassificationHead">
## Classification Head
</a>
They do average pooling (taking the mean of all patch embeddings) and a final linear transformation
to predict the log-probabilities of the image classes.
"""
def __init__(self, d_model: int, n_classes: int):
"""
* `d_model` is the number of channels in patch embeddings, $h$
* `n_classes` is the number of classes in the classification task
"""
super().__init__()
# Average Pool
self.pool = nn.AdaptiveAvgPool2d((1, 1))
# Linear layer
self.linear = nn.Linear(d_model, n_classes)
def forward(self, x: torch.Tensor):
# Average pooling
x = self.pool(x)
# Get the embedding, `x` will have shape `[batch_size, d_model, 1, 1]`
x = x[:, :, 0, 0]
# Linear layer
x = self.linear(x)
#
return x
class ConvMixer(Module):
"""
## ConvMixer
This combines the patch embeddings block, a number of ConvMixer layers and a classification head.
"""
def __init__(self, conv_mixer_layer: ConvMixerLayer, n_layers: int,
patch_emb: PatchEmbeddings,
classification: ClassificationHead):
"""
* `conv_mixer_layer` is a copy of a single [ConvMixer layer](#ConvMixerLayer).
We make copies of it to make ConvMixer with `n_layers`.
* `n_layers` is the number of ConvMixer layers (or depth), $d$.
* `patch_emb` is the [patch embeddings layer](#PatchEmbeddings).
* `classification` is the [classification head](#ClassificationHead).
"""
super().__init__()
# Patch embeddings
self.patch_emb = patch_emb
# Classification head
self.classification = classification
# Make copies of the [ConvMixer layer](#ConvMixerLayer)
self.conv_mixer_layers = clone_module_list(conv_mixer_layer, n_layers)
def forward(self, x: torch.Tensor):
"""
* `x` is the input image of shape `[batch_size, channels, height, width]`
"""
# Get patch embeddings. This gives a tensor of shape `[batch_size, d_model, height / patch_size, width / patch_size]`.
x = self.patch_emb(x)
# Pass through [ConvMixer layers](#ConvMixerLayer)
for layer in self.conv_mixer_layers:
x = layer(x)
# Classification head, to get logits
x = self.classification(x)
#
return x
labml_nn/conv_mixer/experiment.py 0 → 100644
浏览文件 @ f8abd6a2
"""
---
title: Train ConvMixer on CIFAR 10
summary: >
Train ConvMixer on CIFAR 10
---
# Train a [ConvMixer](index.html) on CIFAR 10
This script trains a ConvMixer on CIFAR 10 dataset.
This is not an attempt to reproduce the results of the paper.
The paper uses image augmentations
present in [PyTorch Image Models (timm)](https://github.com/rwightman/pytorch-image-models)
for training. We haven't done this for simplicity - which causes our validation accuracy to drop.
[![View Run](https://img.shields.io/badge/labml-experiment-brightgreen)](https://app.labml.ai/run/0fc344da2cd011ecb0bc3fdb2e774a3d)
"""
from labml import experiment
from labml.configs import option
from labml_nn.experiments.cifar10 import CIFAR10Configs
class Configs(CIFAR10Configs):
"""
## Configurations
We use [`CIFAR10Configs`](../experiments/cifar10.html) which defines all the
dataset related configurations, optimizer, and a training loop.
"""
# Size of a patch, $p$
patch_size: int = 2
# Number of channels in patch embeddings, $h$
d_model: int = 256
# Number of [ConvMixer layers](#ConvMixerLayer) or depth, $d$
n_layers: int = 8
# Kernel size of the depth-wise convolution, $k$
kernel_size: int = 7
# Number of classes in the task
n_classes: int = 10
@option(Configs.model)
def _conv_mixer(c: Configs):
"""
### Create model
"""
from labml_nn.conv_mixer import ConvMixerLayer, ConvMixer, ClassificationHead, PatchEmbeddings
# Create ConvMixer
return ConvMixer(ConvMixerLayer(c.d_model, c.kernel_size), c.n_layers,
PatchEmbeddings(c.d_model, c.patch_size, 3),
ClassificationHead(c.d_model, c.n_classes)).to(c.device)
def main():
# Create experiment
experiment.create(name='ConvMixer', comment='cifar10')
# Create configurations
conf = Configs()
# Load configurations
experiment.configs(conf, {
# Optimizer
'optimizer.optimizer': 'Adam',
'optimizer.learning_rate': 2.5e-4,
# Training epochs and batch size
'epochs': 150,
'train_batch_size': 64,
# Simple image augmentations
'train_dataset': 'cifar10_train_augmented',
# Do not augment images for validation
'valid_dataset': 'cifar10_valid_no_augment',
})
# Set model for saving/loading
experiment.add_pytorch_models({'model': conf.model})
# Start the experiment and run the training loop
with experiment.start():
conf.run()
#
if __name__ == '__main__':
main()
labml_nn/diffusion/ddpm/__init__.py
浏览文件 @ f8abd6a2
......@@ -285,6 +285,3 @@ class DenoiseDiffusion:
# MSE loss
return F.mse_loss(noise, eps_theta)
# ## Here's our Twitter thread with a summary
# <blockquote class="twitter-tweet"><p lang="en" dir="ltr">Annotated <a href="https://twitter.com/PyTorch?ref_src=twsrc%5Etfw">@PyTorch</a> implementation of &quot;Denoising Diffusion Probabilistic Models&quot; by <a href="https://twitter.com/hojonathanho?ref_src=twsrc%5Etfw">@hojonathanho</a> <a href="https://twitter.com/ajayj_?ref_src=twsrc%5Etfw">@ajayj_</a> <a href="https://twitter.com/pabbeel?ref_src=twsrc%5Etfw">@pabbeel</a> <a href="https://twitter.com/berkeley_ai?ref_src=twsrc%5Etfw">@berkeley_ai</a><br><br>📝 Annotated code <a href="https://t.co/IxJMNQxJMa">https://t.co/IxJMNQxJMa</a><br>🖥 Github <a href="https://t.co/he5yIZZlB2">https://t.co/he5yIZZlB2</a><br>📎 Paper <a href="https://t.co/FjpamUVhLI">https://t.co/FjpamUVhLI</a><br><br>🧵👇 <a href="https://t.co/5SIZud6OnH">pic.twitter.com/5SIZud6OnH</a></p>&mdash; labml.ai (@labmlai) <a href="https://twitter.com/labmlai/status/1446676487361290240?ref_src=twsrc%5Etfw">October 9, 2021</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
readme.md
浏览文件 @ f8abd6a2
......@@ -47,6 +47,8 @@ implementations almost weekly.
#### ✨ [ResNet](https://nn.labml.ai/resnet/index.html)
#### ✨ [ConvMixer](https://nn.labml.ai/conv_mixer/index.html)
#### ✨ [Capsule Networks](https://nn.labml.ai/capsule_networks/index.html)
#### ✨ [Generative Adversarial Networks](https://nn.labml.ai/gan/index.html)
......
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或