DDPM from Scratch

Code at: https://github.com/yashwantherukulla/DDPM-From-Scratch

Linear Noise Scheduler

Forward Process

• Given $x_0,t,ϵ$ return $x_t=\sqrt{\overline{{\alpha }_t}}{x}_0+\sqrt{1-\overline{{\alpha }_t}}ϵ$
• In order for the entire thing to be optimized, we will pre-compute and store the $\alpha$ s and the cummulative product terms $\overline{{\alpha }_t}$ in advance.
• Here ${\alpha }_t=1-{\beta }_t$ and $\overline{{\alpha }_t}={\prod }_{i=1}^t{\alpha }_i$
• The authors use a linear noise scheduler where they linearly scale ${\beta }_t$ with ${\beta }_1=0.0001$, ${\beta }_T=0.02$ and $T=1000$.

Reverse Process

• Given $x_T$, returns $x_{T-1}$ i.e., $p_{\theta }(x_{t-1}|x_t)$ by sampling the reverse distribution.
• where,

$${\mu }_{\theta }=\frac{x_t}{\sqrt{\overline{{\alpha }_t}}}-\frac{(1-{\alpha }_t)(\sqrt{1-\overline{{\alpha }_t}})}{(1-\overline{{\alpha }_t})\sqrt{{\alpha }_t}}$$ $$\sum _q(t)=\frac{(1-{\alpha }_t)(1-{\bar{\alpha }}_{t-1})}{1-{\bar{\alpha }}_t}\mathbb{I}$$ $${\mu }_{\theta }+{\sigma }_t\text{z}\to x_{t-1}$$

• We will pre-compute and store, $(1-{\alpha }_t),(\sqrt{1-\overline{{\alpha }_t}}),(1-\overline{{\alpha }_t})$

Time Embedding Block

• Given a 1D tensor $t$ of shape $(B,)$, this block will output a Time Embedding of shape $(B,\text{embedding dimension})$
• the input will pass through this block as follows,
  1. Positional Encoding: Sinusoidal Positional encoding used in Transformers

$$P{E}_{(pos,2i)}=sin(pos/10000^{2i/d_{\mathrm{model}}})$$ $$P{E}_{(pos,2i+1)}=cos(pos/10000^{2i/d_{\mathrm{model}}})$$

2. FC
3. SiLU (Sigmoid Linear Units activation)
4. FC

U-Net Model

• Inspired from SD’s U-Net architecture on HuggingFace
• multiple layers of the above image is the down block in the below image.

Handwritten Notes on Detailed Math