Forward gradients are unbiased estimators of the gradient $\nabla f(\theta)$ for a function $f: \mathbb{R}^n \rightarrow \mathbb{R}$, given by $g(\theta) = \langle \nabla f(\theta) , v \rangle v$. 

Here $v = (v_1, \ldots, v_n)$ is a random vector, which must satisfy the following conditions in order for $g(\theta)$ to be an unbiased estimator of $\nabla f(\theta)$

* $v_i \perp v_j$ for all $i \neq j$
* $\mathbb{E}[v_i] = 0$ for all $i$
* $\mathbb{V}[v_i] = 1$ for all $i$

Forward gradients can be computed with a single jvp (Jacobian Vector Product), which enables the use of the forward mode of autodifferentiation instead of the usual reverse mode, which has worse computational characteristics.

**Normalizing Flows** are a method for constructing complex distributions by transforming a
probability density through a series of invertible mappings. By repeatedly applying the rule for change of variables, the initial density ‘flows’ through the sequence of invertible mappings. At the end of this sequence we obtain a valid probability distribution and hence this type of flow is referred to as a normalizing flow.

In the case of finite flows, the basic rule for the transformation of densities considers an invertible, smooth mapping $f : \mathbb{R}^{d} \rightarrow \mathbb{R}^{d}$ with inverse $f^{-1} = g$, i.e. the composition $g \cdot f\left(z\right) = z$. If we use this mapping to transform a random variable $z$ with distribution $q\left(z\right)$, the resulting random variable $z' = f\left(z\right)$ has a distribution:

$$ q\left(\mathbf{z}'\right) = q\left(\mathbf{z}\right)\bigl\vert{\text{det}}\frac{\delta{f}^{-1}}{\delta{\mathbf{z'}}}\bigr\vert = q\left(\mathbf{z}\right)\bigl\vert{\text{det}}\frac{\delta{f}}{\delta{\mathbf{z}}}\bigr\vert ^{-1} $$

where the last equality can be seen by applying the chain rule (inverse function theorem) and is a property of Jacobians of invertible functions. We can construct arbitrarily complex densities by composing several simple maps and successively applying the above equation. The density $q\_{K}\left(\mathbf{z}\right)$ obtained by successively transforming a random variable $z\_{0}$ with distribution $q\_{0}$ through a chain of $K$ transformations $f\_{k}$ is:

$$ z\_{K} = f\_{K} \cdot \dots \cdot f\_{2} \cdot f\_{1}\left(z\_{0}\right) $$

$$ \ln{q}\_{K}\left(z\_{K}\right) = \ln{q}\_{0}\left(z\_{0}\right) − \sum^{K}\_{k=1}\ln\vert\det\frac{\delta{f\_{k}}}{\delta{\mathbf{z\_{k-1}}}}\vert $$

The path traversed by the random variables $z\_{k} = f\_{k}\left(z\_{k-1}\right)$ with initial distribution $q\_{0}\left(z\_{0}\right)$ is called the flow and the path formed by the successive distributions $q\_{k}$ is a normalizing flow.

Normalizing Flows

Variational Inference with Normalizing Flows

Forward gradient

Gradients without Backpropagation

The **GAN Hinge Loss** is a hinge loss based loss function for [generative adversarial networks](https://paperswithcode.com/methods/category/generative-adversarial-networks):

$$ L\_{D} = -\mathbb{E}\_{\left(x, y\right)\sim{p}\_{data}}\left[\min\left(0, -1 + D\left(x, y\right)\right)\right] -\mathbb{E}\_{z\sim{p\_{z}}, y\sim{p\_{data}}}\left[\min\left(0, -1 - D\left(G\left(z\right), y\right)\right)\right] $$

$$ L\_{G} = -\mathbb{E}\_{z\sim{p\_{z}}, y\sim{p\_{data}}}D\left(G\left(z\right), y\right) $$

Source	Gradients without Backpropagation
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com