**Demon CM**, or **SGD with Momentum and Demon**,  is the [Demon](https://paperswithcode.com/method/demon) momentum rule applied to [SGD with momentum](https://paperswithcode.com/method/sgd-with-momentum).

$$ \beta\_{t} = \beta\_{init}\cdot\frac{\left(1-\frac{t}{T}\right)}{\left(1-\beta\_{init}\right) + \beta\_{init}\left(1-\frac{t}{T}\right)} $$

$$ \theta\_{t+1} = \theta\_{t} - \eta{g}\_{t} + \beta\_{t}v\_{t} $$

$$ v\_{t+1} = \beta\_{t}{v\_{t}} - \eta{g\_{t}} $$

A [GCN](https://paperswithcode.com/method/gcn) method targeted at the unique spectral properties of dense graphs and hypergraphs, enabled by efficient numerical linear algebra.

PinvGCN

Pseudoinverse Graph Convolutional Networks: Fast Filters Tailored for Large Eigengaps of Dense Graphs and Hypergraphs

Demon CM

Demon: Improved Neural Network Training with Momentum Decay

Unified VLP is unified encoder-decoder model for general vision-language pre-training. The models uses a shared multi-layer transformers network for both encoding and decoding. The model is pre-trained on large amount of image-text pairs using the unsupervised learning objectives of two tasks: bidirectional and sequence-to-sequence (seq2seq) masked vision-language prediction. Model architecture for pre-training. For pre-training , the input comprises of image input, sentence input, and three special tokens ([CLS], [SEP], [STOP]). The image is processed as $N$ Region of Interests (RoIs) and region features are extracted. The sentence is tokenized and masked with [MASK] tokens for the later masked language modeling task. The model consists of 12 layers of Transformer blocks, each having a masked self-attention layer and feed-forward module, where the self-attention mask controls what input context the prediction conditions on. Two self-attention masks are implemented depending on whether the objective is bidirectional or seq2seq. The model is fine-tuned for image captioning and visual question answering.

Source	Demon: Improved Neural Network Training with Momentum Decay
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com