Gradient Origin Networks (Paper Explained w/ Live Coding)

Neural networks for implicit representations, such as SIRENs, have been very successful at modeling natural signals. However, in the classical approach, each data point requires its own neural network to be fit. This paper extends implicit representations to an entire dataset by introducing latent vectors of data points to SIRENs. Interestingly, the paper shows that such latent vectors can be obtained without the need for an explicit encoder, by simply looking at the negative gradient of the zero-vector through the representation function.

OUTLINE:
0:00 - Intro & Overview
2:10 - Implicit Generative Models
5:30 - Implicitly Represent a Dataset
11:00 - Gradient Origin Networks
23:55 - Relation to Gradient Descent
28:05 - Messing with their Code
37:40 - Implicit Encoders
38:50 - Using GONs as classifiers
40:55 - Experiments & Conclusion

Paper: https://arxiv.org/abs/2007.02798
Code: https://github.com/cwkx/GON
Project Page: https://cwkx.github.io/data/GON/

My Video on SIREN: https://youtu.be/Q5g3p9Zwjrk

Abstract:
This paper proposes a new type of implicit generative model that is able to quickly learn a latent representation without an explicit encoder. This is achieved with an implicit neural network that takes as inputs points in the coordinate space alongside a latent vector initialised with zeros. The gradients of the data fitting loss with respect to this zero vector are jointly optimised to act as latent points that capture the data manifold. The results show similar characteristics to autoencoders, but with fewer parameters and the advantages of implicit representation networks.

Authors: Sam Bond-Taylor, Chris G. Willcocks

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher