Divine Benevolence Explains SwiGLU

"We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence" – Noam Shazeer (second author of the transformer paper, now CEO of Character AI) from the SwiGLU paper: arxiv.org/abs/2002.05202... Conclusions We have extended the GLU family of layers and proposed their use in Transformer. In a transfer-learning setting, the new variants seem to produce better perplexities for the de-noising objective used in pre-training, as well as better results on many downstream language-understanding tasks. These are improvements are substantial, and we have no apparent practical drawbacks. We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.

A screenshot of a tweet quoting Noam Shazeer about the unexplained success of neural-network architectures, alongside a highlighted passage from the SwiGLU research paper. The joke is that the paper humorously attributes the model’s effectiveness to “divine benevolence” instead of offering a technical explanation, making it a reaction to unexpectedly effective but poorly understood machine-learning results.

• @gentoomemes / 3104 2023-03-14
• @rzeczyzabawne / 7252 2023-03-14