torch.manual_seed Is All You Need

TORCH.MANUAL.SEED(3407) IS ALL YOU NEED torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision David Picard LIGM, École des Ponts, 77455 Marne la vallée, France DAVID.PICARD@ENPC.FR Abstract In this paper I investigate the effect of random seed selection on the accuracy when using popular deep learning architectures for computer vision. I scan a large amount of seeds (up to 10⁴) on CIFAR 10 and I also scan for seeds on Imagenet using pre-trained models to evaluate large scale datasets. The conclusions are that even if the variance is not very large, it is surprisingly easy to find an outlier that performs much better or much worse than the average. Keywords: Deep Learning, Computer Vision, Randomness 1. Introduction In this report, I want to test the influence of the random generator seed on the results of ubiquitous deep learning models used in computer vision. I ask the following question: 1. What is the distribution of scores with respect to the choice of seed? 2. Are there black swans, i.e., seeds that produce radically different results? 3. Does pretraining on larger datasets mitigate variability induced by the choice of seed? I feel this questions are important to ask and test because with the rise of large training sets and large deep learning models, it has become common practice in computer vision (and to other domains relying heavily on machine learning) to report only a single run of the experiment, without testing for its statistical validity. This trend started when computing power was limited, and it is perfectly understandable that the result of a single experiment is better than no result at all. To the best of my understanding, this seems to happen all the time in physics - first results get quickly announced, and are later confirmed or denied. The deep learning community has not made a faulty approach in that it never confirms results by running as many replications as required. Just as the noise in the measurement of a physical experiment, we know there are random factors in a deep learning experiment, like the split between training and testing data, the random initialization, the stochasticity of the optimization process, etc. It has been shown in Mehrer et al. (2020) that these affect the obtained networks such a way that they differ measurably. I strongly believe it is important to have a sense of the influence of these factors on the results in terms of evaluation metrics. This is essentially what was done by Bouthillier et al. (2021) on different tasks, including computer vision, but limited to smaller datasets and only 200 runs. Instead

This image presents the first page of a mock academic paper titled “torch.manual_seed(3407) is all you need,” parodying the paper “Attention Is All You Need.” It discusses how random seeds can significantly affect deep-learning results and jokes that choosing the right seed may produce unusually strong performance. The meme is used by machine-learning practitioners to mock irreproducible experiments, cherry-picked results, and the outsized importance of setting a random seed. EN: This image presents the first page of a mock academic paper titled “torch.manual_seed(3407) is all you need,” parodying the paper “Attention Is All You Need.” It discusses how random seeds can significantly affect deep-learning results and jokes that choosing the right seed may produce unusually strong performance. The meme is used by machine-learning practitioners to mock irreproducible experiments, cherry-picked results, and the outsized importance of setting a random seed.

• @ithueti / 4805 2021-09-28