a2c [1910.10844] Diametrical Risk Minimization: Theory and Computations

Enable JavaScript to see more content

arXiv2code // top new 14d 1m 2m 3m // Enable JavaScript to see more content

[share]	Go to arXiv Download as Jupyter Notebook	2019-10-25
[1910.10844] Diametrical Risk Minimization: Theory and Computations While these experiments support the proposition that minimization of diametrical risk leads to good generalization and can be used to handle problems with large Lipschitz moduli, there is still much to be explored with DRM Abstract: The theoretical and empirical performance of Empirical Risk Minimization (ERM) often suffers when loss functions are poorly behaved with large Lipschitz moduli and spurious sharp minimizers. We propose and analyze a counterpart to ERM called Diametrical Risk Minimization (DRM), which accounts for worst-case empirical risks within neighborhoods in parameter space. DRM has generalization bounds that are independent of Lipschitz moduli for convex as well as nonconvex problems and it can be implemented using a practical algorithm based on stochastic gradient descent. Numerical results illustrate the ability of DRM to find quality solutions with low generalization error in chaotic landscapes from benchmark neural network classification problems with corrupted labels. ‹ Figure 1: Sharp vs flat minimizers. Figure 2: Sharp minimizer from label noise in training set. Figure 3 (Introduction)Figure 4 Figure 5 Figure 6 Figure 7 Figure 8: MNIST experiments: (a) Test accuracy when trained for 150 epochs. (b) Test accuracy when trained for 350 epochs. (c) Distribution of empirical risk for 10,000 random points in γneighborhood of final 150 epoch DRM and ERM solution. (d) Distribution of empirical risk for 10,000 random points in γ-neighborhood of final 350 epoch DRM and ERM solution. (Experiments)Figure 9: Fully Connected Figure 10: ResNet20 Figure 11: Fully Connected Figure 12: ResNet20 Figure 13: CIFAR-10 experiments: (a-b) Test accuracy when trained for 800 epochs. (c-d) Distribution of empirical risk for 10,000 random points in γ-neighborhood of final 800 epoch DRM and ERM solution. (MNIST)›

Related: TFIDF [1905.01004v2] Stability and Generalization of Graph Convolutional Neural Networks[1610.02581] Variance-based regularization with convex objectives[1509.01240] Train faster, generalize better: Stability of stochastic gradient descent[1801.04295] Generalization Error Bounds for Noisy, Iterative Algorithms[1801.04295v1] Generalization Error Bounds for Noisy, Iterative Algorithms[1703.11008] Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data[1802.06903] Generalization Error Bounds with Probabilistic Guarantee for SGD in Nonconvex Optimization[1907.04371] A Stochastic First-Order Method for Ordered Empirical Risk Minimization[1906.05032] Decoupling Gating from Linearity[1808.01174] Generalization Error in Deep Learning Related: Semantic Math [1310.8418] An efficient distributed learning algorithm based on effective local functional approximations[1611.01540] Topology and Geometry of Half-Rectified Network Optimization[1506.08350] Stochastic Gradient Made Stable: A Manifold Propagation Approach for Large-Scale Optimization[1607.00076] Multi-class classification: mirror descent approach[1602.03552] Learning Privately from Multiparty Data[1501.01181] Object localization in ImageNet by looking out of the window[1809.04673] A Unified Batch Online Learning Framework for Click Prediction[1808.06167] Source-Critical Reinforcement Learning for Transferring Spoken Language Understanding to a New Language[1907.05106] Spatiotemporal Local Propagation