[1912.02178] Fantastic Generalization Measures and Where to Find Them
Our experiments demonstrate that the study of generalization measure can be misleading when the number of models studied is small and the metric of quantifying the relationship is not carefully chosen

Abstract Generalization of deep networks has been of great interest in recent years, resulting in a number of theoretically and empirically motivated complexity measures. However, most papers proposing such measures study only a small set of models, leaving open the question of whether the conclusion drawn from those experiments would remain valid in other settings. We present the first large scale study of generalization in deep networks. We investigate more then 40 complexity measures taken from both theoretical bounds and empirical studies. We train over 10,000 convolutional networks by systematically varying commonly used hyperparameters. Hoping to uncover potentially causal relationships between each measure and generalization, we analyze carefully controlled experiments and show surprising failures of some measures as well as promising measures for further research.
‹

m_i = abs(Theta_1 x dots x Theta_(i - 1) x Theta_(i + 1) x dots x Theta_n)

Figure 1: Left: Graph at initialization of IC algorithm. Middle: The ideal graph where the measure µ can directly explain observed generalization. Right: Graph for correlation where µ cannot explain observed generalization. (Conditional Independence Test: Towards Capturing the Causal Relationships)Figure 2: Left: Number of models with training accuracy above 0.99 for each hyperparameter type. Middle: Distribution of training cross-entropy; distribution of training error can be found in Fig. ??. Right: Distribution of generalization gap. (Generating a Family of Trained Models)

m_i = abs(Theta_1 x dots x Theta_(i - 1) x Theta_(i + 1) x dots x Theta_n)

Figure 3: Joint Probability table for a single Sab (Definition of Random Variables)Figure 4: Distribution of training error on the trained models. (All Results)