Practical Advice For Building Neural Nets Deep Learning and Neural Nets Spring 2015 Day’s Agenda 1. Celebrity guests 2. Discuss issues and observations from homework 3. Catrin Mills on climate change in the Arctic 4. Practical issues in building nets Homework Discussion Lei’s problem with the perceptron not converging demo Manjunath’s questions about priors Why does the balance of data make a difference in what is learned? (I believe it will for Assignment 2) If I’d told you the priors should be 50/50, what would you have to do differently? What’s the relationship between statistics of the training set and the test set? Suppose we have a classifier that outputs a probability (not just 0 or 1). If you know the priors in the training set and you know the priors in the test set to be different, how do you repair the classifier to accommodate the mismatch between training and testing priors? Weight Initialization Break symmetries use small random values Weight magnitudes should depend on fan in What Mike has always done Draw all weights feeding into neuron j (including bias) via w ji ∼ Normal(0,1) Normalize weights such that i.e., 2 w ji ¬ w ji å w ji =2 i w ji Works well for logistic units; to be determined for ReLU units Weight Initialization A perhaps better idea (due to Kelvin Wagner) Draw all weights feeding into neuron j (including bias) via w ji ∼ Normal(0,1) If input activities lie in [-1, +1], then variance of input to unit j grows with fan-in to j, fj Normalize such that variance of input is equal to C2 i.e., w ¬ C w ji fj ji If input activities lie in [-1, +1], most net inputs will be in [-2C, +2C] Does Weight Initialization Scheme Matter? Trained 3 layer net on 2500 digit patterns 10 output classes, tanh units 20-1600 hidden units automatic adaptation of learning rates 10 minibatches per epoch 500 epochs 12 replications of each architecture Does Weight Initialization Scheme Matter? Weight initialization schemes Gaussian random weights, N(0,.0012) Gaussian random weights, N(0,.012) L1 constraint on Gaussian random weights [conditioning for worst case] Gaussian weights, N(0, 4/FanIn) [conditioning for average case] Gaussian weights, N(0, 1/FanIn) [conditioning for average case] å w ji i =2 Small Random Weights Strangeness training set can’t be learned caveat: plotting accuracy not MSE if there’s overfitting, doesn’t happen until 200 hidden Mike’s L1 Normalization Scheme About the same as small random weights Normalization Based On Fan In Perfect performance on training set Test set performance dependent on scaling Conditioning The Input Vectors If mi is the mean activity of input unit i over the training set and si is the std dev over the training set For each input (in both training and test sets), normalize by a x a i - xi xi ¬ si where xi is the training set mean activity and si is the std deviation of the training set activities Conditioning The Hidden Units If you’re using logistic units, then replace logistic output with function scaled from -1 to +1 2 y= -net - 1 1+ e ¶y 1 = (1+ y)(1- y) ¶net 2 With net=0, y=0 Will tend to cause biases to be closer to zero and more on the same scale as other weights in network Will also satisfy assumption I make to condition initial weights and weight updates for the units in the next layer tanh function Setting Learning Rates I Initial guess for learning rate If error doesn’t drop consistently, lower initial learning rate and try again If error falls reliably but slowly, increase learning rate. Toward end of training Error will often jitter, at which point you can lower the learning rate down to 0 gradually to clean up weights Remember, plateaus in error often look like minima be patient have some idea a priori how well you expect your network to be doing, and print statistics during training that tell you how well it’s doing plot epochwise error as a function of epoch, even if you’re doing minibatches (t a - ya )2 å NormalizedError = å (t a - t )2 Setting Learning Rates II Momentum ¶E Dwt+1 = qDwt - (1- q )e ¶w t Adaptive and neuron-specific learning rates Observe error on epoch t-1 and epoch t If decreasing, then increase global learning rate, εglobal, by an additive constant If increasing, decrease global learning rate by a multiplicative constant If fan-in of neuron j is fj, then e j = e global fj Setting Learning Rates III Mike’s hack Initialization epsilon = .01 inc = epsilon / 10 if (batch_mode_training) scale = .5 else scale = .9 Update if (current_epoch_error < previous_epoch_error) epsilon = epsilon + inc saved_weights = weights else epsilon = epsilon * scale inc = epsilon / 10 if (batch_mode_training) weights = saved_weights Setting Learning Rates IV rmsprop Hinton lecture Exploit optimization methods using curvature Requires computation of Hessian When To Stop Training 1. Train n epochs; lower learning rate; train m epochs bad idea: can’t assume one-size-fits-all approach 2. Error-change criterion stop when error isn’t dropping My recommendation: criterion based on % drop over a window of, say, 10 epochs 1 epoch is too noisy absolute error criterion is too problem dependent Karl’s idea: train for a fixed number of epochs after criterion is reached (possibly with lower learning rate) When To Stop Training 3. Weight-change criterion Compare weights at epochs t-10 and t and test: t t-10 maxi wi - wi <q Don’t base on length of overall weight change vector Possibly express as a percentage of the weight Be cautious: small weight changes at critical points can result in rapid drop in error