In previous section, we studied about Local vs. Global Minimum
In this post we will understand one of the most important part of a Neural Network : Hidden Layers. We will be able to use this information to control the parameters related to hidden layers.
Hidden layers and their role
Multi Layer Neural Network
The Role of Hidden Layers
- The First hidden layer
- The first layer is nothing but the liner decision boundaries
- The simple logistic regression line outputs
- We can see them as multiple lines on the decision space
- The Second hidden layer
- The Second layer combines these lines and forms simple decision boundary shapes
- The third hidden layer forms even complex shapes within the boundaries generated by second layer.
- You can imagine all these layers together divide the whole objective space into multiple decision boundary shapes, the cases within the shape are class-1 outside the shape are class-2.
The Number of Hidden Layers
- There is no concrete rule to choose the right number. We need to choose by trial and error validation.
- Too few hidden layers might result in imperfect models. The error rate will be high.
- High number of hidden layers might lead to over-fitting, but it can be identified by using some validation techniques.
- The final number is based on the number of predictor variables, training data size and the complexity in the target.
- When we are in doubt, its better to go with many hidden nodes than few. It will ensure higher accuracy. The training process will be slower though.
- Cross validation and testing error can help us in determining the model with optimal hidden layers.
The next post is about Practice Digit Recognizer.
