Possible corrections to feed-forward tutorial

[ari62]

Concerning the tutorial here:
https://deeplearning4j.org/tutorials/04-feed-forward
A number of possible corrections I may or may not be right about:
The first example has an input and output layer but it says:
"As you can see above that we have made a feed-forward network configuration with one hidden layer. "

Also the input layer is a DenseLayer, not a FeedForwardLayer as I would assume it would be for this tutorial. In the second example there are multiple OutputLayers and no FeedForwardLayers as well.

Also the code seems to be an old version of dl4j. NeuralNetConfiguration.Builder doesn't seem to have learningRate or regularization anymore.

[saudet]

Please send a pull request! It makes it easier to review. Thanks

[printomi]

I had been really confused because the definition of input layer and hidden layer was not clear before I started using DL4J. Now I think that I understand.

1. I guess that there is no separate "input layer" in DL4J, as one would expect after reading this description of the feed-forward architecture:

A typical feed-forward network looks like this:
graphics
Here you can see a different layer named as a hidden layer. The layers in between our input and output layers are called hidden layers. It’s called hidden because we don’t directly deal with them and hence not visible. There can be more than one hidden layer in the network.

As I understand, hidden layers are called hidden because they are behind the output layer, thus error back-propagation is necessary to train them. So I think that:

• the graphic of separate input, hidden and output layers is misleading,
• the above quoted description from the tutorial is not true,
• and the following statement of the tutorial, also quoted by @ari62 is true:

As you can see above that we have made a feed-forward network configuration with one hidden layer.
If I understand correctly, input data does not go to an "input layer", but it goes directly to the inputs of first (hidden) layer.

2. @ari62 wrote:

Also the input layer is a DenseLayer, not a FeedForwardLayer as I would assume it would be for this tutorial.
Actually, DenseLayer, is a FeedForwardLayer, so this is not an error. See the API.

3. I confirm from @ari62 that

In the second example there are multiple OutputLayers and no FeedForwardLayers as well.
Here the example code is wrong. Layer # 0 is a DenseLayer, but layer # 1 and 2 should be
DenseLayer too.
You can similarly make network configurations with more hidden layers as:

...
    .layer(0, new DenseLayer.Builder().nIn(784).nOut(250).weightInit(WeightInit.XAVIER).activation(Activation.RELU) //First hidden layer
            .build())
    .layer(1, new OutputLayer.Builder().nIn(250).nOut(100).weightInit(WeightInit.XAVIER).activation(Activation.RELU) //Second hidden layer
            .build())
    .layer(2, new OutputLayer.Builder().nIn(100).nOut(50).weightInit(WeightInit.XAVIER).activation(Activation.RELU) //Third hidden layer
            .build())
    .layer(3, new OutputLayer.Builder().nIn(50).nOut(10).weightInit(WeightInit.XAVIER).activation(Activation.SOFTMAX) //Output layer
            .lossFunction(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD)
            .build())

4. I confirm from @ari62 that

Also the code seems to be an old version of dl4j.

5. I have to mention that in the examples repository, MLPMnistTwoLayerExample also uses false terminology when it calls the two DenseLayers input layers, and the OutputLayer a hidden layer. It would be two hidden layers and an output layer:

            .layer(new DenseLayer.Builder() //create the first input layer.
                    .nIn(numRows * numColumns)
                    .nOut(500)
                    .build())
            .layer(new DenseLayer.Builder() //create the second input layer
                    .nIn(500)
                    .nOut(100)
                    .build())
            .layer(new OutputLayer.Builder(LossFunction.NEGATIVELOGLIKELIHOOD) //create hidden layer
                    .activation(Activation.SOFTMAX)
                    .nOut(outputNum)
                    .build())

It would be good to clarify these terms in the tutorial and also in the example codes.

@saudet I would make a pull request if my notices seem to be right.

[saudet]

@printomi Yes, what you're saying is correct with regards to the usual conventions.
That material needs to be fixed!