Input conversation

[Lawrence]

Andrew,

I am converting all inputs into numbers between -1,1 is that correct for your NeuralNet?

Or should it be 0,1?

[andrew.kirillov]

Hello,

Converting inputs is not that critical as converting desired outputs. If outputs are not converted to the range of your sigmoid function, the network will never reach the right answer. The range for outputs is determined by the type of sigmoid function you use (which is what you need to check in your code). If you use SigmoidFunction, then its output is in [0, 1] range. If you use BipolarSigmoidFunction, then [-1, 1] range. Documentation is always the first place to check.

Personally I try to avoid [0, 1] range for inputs. The [-1, 1] range leads to quicker learning. And there is a reason for this. If you check theory (and the code), you will find that weights' updates are calculated by multiplying neuron's error value with its inputs. If inputs are zeros, then there is no update to weights of the neuron.

[nguyendo891]

Hi Andrew,
I think it is not really true that all outputs or inputs need to be set in the range of [0,1] or [-1,1] . To me, It could be 1 of the reason to cause wrong testing outputs.I have test with 9600 inputs and 2 output in the range [-1,1] like [1,1,1,-1,-1...,1,-1] or [-0.5,0,5,-0,5...0.5] or[0.001,-0.001,....,0.001] .In the end, the output I get doesnt match with the outputdersired. I think it still depend on input's quantity because sum += weight * input and somehow this sum is larger than the range [-1,1] even threshold cannot compensate this amount.
For example,assuming that in 9600 inputs we have 7000 inputs "-1" and 2600 input"1" and the weight are all positive , then the sum somehow << [-1,1] then applying the bipolarsigmoidfunction we get the output always become -1 .

[andrew.kirillov]

I think it is not really true that all outputs or inputs need to be set in the range of [0,1] or [-1,1]

Did I say they must be exactly in this range? What I meant is that they are not supposed to go out of that range. If your sigmoid function can not provide anything out of the [-1, 1] (or whatever), then all expected outputs must be in that range. But then it is up to you to use full range or smaller sub-range like [-0.5, 0.5], etc.

[nguyendo891]

Hi Andrew,
Yes, you are right , but I mean that with the large number of inputs and outputs are expected in [-1,1] or [-0.5,0.5] , then the output results are not distributed in the range[-1,1], it is almost be a constant (1 or -1) if I use sigmoid function. I dont know how to overcome this problem , do you have any suggestion ? Even I scaled outputs and inputs many times , I still get stuck in that problem.

[andrew.kirillov]

For large number of inputs, you may need to play with alpha parameter of sigmoid function. Make it a small value.

[nguyendo891]

I have 9600 inputs as mention before. And 7000 of them is "-1" and 2600 is "1" then the sum is aprroximate -4k , but the sum sometimes
has different value like -0.7 aprroximate 0 , the reason could be the percentage between "-1" and "1" is 50:50 . I trained 1000 samples ,each sample has 9600 inputs . Inputs and outputs are [-1,1...-1] and[-1,1] , learning rate :0.1, momentum :0.1, alpha : 2, neurons of 1st layer :2 , neurons of 2nd layer : 1

exsigmoid2.png (6.97 KiB) Viewed 20803 times

exsigmoid1.png (6.76 KiB) Viewed 20803 times

After that I try to change alpha to 1/4k6 and the errors decrease from 609 to 608 for 1000 iterations .In order to make the error decreasing to 0 , I have to set 600k iterations (I think). To me , it takes a lot of time just for 1000 samples . If someone want to increase the number of samples to 10k or 100k just for simple structure network ( 9600 inputs, 2 outputs, 2 neurons of 1st layer, 1 neuron of 2nd layer), are there any better solutions or better approaches ?

[andrew.kirillov]

are there any better solutions or better approaches ?

Maybe convolutional neural networks?

For example, in CIFAR-10, images are only of size 32x32x3 (32 wide, 32 high, 3 color channels), so a single fully connected neuron in a first hidden layer of a regular neural network would have 32*32*3 = 3,072 weights. A 200x200 image, however, would lead to neurons that have 200*200*3 = 120,000 weights.

Such network architecture does not take into account the spatial structure of data, treating input pixels which are far apart and close together on exactly the same footing. Clearly, the full connectivity of neurons is wasteful in the framework of image recognition, and the huge number of parameters quickly leads to overfitting.