1. AMORE
1.1 newff
newff(n.neurons, learning.rate.global, momentum.global, error.criterium, Stao, hidden.layer, output.layer, method)
- n.neurons：包含每層神經元的數值向量。第一個元素是輸入神經元的數量，最后一個元素是輸出神經元的數量，剩余的是隱含層神經元的數量；
- learning.rate.global：每個神經元訓練時的學習效率；
- momentum.global：每個神經元的動量，僅幾個訓練算法需要該參數；
- error.criterium：用於度量神經網絡預測目標值接近程度的標准，可以使用以下幾項：
"LMS"：Least Mean Squares
"LMLS"：Least Mean Logarithm Squared
"TAO"：TAO Error
- Stao：當上一項為TAO時的Stao參數，對於其他的誤差標准無用
- hidden.layer：隱含層神經元的激活函數，可用："purelin"、"tansig"、"sigmoid"、"hardlim"、"custom"，其中"custom"表示需要用戶自定義神經元的f0和f1元素；
- output.layer：輸出層神經元的激活函數；
- method：優先選擇的訓練方法，包括：
"ADAPTgd": Adaptative gradient descend.（自適應的梯度下降方法）
"ADAPTgdwm": Adaptative gradient descend with momentum.（基於動量因子的自適應梯度下降方法）
"BATCHgd": BATCH gradient descend.（批量梯度下降方法）
"BATCHgdwm": BATCH gradient descend with momentum.（基於動量因子的批量梯度下降方法）
1.2 train
train(net, P, T, Pval=NULL, Tval=NULL, error.criterium="LMS", report=TRUE, n.shows, show.step, Stao=NA, prob=NULL, n.threads=0L)
- net：要訓練的神經網絡；
- P：訓練集輸入值；
- T：訓練集輸出值；
- Pval：驗證集輸入值；
- Tval：驗證集輸出值；
- error.criterium：度量擬合優劣程度的標准；
- Stao：用於TAO算法的參數初始值；
- report：邏輯值，是否打印訓練過程信息；
- n.shows：當上一項為TRUE時，打印的總次數；
- show.step：訓練過程一直進行到函數允許打印報告信息時經歷的迭代次數；
- prob：每一個樣本應用到再抽樣訓練時的概率向量；
- n.threads：用於BATCH*訓練方法的線程數量，如果小於1，它將產生NumberProcessors-1個線程，其中NumberProcessors為處理器的個數，如果沒有找到OpenMP庫，該參數將被忽略。
1.3 example
library(AMORE)
# P is the input vector
P <- matrix(sample(seq(-1,1,length=1000), 1000, replace=FALSE), ncol=1)
# The network will try to approximate the target P^2
target <- P^2
# We create a feedforward network, with two hidden layers.
# The first hidden layer has three neurons and the second has two neurons.
# The hidden layers have got Tansig activation functions and the output layer is Purelin.
net <- newff(n.neurons=c(1,3,2,1), learning.rate.global=1e-2, momentum.global=0.5,
error.criterium="LMS", Stao=NA, hidden.layer="tansig",
output.layer="purelin", method="ADAPTgdwm")
result <- train(net, P, target, error.criterium="LMS", report=TRUE, show.step=100, n.shows=5 )
P_test <- matrix(sample(seq(-1,1,length=1000), 100, replace=FALSE), ncol=1)
target_test <- P_test^2
y_test <- sim(result$net, P_test)
plot(P_test,y_test-target_test)
index.show: 1 LMS 0.0893172434474773
index.show: 2 LMS 0.0892277761187557
index.show: 3 LMS 0.000380711026069436
index.show: 4 LMS 0.000155618390342181
index.show: 5 LMS 9.53881309223154e-05
1.4 exercise
P <- matrix(rnorm(2000),ncol=2)
target <- apply(P^2,1,mean)
net <- newff(n.neurons=c(2,20,1),
learning.rate.global=1e-2,
momentum.global=0.5,
error.criterium="LMS",
Stao=NA,
hidden.layer="tansig",
output.layer="purelin",
method="ADAPTgdwm")
result <- train(net, P, target,
error.criterium="LMS",
report=TRUE,
show.step=100,
n.shows=10)
P_test <- matrix(rnorm(200),ncol=2)
target_test <- apply(P_test^2,1,mean)
y_test <- sim(result$net, P_test)
plot(1:100,y_test-target_test)
(mean(abs(y_test-target_test)/target_test))
index.show: 1 LMS 0.00334168377394762
index.show: 2 LMS 0.00234274830312042
index.show: 3 LMS 0.00153074744985914
index.show: 4 LMS 0.00107702325684643
index.show: 5 LMS 0.000808644341077497
index.show: 6 LMS 0.000618371026226565
index.show: 7 LMS 0.000464896136767987
index.show: 8 LMS 0.000367196186841271
index.show: 9 LMS 0.00030261519286547
index.show: 10 LMS 0.000256892157194439
[1] 0.02934992
2. RSNNS
2.1 mlp
mlp(x, y, size = c(5), maxit = 100, initFunc = "Randomize_Weights", initFuncParams = c(-0.3, 0.3), learnFunc = "Std_Backpropagation", learnFuncParams = c(0.2, 0), updateFunc = "Topological_Order", updateFuncParams = c(0),
hiddenActFunc = "Act_Logistic", shufflePatterns = TRUE, linOut = FALSE, outputActFunc = if (linOut) "Act_Identity" else "Act_Logistic", inputsTest = NULL, targetsTest = NULL, pruneFunc = NULL, pruneFuncParams = NULL, ...)
- x：一個矩陣，作為訓練數據用於神經網絡的輸入；
- y：對應的目標值；
- size：隱含層單元的數量，默認為c(5)，當設置多個值時，表示有多個隱含層；
- maxit：學習過程的最大迭代次數，默認為100；
- initFunc：使用的初始化函數，默認為Randomize_Weights，即隨機權重；
- initFuncParams：用於初始化函數的參數，默認參數為c(-0.3,0.3)；
- learnFunc：使用的學習函數，默認為Std_Backpropagation；
- learnFuncParams：用於學習函數的參數，默認為c(0.2,0)；
- updateFunc：使用的更新函數，默認為Topological_Order；
- undateFuncParams：用於更新函數的參數，默認為c(0)；
- hiddenActFunc：所有隱含層神經元的激活函數，默認為Act_Logistic；
- shufflePatterns：是否將模式打亂，默認為TRUE；
- linOut，設置輸出神經元的激活函數，linear或者logistic，默認為FALSE；
- inputsTest：一個矩陣，作為測試數據用於神經網絡的輸入，默認為NULL；
- targetsTest：與測試輸入對應的目標值，默認為NULL；
- pruneFunc：使用的修剪函數，默認為NULL；
- pruneFuncParams：用於修剪函數的函數，默認為NULL。
2.2 example
數據准備
library(RSNNS) data(iris) #shuffle the vector iris <- iris[sample(1:nrow(iris),nrow(iris)),] irisValues <- iris[,1:4] irisTargets <- decodeClassLabels(iris[,5])
#irisTargets <- decodeClassLabels(iris[,5], valTrue=0.9, valFalse=0.1)
- decodeClassLabels: This method decodes class labels from a numerical or levels vector to a binary matrix, i.e., it converts the input vector to a binary matrix.
> head(irisTargets)
setosa versicolor virginica
[1,] 1 0 0
[2,] 0 1 0
[3,] 0 0 1
[4,] 1 0 0
[5,] 0 0 1
[6,] 0 0 1
抽樣
iris <- splitForTrainingAndTest(irisValues, irisTargets, ratio=0.15) iris <- normTrainingAndTestSet(iris)
- splitForTrainingAndTest: Split the input and target values to a training and a test set. Test set is taken from the end of the data. If the data is to be shuffled, this should be done before calling this function.
- normTrainingAndTestSet: Normalize training and test set as obtained by splitForTrainingAndTest in the following way: The inputsTrain member is normalized using normalizeData with the parameters given in type. The normalization parameters obtained during this normalization are then used to normalize the inputsTest member. if dontNormTargets is not set, then the targets are normalized in the same way. In classification problems, normalizing the targets normally makes no sense. For regression, normalizing also the targets is usually a good idea. The default is to not normalize targets values.
> attr(iris$inputsTrain,"normParams")$type [1] "norm" > attr(iris$inputsTest,"normParams")$type [1] "norm"
訓練&測試
model <- mlp(iris$inputsTrain, iris$targetsTrain,
size=5,
learnFuncParams=c(0.1),
maxit=50,
inputsTest=iris$inputsTest,
targetsTest=iris$targetsTest)
summary(model)
SNNS network definition file V1.4-3D
generated at Tue Feb 18 16:06:54 2020
network name : RSNNS_untitled
source files :
no. of units : 12（4+5+3=12個神經元）
no. of connections : 35（4×5+5×3=35權重）
no. of unit types : 0
no. of site types : 0
learning function : Std_Backpropagation
update function : Topological_Order
unit default section :
act | bias | st | subnet | layer | act func | out func
---------|----------|----|--------|-------|--------------|-------------
0.00000 | 0.00000 | i | 0 | 1 | Act_Logistic | Out_Identity
---------|----------|----|--------|-------|--------------|-------------
unit definition section :（給出偏置等參數）
no. | typeName | unitName | act | bias | st | position | act func | out func | sites
----|----------|-------------------|----------|----------|----|----------|--------------|----------|-------
1 | | Input_1 | 0.30243 | 0.27615 | i | 1,0,0 | Act_Identity | |
2 | | Input_2 | -0.55103 | 0.28670 | i | 2,0,0 | Act_Identity | |
3 | | Input_3 | 0.51752 | 0.05927 | i | 3,0,0 | Act_Identity | |
4 | | Input_4 | -0.00309 | 0.27664 | i | 4,0,0 | Act_Identity | |
5 | | Hidden_2_1 | 0.76338 | 1.91173 | h | 1,2,0 |||
6 | | Hidden_2_2 | 0.20872 | -2.48788 | h | 2,2,0 |||
7 | | Hidden_2_3 | 0.19751 | -0.35229 | h | 3,2,0 |||
8 | | Hidden_2_4 | 0.03343 | -1.45754 | h | 4,2,0 |||
9 | | Hidden_2_5 | 0.89784 | 0.82294 | h | 5,2,0 |||
10 | | Output_setosa | 0.05013 | -0.63914 | o | 1,4,0 |||
11 | | Output_versicolor | 0.82301 | -0.62233 | o | 2,4,0 |||
12 | | Output_virginica | 0.11542 | -1.02579 | o | 3,4,0 |||
----|----------|-------------------|----------|----------|----|----------|--------------|----------|-------
connection definition section :（給出權重矩陣）
target | site | source:weight
-------|------|---------------------------------------------------------------------------------------------------------------------
5 | | 4:-2.27555, 3:-1.59991, 2: 0.00019, 1: 0.26665
6 | | 4: 2.54355, 3: 1.83954, 2:-0.49918, 1:-0.21152
7 | | 4:-1.29117, 3:-0.90530, 2: 0.95187, 1:-0.20054
8 | | 4:-1.49130, 3:-1.54012, 2: 1.55497, 1:-0.85142
9 | | 4: 1.35965, 3: 1.07292, 2:-1.20123, 1: 0.45480
10 | | 9:-2.51547, 8: 2.26887, 7: 1.42713, 6:-2.24000, 5: 0.08607
11 | | 9: 1.31081, 8:-3.15358, 7:-1.09917, 6:-3.07740, 5: 2.55070
12 | | 9: 1.16730, 8:-2.03073, 7:-2.15568, 6: 2.90927, 5:-2.84570
-------|------|---------------------------------------------------------------------------------------------------------------------
> model
Class: mlp->rsnns
Number of inputs: 4
Number of outputs: 3
Maximal iterations: 50
Initialization function: Randomize_Weights
Initialization function parameters: -0.3 0.3
Learning function: Std_Backpropagation
Learning function parameters: 0.1
Update function:Topological_Order
Update function parameters: 0
Patterns are shuffled internally: TRUE
Compute error in every iteration: TRUE
Architecture Parameters:
$`size`
[1] 5
All members of model:
[1] "nInputs" "maxit"
[3] "initFunc" "initFuncParams"
[5] "learnFunc" "learnFuncParams"
[7] "updateFunc" "updateFuncParams"
[9] "shufflePatterns" "computeIterativeError"
[11] "snnsObject" "archParams"
[13] "IterativeFitError" "IterativeTestError"
[15] "fitted.values" "fittedTestValues"
[17] "nOutputs"
weightMatrix(model) #extractNetInfo(model)
| Hidden_2_1 | Hidden_2_2 | Hidden_2_3 | Hidden_2_4 | Hidden_2_5 | Output_setosa | Output_versicolor | Output_virginica | |
| Input_1 | 0.2666532695 | -0.2115191 | -0.2005364 | -0.8514175 | 0.4547978 | 0.00000000 | 0.000000 | 0.000000 |
| Input_2 | 0.0001948424 | -0.4991824 | 0.9518659 | 1.5549698 | -1.2012329 | 0.00000000 | 0.000000 | 0.000000 |
| Input_3 | -1.5999110937 | 1.8395387 | -0.9053020 | -1.5401160 | 1.0729166 | 0.00000000 | 0.000000 | 0.000000 |
| Input_4 | -2.2755548954 | 2.5435514 | -1.2911705 | -1.4912956 | 1.3596520 | 0.00000000 | 0.000000 | 0.000000 |
| Hidden_2_1 | 0.0000000000 | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 0.08606748 | 2.550698 | -2.845704 |
| Hidden_2_2 | 0.0000000000 | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | -2.24000168 | -3.077400 | 2.909271 |
| Hidden_2_3 | 0.0000000000 | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.42712903 | -1.099168 | -2.155685 |
| Hidden_2_4 | 0.0000000000 | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 2.26887059 | -3.153584 | -2.030726 |
| Hidden_2_5 | 0.0000000000 | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | -2.51547217 | 1.310814 | 1.167299 |
| Output_setosa | 0.0000000000 | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 0.00000000 | 0.000000 | 0.000000 |
| Output_versicolor | 0.0000000000 | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 0.00000000 | 0.000000 | 0.000000 |
| Output_virginica | 0.0000000000 | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 0.00000000 | 0.000000 | 0.000000 |
結果分析
繪出隨迭代次數增加，誤差平方和（SSE）的變化情況。
plotIterativeError(model)
散點為真實值和預測值的分布，黑色線為$y=x$，紅色線為一次回歸線。
predictions <- predict(model,iris$inputsTest) plotRegressionError(predictions[,2], iris$targetsTest[,2])
列出混淆矩陣，繪出EOC曲線。
> confusionMatrix(iris$targetsTrain,fitted.values(model))
predictions
targets 1 2 3
1 42 0 0
2 0 38 2
3 0 1 44
> confusionMatrix(iris$targetsTest,predictions)
predictions
targets 1 2 3
1 8 0 0
2 0 9 1
3 0 0 5
par(mfrow=c(1,2))
plotROC(fitted.values(model)[,2], iris$targetsTrain[,2]) plotROC(predictions[,2], iris$targetsTest[,2])
給出更嚴格的檢驗，若3項中1項大於0.6，其余兩項小於0.4，則將其歸為該項，否則標0，視為無法判定。
> confusionMatrix(iris$targetsTrain,
+ encodeClassLabels(fitted.values(model),
+ method="402040", l=0.4, h=0.6))
predictions
targets 0 1 2 3
1 0 42 0 0
2 4 0 36 0
3 3 0 0 42