卷積神經網絡(CNN)代碼實現(MNIST)解析

在http://blog.csdn.net/fengbingchun/article/details/50814710中給出了CNN的簡單實現，這里對每一步的實現作個說明：

共7層：依次為輸入層、C1層、S2層、C3層、S4層、C5層、輸出層。C代表卷積層(特征提取)。S代表降採樣層或池化層(Pooling)，輸出層為全連接層。

1.        各層權值、偏置(閾值)初始化：

各層權值、偏置個數計算例如以下：

(1)、輸入層：預處理后的32*32圖像數據。無權值和偏置；

(2)、C1層：卷積窗大小5*5，輸出特征圖數量6，卷積窗種類1*6=6。輸出特征圖大小28*28，因此可訓練參數(權值+偏置)：(5*5*1)*6+6=150+6。

         (3)、S2層：卷積窗大小2*2。輸出下採樣圖數量6，卷積窗種類6，輸出下採樣圖大小14*14，因此可訓練參數(權值+偏置)：1*6+6=6+6。

         (4)、C3層：卷積窗大小5*5。輸出特征圖數量16。卷積窗種類6*16=96，輸出特征圖大小10*10。因此可訓練參數(權值+偏置)：(5*5*6)*16+16=2400+16。

         (5)、S4層：卷積窗大小2*2。輸出下採樣圖數量16，卷積窗種類16，輸出下採樣圖大小5*5，因此可訓練參數(權值+偏置)：1*16+16=16+16。

         (6)、C5層：卷積窗大小5*5。輸出特征圖數量120，卷積窗種類16*120=1920，輸出特征圖大小1*1，因此可訓練參數(權值+偏置)：(5*5*16)*120+120=48000+120；

         (7)、輸出層：卷積窗大小1*1，輸出特征圖數量10。卷積窗種類120*10=1200，輸出特征圖大小1*1，因此可訓練參數(權值+偏置)：(1*120)*10+10=1200+10.

         代碼段例如以下：

#define num_map_input_CNN		1 //輸入層map個數
#define num_map_C1_CNN			6 //C1層map個數
#define num_map_S2_CNN			6 //S2層map個數
#define num_map_C3_CNN			16 //C3層map個數
#define num_map_S4_CNN			16 //S4層map個數
#define num_map_C5_CNN			120 //C5層map個數
#define num_map_output_CNN		10 //輸出層map個數

#define len_weight_C1_CNN		150 //C1層權值數，(5*5*1)*6=150
#define len_bias_C1_CNN			6 //C1層閾值數，6
#define len_weight_S2_CNN		6 //S2層權值數,1*6=6
#define len_bias_S2_CNN			6 //S2層閾值數,6
#define len_weight_C3_CNN		2400 //C3層權值數，(5*5*6)*16=2400
#define len_bias_C3_CNN			16 //C3層閾值數,16
#define len_weight_S4_CNN		16 //S4層權值數。1*16=16
#define len_bias_S4_CNN			16 //S4層閾值數。16
#define len_weight_C5_CNN		48000 //C5層權值數，(5*5*16)*120=48000
#define len_bias_C5_CNN			120 //C5層閾值數，120
#define len_weight_output_CNN	1200 //輸出層權值數。(1*120)*10=1200
#define len_bias_output_CNN		10 //輸出層閾值數，10

#define num_neuron_input_CNN	1024 //輸入層神經元數，(32*32)*1=1024
#define num_neuron_C1_CNN		4704 //C1層神經元數，(28*28)*6=4704
#define num_neuron_S2_CNN		1176 //S2層神經元數。(14*14)*6=1176
#define num_neuron_C3_CNN		1600 //C3層神經元數。(10*10)*16=1600
#define num_neuron_S4_CNN		400 //S4層神經元數。(5*5)*16=400
#define num_neuron_C5_CNN		120 //C5層神經元數。(1*1)*120=120
#define num_neuron_output_CNN	10 //輸出層神經元數，(1*1)*10=10

         權值、偏置初始化：

(1)、權值使用函數uniform_real_distribution均勻分布初始化。tiny-cnn中每次初始化權值數值都同樣。這里作了調整，使每次初始化的權值均不同。每層權值初始化大小范圍都不一樣；

(2)、全部層的偏置均初始化為0.

         代碼段例如以下：

double CNN::uniform_rand(double min, double max)
{
	//static std::mt19937 gen(1);
	std::random_device rd;
	std::mt19937 gen(rd());
	std::uniform_real_distribution<double> dst(min, max);
	return dst(gen);
}

bool CNN::uniform_rand(double* src, int len, double min, double max)
{
	for (int i = 0; i < len; i++) {
		src[i] = uniform_rand(min, max);
	}

	return true;
}

bool CNN::initWeightThreshold()
{
	srand(time(0) + rand());
	const double scale = 6.0;

	double min_ = -std::sqrt(scale / (25.0 + 150.0));
	double max_ = std::sqrt(scale / (25.0 + 150.0));
	uniform_rand(weight_C1, len_weight_C1_CNN, min_, max_);
	for (int i = 0; i < len_bias_C1_CNN; i++) {
		bias_C1[i] = 0.0;
	}

	min_ = -std::sqrt(scale / (4.0 + 1.0));
	max_ = std::sqrt(scale / (4.0 + 1.0));
	uniform_rand(weight_S2, len_weight_S2_CNN, min_, max_);
	for (int i = 0; i < len_bias_S2_CNN; i++) {
		bias_S2[i] = 0.0;
	}

	min_ = -std::sqrt(scale / (150.0 + 400.0));
	max_ = std::sqrt(scale / (150.0 + 400.0));
	uniform_rand(weight_C3, len_weight_C3_CNN, min_, max_);
	for (int i = 0; i < len_bias_C3_CNN; i++) {
		bias_C3[i] = 0.0;
	}

	min_ = -std::sqrt(scale / (4.0 + 1.0));
	max_ = std::sqrt(scale / (4.0 + 1.0));
	uniform_rand(weight_S4, len_weight_S4_CNN, min_, max_);
	for (int i = 0; i < len_bias_S4_CNN; i++) {
		bias_S4[i] = 0.0;
	}

	min_ = -std::sqrt(scale / (400.0 + 3000.0));
	max_ = std::sqrt(scale / (400.0 + 3000.0));
	uniform_rand(weight_C5, len_weight_C5_CNN, min_, max_);
	for (int i = 0; i < len_bias_C5_CNN; i++) {
		bias_C5[i] = 0.0;
	}

	min_ = -std::sqrt(scale / (120.0 + 10.0));
	max_ = std::sqrt(scale / (120.0 + 10.0));
	uniform_rand(weight_output, len_weight_output_CNN, min_, max_);
	for (int i = 0; i < len_bias_output_CNN; i++) {
		bias_output[i] = 0.0;
	}

	return true;
}

2.        載入MNIST數據：

關於MNIST的介紹能夠參考：http://blog.csdn.net/fengbingchun/article/details/49611549

使用MNIST庫作為訓練集和測試集。訓練樣本集為60000個，測試樣本集為10000個。

(1)、MNIST庫中圖像原始大小為28*28，這里縮放為32*32，數據取值范圍為[-1,1]，擴充值均取-1，作為輸入層輸入數據。

代碼段例如以下：

static void readMnistImages(std::string filename, double* data_dst, int num_image)
{
	const int width_src_image = 28;
	const int height_src_image = 28;
	const int x_padding = 2;
	const int y_padding = 2;
	const double scale_min = -1;
	const double scale_max = 1;

	std::ifstream file(filename, std::ios::binary);
	assert(file.is_open());

	int magic_number = 0;
	int number_of_images = 0;
	int n_rows = 0;
	int n_cols = 0;
	file.read((char*)&magic_number, sizeof(magic_number));
	magic_number = reverseInt(magic_number);
	file.read((char*)&number_of_images, sizeof(number_of_images));
	number_of_images = reverseInt(number_of_images);
	assert(number_of_images == num_image);
	file.read((char*)&n_rows, sizeof(n_rows));
	n_rows = reverseInt(n_rows);
	file.read((char*)&n_cols, sizeof(n_cols));
	n_cols = reverseInt(n_cols);
	assert(n_rows == height_src_image && n_cols == width_src_image);

	int size_single_image = width_image_input_CNN * height_image_input_CNN;

	for (int i = 0; i < number_of_images; ++i) {
		int addr = size_single_image * i;

		for (int r = 0; r < n_rows; ++r) {
			for (int c = 0; c < n_cols; ++c) {
				unsigned char temp = 0;
				file.read((char*)&temp, sizeof(temp));
				data_dst[addr + width_image_input_CNN * (r + y_padding) + c + x_padding] = (temp / 255.0) * (scale_max - scale_min) + scale_min;
			}
		}
	}
}

(2)、對於Label,輸出層有10個節點，相應位置的節點值設為0.8。其他節點設為-0.8，作為輸出層數據。

代碼段例如以下：

static void readMnistLabels(std::string filename, double* data_dst, int num_image)
{
	const double scale_max = 0.8;

	std::ifstream file(filename, std::ios::binary);
	assert(file.is_open());

	int magic_number = 0;
	int number_of_images = 0;
	file.read((char*)&magic_number, sizeof(magic_number));
	magic_number = reverseInt(magic_number);
	file.read((char*)&number_of_images, sizeof(number_of_images));
	number_of_images = reverseInt(number_of_images);
	assert(number_of_images == num_image);

	for (int i = 0; i < number_of_images; ++i) {
		unsigned char temp = 0;
		file.read((char*)&temp, sizeof(temp));
		data_dst[i * num_map_output_CNN + temp] = scale_max;
	}
}static void readMnistLabels(std::string filename, double* data_dst, int num_image)
{
	const double scale_max = 0.8;

	std::ifstream file(filename, std::ios::binary);
	assert(file.is_open());

	int magic_number = 0;
	int number_of_images = 0;
	file.read((char*)&magic_number, sizeof(magic_number));
	magic_number = reverseInt(magic_number);
	file.read((char*)&number_of_images, sizeof(number_of_images));
	number_of_images = reverseInt(number_of_images);
	assert(number_of_images == num_image);

	for (int i = 0; i < number_of_images; ++i) {
		unsigned char temp = 0;
		file.read((char*)&temp, sizeof(temp));
		data_dst[i * num_map_output_CNN + temp] = scale_max;
	}
}

3.        前向傳播：主要計算每層的神經元值。當中C1層、C3層、C5層操作過程同樣。S2層、S4層操作過程同樣。

(1)、輸入層：神經元數為(32*32)*1=1024。

(2)、C1層：神經元數為(28*28)*6=4704，分別用每個5*5的卷積圖像去乘以32*32的圖像，獲得一個28*28的圖像。即相應位置相加再求和，stride長度為1；一共6個5*5的卷積圖像，然后對每個神經元加上一個閾值。最后再通過tanh激活函數對每一神經元進行運算得到終於每個神經元的結果。

激活函數的作用：它是用來增加非線性因素的，解決線性模型所不能解決的問題。提供網絡的非線性建模能力。

假設沒有激活函數。那么該網絡僅能夠表達線性映射。此時即便有再多的隱藏層，其整個網絡跟單層神經網絡也是等價的。因此也能夠覺得，唯獨增加了激活函數之后，深度神經網絡才具備了分層的非線性映射學習能力。

代碼段例如以下：

double CNN::activation_function_tanh(double x)
{
	double ep = std::exp(x);
	double em = std::exp(-x);

	return (ep - em) / (ep + em);
}

bool CNN::Forward_C1()
{
	init_variable(neuron_C1, 0.0, num_neuron_C1_CNN);

	for (int o = 0; o < num_map_C1_CNN; o++) {
		for (int inc = 0; inc < num_map_input_CNN; inc++) {
			int addr1 = get_index(0, 0, num_map_input_CNN * o + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_C1_CNN * num_map_input_CNN);
			int addr2 = get_index(0, 0, inc, width_image_input_CNN, height_image_input_CNN, num_map_input_CNN);
			int addr3 = get_index(0, 0, o, width_image_C1_CNN, height_image_C1_CNN, num_map_C1_CNN);

			const double* pw = &weight_C1[0] + addr1;
			const double* pi = data_single_image + addr2;
			double* pa = &neuron_C1[0] + addr3;

			for (int y = 0; y < height_image_C1_CNN; y++) {
				for (int x = 0; x < width_image_C1_CNN; x++) {
					const double* ppw = pw;
					const double* ppi = pi + y * width_image_input_CNN + x;
					double sum = 0.0;

					for (int wy = 0; wy < height_kernel_conv_CNN; wy++) {
						for (int wx = 0; wx < width_kernel_conv_CNN; wx++) {
							sum += *ppw++ * ppi[wy * width_image_input_CNN + wx];
						}
					}

					pa[y * width_image_C1_CNN + x] += sum;
				}
			}
		}

		int addr3 = get_index(0, 0, o, width_image_C1_CNN, height_image_C1_CNN, num_map_C1_CNN);
		double* pa = &neuron_C1[0] + addr3;
		double b = bias_C1[o];
		for (int y = 0; y < height_image_C1_CNN; y++) {
			for (int x = 0; x < width_image_C1_CNN; x++) {
				pa[y * width_image_C1_CNN + x] += b;
			}
		}
	}

	for (int i = 0; i < num_neuron_C1_CNN; i++) {
		neuron_C1[i] = activation_function_tanh(neuron_C1[i]);
	}

	return true;
}

(3)、S2層：神經元數為(14*14)*6=1176，對C1中6個28*28的特征圖生成6個14*14的下採樣圖，相鄰四個神經元分別乘以同一個權值再進行相加求和，再求均值即除以4，然后再加上一個閾值，最后再通過tanh激活函數對每一神經元進行運算得到終於每個神經元的結果。

代碼段例如以下：

bool CNN::Forward_S2()
{
	init_variable(neuron_S2, 0.0, num_neuron_S2_CNN);
	double scale_factor = 1.0 / (width_kernel_pooling_CNN * height_kernel_pooling_CNN);

	assert(out2wi_S2.size() == num_neuron_S2_CNN);
	assert(out2bias_S2.size() == num_neuron_S2_CNN);

	for (int i = 0; i < num_neuron_S2_CNN; i++) {
		const wi_connections& connections = out2wi_S2[i];
		neuron_S2[i] = 0;

		for (int index = 0; index < connections.size(); index++) {
			neuron_S2[i] += weight_S2[connections[index].first] * neuron_C1[connections[index].second];
		}

		neuron_S2[i] *= scale_factor;
		neuron_S2[i] += bias_S2[out2bias_S2[i]];
	}

	for (int i = 0; i < num_neuron_S2_CNN; i++) {
		neuron_S2[i] = activation_function_tanh(neuron_S2[i]);
	}

	return true;
}

(4)、C3層：神經元數為(10*10)*16=1600。C3層實現方式與C1層全然同樣。由S2中的6個14*14下採樣圖生成16個10*10特征圖，對於生成的每個10*10的特征圖，是由6個5*5的卷積圖像去乘以6個14*14的下採樣圖，然后相應位置相加求和，然后對每個神經元加上一個閾值，最后再通過tanh激活函數對每一神經元進行運算得到終於每個神經元的結果。

也可依照Y.Lecun給出的表進行計算。即對於生成的每個10*10的特征圖，是由n個5*5的卷積圖像去乘以n個14*14的下採樣圖，當中n是小於6的，即不全然連接。這樣做的原因：第一，不全然的連接機制將連接的數量保持在合理的范圍內。第二，也是最重要的，其破壞了網絡的對稱性。因為不同的特征圖有不同的輸入，所以迫使他們抽取不同的特征。

代碼段例如以下：

// connection table [Y.Lecun, 1998 Table.1]
#define O true
#define X false
static const bool tbl[6][16] = {
	O, X, X, X, O, O, O, X, X, O, O, O, O, X, O, O,
	O, O, X, X, X, O, O, O, X, X, O, O, O, O, X, O,
	O, O, O, X, X, X, O, O, O, X, X, O, X, O, O, O,
	X, O, O, O, X, X, O, O, O, O, X, X, O, X, O, O,
	X, X, O, O, O, X, X, O, O, O, O, X, O, O, X, O,
	X, X, X, O, O, O, X, X, O, O, O, O, X, O, O, O
};
#undef O
#undef X

bool CNN::Forward_C3()
{
	init_variable(neuron_C3, 0.0, num_neuron_C3_CNN);

	for (int o = 0; o < num_map_C3_CNN; o++) {
		for (int inc = 0; inc < num_map_S2_CNN; inc++) {
			if (!tbl[inc][o]) continue;

			int addr1 = get_index(0, 0, num_map_S2_CNN * o + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_C3_CNN * num_map_S2_CNN);
			int addr2 = get_index(0, 0, inc, width_image_S2_CNN, height_image_S2_CNN, num_map_S2_CNN);
			int addr3 = get_index(0, 0, o, width_image_C3_CNN, height_image_C3_CNN, num_map_C3_CNN);

			const double* pw = &weight_C3[0] + addr1;
			const double* pi = &neuron_S2[0] + addr2;
			double* pa = &neuron_C3[0] + addr3;

			for (int y = 0; y < height_image_C3_CNN; y++) {
				for (int x = 0; x < width_image_C3_CNN; x++) {
					const double* ppw = pw;
					const double* ppi = pi + y * width_image_S2_CNN + x;
					double sum = 0.0;

					for (int wy = 0; wy < height_kernel_conv_CNN; wy++) {
						for (int wx = 0; wx < width_kernel_conv_CNN; wx++) {
							sum += *ppw++ * ppi[wy * width_image_S2_CNN + wx];
						}
					}

					pa[y * width_image_C3_CNN + x] += sum;
				}
			}
		}

		int addr3 = get_index(0, 0, o, width_image_C3_CNN, height_image_C3_CNN, num_map_C3_CNN);
		double* pa = &neuron_C3[0] + addr3;
		double b = bias_C3[o];
		for (int y = 0; y < height_image_C3_CNN; y++) {
			for (int x = 0; x < width_image_C3_CNN; x++) {
				pa[y * width_image_C3_CNN + x] += b;
			}
		}
	}

	for (int i = 0; i < num_neuron_C3_CNN; i++) {
		neuron_C3[i] = activation_function_tanh(neuron_C3[i]);
	}

	return true;
}

(5)、S4層：神經元數為(5*5)*16=400，S4層實現方式與S2層全然同樣。由C3中16個10*10的特征圖生成16個5*5下採樣圖，相鄰四個神經元分別乘以同一個權值再進行相加求和，再求均值即除以4，然后再加上一個閾值。最后再通過tanh激活函數對每一神經元進行運算得到終於每個神經元的結果。

代碼段例如以下：

bool CNN::Forward_S4()
{
	double scale_factor = 1.0 / (width_kernel_pooling_CNN * height_kernel_pooling_CNN);
	init_variable(neuron_S4, 0.0, num_neuron_S4_CNN);

	assert(out2wi_S4.size() == num_neuron_S4_CNN);
	assert(out2bias_S4.size() == num_neuron_S4_CNN);

	for (int i = 0; i < num_neuron_S4_CNN; i++) {
		const wi_connections& connections = out2wi_S4[i];
		neuron_S4[i] = 0.0;

		for (int index = 0; index < connections.size(); index++) {
			neuron_S4[i] += weight_S4[connections[index].first] * neuron_C3[connections[index].second];
		}

		neuron_S4[i] *= scale_factor;
		neuron_S4[i] += bias_S4[out2bias_S4[i]];
	}

	for (int i = 0; i < num_neuron_S4_CNN; i++) {
		neuron_S4[i] = activation_function_tanh(neuron_S4[i]);
	}

	return true;
}

(6)、C5層：神經元數為(1*1)*120=120，也可看為全連接層，C5層實現方式與C1、C3層全然同樣。由S4中16個5*5下採樣圖生成120個1*1特征圖，對於生成的每個1*1的特征圖，是由16個5*5的卷積圖像去乘以16個5*5的下採用圖，然后相加求和，然后對每個神經元加上一個閾值，最后再通過tanh激活函數對每一神經元進行運算得到終於每個神經元的結果。

代碼段例如以下：

bool CNN::Forward_C5()
{
	init_variable(neuron_C5, 0.0, num_neuron_C5_CNN);

	for (int o = 0; o < num_map_C5_CNN; o++) {
		for (int inc = 0; inc < num_map_S4_CNN; inc++) {
			int addr1 = get_index(0, 0, num_map_S4_CNN * o + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_C5_CNN * num_map_S4_CNN);
			int addr2 = get_index(0, 0, inc, width_image_S4_CNN, height_image_S4_CNN, num_map_S4_CNN);
			int addr3 = get_index(0, 0, o, width_image_C5_CNN, height_image_C5_CNN, num_map_C5_CNN);

			const double *pw = &weight_C5[0] + addr1;
			const double *pi = &neuron_S4[0] + addr2;
			double *pa = &neuron_C5[0] + addr3;

			for (int y = 0; y < height_image_C5_CNN; y++) {
				for (int x = 0; x < width_image_C5_CNN; x++) {
					const double *ppw = pw;
					const double *ppi = pi + y * width_image_S4_CNN + x;
					double sum = 0.0;

					for (int wy = 0; wy < height_kernel_conv_CNN; wy++) {
						for (int wx = 0; wx < width_kernel_conv_CNN; wx++) {
							sum += *ppw++ * ppi[wy * width_image_S4_CNN + wx];
						}
					}

					pa[y * width_image_C5_CNN + x] += sum;
				}
			}
		}

		int addr3 = get_index(0, 0, o, width_image_C5_CNN, height_image_C5_CNN, num_map_C5_CNN);
		double *pa = &neuron_C5[0] + addr3;
		double b = bias_C5[o];
		for (int y = 0; y < height_image_C5_CNN; y++) {
			for (int x = 0; x < width_image_C5_CNN; x++) {
				pa[y * width_image_C5_CNN + x] += b;
			}
		}
	}

	for (int i = 0; i < num_neuron_C5_CNN; i++) {
		neuron_C5[i] = activation_function_tanh(neuron_C5[i]);
	}

	return true;
}

(7)、輸出層：神經元數為(1*1)*10=10。為全連接層。輸出層中的每個神經元均是由C5層中的120個神經元乘以相相應的權值。然后相加求和；然后對每個神經元加上一個閾值。最后再通過tanh激活函數對每一神經元進行運算得到終於每個神經元的結果。

代碼段例如以下：

bool CNN::Forward_output()
{
	init_variable(neuron_output, 0.0, num_neuron_output_CNN);

	for (int i = 0; i < num_neuron_output_CNN; i++) {
		neuron_output[i] = 0.0;

		for (int c = 0; c < num_neuron_C5_CNN; c++) {
			neuron_output[i] += weight_output[c * num_neuron_output_CNN + i] * neuron_C5[c];
		}

		neuron_output[i] += bias_output[i];
	}

	for (int i = 0; i < num_neuron_output_CNN; i++) {
		neuron_output[i] = activation_function_tanh(neuron_output[i]);
	}

	return true;
}

4.        反向傳播：主要計算每層權值和偏置的誤差以及每層神經元的誤差；當中輸入層、S2層、S4層操作過程同樣。C1層、C3層操作過程同樣。

(1)、輸出層：計算輸出層神經元誤差；通過mse損失函數的導數函數和tanh激活函數的導數函數來計算輸出層神經元誤差，即a、已計算出的輸出層神經元值減去相應label值，b、1.0減去輸出層神經元值的平方，c、a與c的乘積和。

損失函數作用：在統計學中損失函數是一種衡量損失和錯誤(這樣的損失與”錯誤地”預計有關)程度的函數。損失函數在實踐中最重要的運用。在於協助我們通過過程的改善而持續降低目標值的變異。並不是只追求符合邏輯。

在深度學習中，對於損失函數的收斂特性。我們期望是當誤差越大的時候。收斂(學習)速度應該越快。成為損失函數須要滿足兩點要求：非負性；預測值和期望值接近時，函數值趨於0.

代碼段例如以下：

double CNN::loss_function_mse_derivative(double y, double t)
{
	return (y - t);
}

void CNN::loss_function_gradient(const double* y, const double* t, double* dst, int len)
{
	for (int i = 0; i < len; i++) {
		dst[i] = loss_function_mse_derivative(y[i], t[i]);
	}
}

double CNN::activation_function_tanh_derivative(double x)
{
	return (1.0 - x * x);
}

double CNN::dot_product(const double* s1, const double* s2, int len)
{
	double result = 0.0;

	for (int i = 0; i < len; i++) {
		result += s1[i] * s2[i];
	}

	return result;
}

bool CNN::Backward_output()
{
	init_variable(delta_neuron_output, 0.0, num_neuron_output_CNN);

	double dE_dy[num_neuron_output_CNN];
	init_variable(dE_dy, 0.0, num_neuron_output_CNN);
	loss_function_gradient(neuron_output, data_single_label, dE_dy, num_neuron_output_CNN); // 損失函數: mean squared error(均方差)
	
	// delta = dE/da = (dE/dy) * (dy/da)
	for (int i = 0; i < num_neuron_output_CNN; i++) {
		double dy_da[num_neuron_output_CNN];
		init_variable(dy_da, 0.0, num_neuron_output_CNN);

		dy_da[i] = activation_function_tanh_derivative(neuron_output[i]);
		delta_neuron_output[i] = dot_product(dE_dy, dy_da, num_neuron_output_CNN);
	}

	return true;
}

(2)、C5層：計算C5層神經元誤差、輸出層權值誤差、輸出層偏置誤差；通過輸出層神經元誤差乘以輸出層權值。求和。結果再乘以C5層神經元的tanh激活函數的導數(即1-C5層神經元值的平方)。獲得C5層每個神經元誤差。通過輸出層神經元誤差乘以C5層神經元獲得輸出層權值誤差；輸出層偏置誤差即為輸出層神經元誤差。

代碼段例如以下：

bool CNN::muladd(const double* src, double c, int len, double* dst)
{
	for (int i = 0; i < len; i++) {
		dst[i] += (src[i] * c);
	}

	return true;
}

bool CNN::Backward_C5()
{
	init_variable(delta_neuron_C5, 0.0, num_neuron_C5_CNN);
	init_variable(delta_weight_output, 0.0, len_weight_output_CNN);
	init_variable(delta_bias_output, 0.0, len_bias_output_CNN);

	for (int c = 0; c < num_neuron_C5_CNN; c++) {
		// propagate delta to previous layer
		// prev_delta[c] += current_delta[r] * W_[c * out_size_ + r]
		delta_neuron_C5[c] = dot_product(&delta_neuron_output[0], &weight_output[c * num_neuron_output_CNN], num_neuron_output_CNN);
		delta_neuron_C5[c] *= activation_function_tanh_derivative(neuron_C5[c]);
	}

	// accumulate weight-step using delta
	// dW[c * out_size + i] += current_delta[i] * prev_out[c]
	for (int c = 0; c < num_neuron_C5_CNN; c++) {
		muladd(&delta_neuron_output[0], neuron_C5[c], num_neuron_output_CNN, &delta_weight_output[0] + c * num_neuron_output_CNN);
	}

	for (int i = 0; i < len_bias_output_CNN; i++) {
		delta_bias_output[i] += delta_neuron_output[i];
	}

	return true;
}

(3)、S4層：計算S4層神經元誤差、C5層權值誤差、C5層偏置誤差；通過C5層權值乘以C5層神經元誤差。求和。結果再乘以S4層神經元的tanh激活函數的導數(即1-S4神經元的平方)，獲得S4層每個神經元誤差。通過S4層神經元乘以C5層神經元誤差，求和，獲得C5層權值誤差。C5層偏置誤差即為C5層神經元誤差。

代碼段例如以下：

bool CNN::Backward_S4()
{
	init_variable(delta_neuron_S4, 0.0, num_neuron_S4_CNN);
	init_variable(delta_weight_C5, 0.0, len_weight_C5_CNN);
	init_variable(delta_bias_C5, 0.0, len_bias_C5_CNN);

	// propagate delta to previous layer
	for (int inc = 0; inc < num_map_S4_CNN; inc++) {
		for (int outc = 0; outc < num_map_C5_CNN; outc++) {
			int addr1 = get_index(0, 0, num_map_S4_CNN * outc + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_S4_CNN * num_map_C5_CNN);
			int addr2 = get_index(0, 0, outc, width_image_C5_CNN, height_image_C5_CNN, num_map_C5_CNN);
			int addr3 = get_index(0, 0, inc, width_image_S4_CNN, height_image_S4_CNN, num_map_S4_CNN);

			const double* pw = &weight_C5[0] + addr1;
			const double* pdelta_src = &delta_neuron_C5[0] + addr2;
			double* pdelta_dst = &delta_neuron_S4[0] + addr3;

			for (int y = 0; y < height_image_C5_CNN; y++) {
				for (int x = 0; x < width_image_C5_CNN; x++) {
					const double* ppw = pw;
					const double ppdelta_src = pdelta_src[y * width_image_C5_CNN + x];
					double* ppdelta_dst = pdelta_dst + y * width_image_S4_CNN + x;

					for (int wy = 0; wy < height_kernel_conv_CNN; wy++) {
						for (int wx = 0; wx < width_kernel_conv_CNN; wx++) {
							ppdelta_dst[wy * width_image_S4_CNN + wx] += *ppw++ * ppdelta_src;
						}
					}
				}
			}
		}
	}

	for (int i = 0; i < num_neuron_S4_CNN; i++) {
		delta_neuron_S4[i] *= activation_function_tanh_derivative(neuron_S4[i]);
	}

	// accumulate dw
	for (int inc = 0; inc < num_map_S4_CNN; inc++) {
		for (int outc = 0; outc < num_map_C5_CNN; outc++) {
			for (int wy = 0; wy < height_kernel_conv_CNN; wy++) {
				for (int wx = 0; wx < width_kernel_conv_CNN; wx++) {
					int addr1 = get_index(wx, wy, inc, width_image_S4_CNN, height_image_S4_CNN, num_map_S4_CNN);
					int addr2 = get_index(0, 0, outc, width_image_C5_CNN, height_image_C5_CNN, num_map_C5_CNN);
					int addr3 = get_index(wx, wy, num_map_S4_CNN * outc + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_S4_CNN * num_map_C5_CNN);

					double dst = 0.0;
					const double* prevo = &neuron_S4[0] + addr1;
					const double* delta = &delta_neuron_C5[0] + addr2;

					for (int y = 0; y < height_image_C5_CNN; y++) {
						dst += dot_product(prevo + y * width_image_S4_CNN, delta + y * width_image_C5_CNN, width_image_C5_CNN);
					}

					delta_weight_C5[addr3] += dst;
				}
			}
		}
	}

	// accumulate db
	for (int outc = 0; outc < num_map_C5_CNN; outc++) {
		int addr2 = get_index(0, 0, outc, width_image_C5_CNN, height_image_C5_CNN, num_map_C5_CNN);
		const double* delta = &delta_neuron_C5[0] + addr2;

		for (int y = 0; y < height_image_C5_CNN; y++) {
			for (int x = 0; x < width_image_C5_CNN; x++) {
				delta_bias_C5[outc] += delta[y * width_image_C5_CNN + x];
			}
		}
	}

	return true;
}

(4)、C3層：計算C3層神經元誤差、S4層權值誤差、S4層偏置誤差。通過S4層權值乘以S4層神經元誤差。求和，結果再乘以C3層神經元的tanh激活函數的導數(即1-S4神經元的平方)，然后再乘以1/4。獲得C3層每個神經元誤差；通過C3層神經元乘以S4神經元誤差，求和。再乘以1/4。獲得S4層權值誤差；通過S4層神經元誤差求和，來獲得S4層偏置誤差。

代碼段例如以下：

bool CNN::Backward_C3()
{
	init_variable(delta_neuron_C3, 0.0, num_neuron_C3_CNN);
	init_variable(delta_weight_S4, 0.0, len_weight_S4_CNN);
	init_variable(delta_bias_S4, 0.0, len_bias_S4_CNN);

	double scale_factor = 1.0 / (width_kernel_pooling_CNN * height_kernel_pooling_CNN);

	assert(in2wo_C3.size() == num_neuron_C3_CNN);
	assert(weight2io_C3.size() == len_weight_S4_CNN);
	assert(bias2out_C3.size() == len_bias_S4_CNN);

	for (int i = 0; i < num_neuron_C3_CNN; i++) {
		const wo_connections& connections = in2wo_C3[i];
		double delta = 0.0;

		for (int j = 0; j < connections.size(); j++) {
			delta += weight_S4[connections[j].first] * delta_neuron_S4[connections[j].second];
		}

		delta_neuron_C3[i] = delta * scale_factor * activation_function_tanh_derivative(neuron_C3[i]);
	}

	for (int i = 0; i < len_weight_S4_CNN; i++) {
		const io_connections& connections = weight2io_C3[i];
		double diff = 0;

		for (int j = 0; j < connections.size(); j++) {
			diff += neuron_C3[connections[j].first] * delta_neuron_S4[connections[j].second];
		}

		delta_weight_S4[i] += diff * scale_factor;
	}

	for (int i = 0; i < len_bias_S4_CNN; i++) {
		const std::vector<int>& outs = bias2out_C3[i];
		double diff = 0;

		for (int o = 0; o < outs.size(); o++) {
			diff += delta_neuron_S4[outs[o]];
		}

		delta_bias_S4[i] += diff;
	}

	return true;
}

(5)、S2層：計算S2層神經元誤差、C3層權值誤差、C3層偏置誤差。通過C3層權值乘以C3層神經元誤差。求和，結果再乘以S2層神經元的tanh激活函數的導數(即1-S2神經元的平方)，獲得S2層每個神經元誤差；通過S2層神經元乘以C3層神經元誤差。求和，獲得C3層權值誤差；C3層偏置誤差即為C3層神經元誤差和。

代碼段例如以下：

bool CNN::Backward_S2()
{
	init_variable(delta_neuron_S2, 0.0, num_neuron_S2_CNN);
	init_variable(delta_weight_C3, 0.0, len_weight_C3_CNN);
	init_variable(delta_bias_C3, 0.0, len_bias_C3_CNN);

	// propagate delta to previous layer
	for (int inc = 0; inc < num_map_S2_CNN; inc++) {
		for (int outc = 0; outc < num_map_C3_CNN; outc++) {
			if (!tbl[inc][outc]) continue;

			int addr1 = get_index(0, 0, num_map_S2_CNN * outc + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_S2_CNN * num_map_C3_CNN);
			int addr2 = get_index(0, 0, outc, width_image_C3_CNN, height_image_C3_CNN, num_map_C3_CNN);
			int addr3 = get_index(0, 0, inc, width_image_S2_CNN, height_image_S2_CNN, num_map_S2_CNN);

			const double *pw = &weight_C3[0] + addr1;
			const double *pdelta_src = &delta_neuron_C3[0] + addr2;;
			double* pdelta_dst = &delta_neuron_S2[0] + addr3;

			for (int y = 0; y < height_image_C3_CNN; y++) {
				for (int x = 0; x < width_image_C3_CNN; x++) {
					const double* ppw = pw;
					const double ppdelta_src = pdelta_src[y * width_image_C3_CNN + x];
					double* ppdelta_dst = pdelta_dst + y * width_image_S2_CNN + x;

					for (int wy = 0; wy < height_kernel_conv_CNN; wy++) {
						for (int wx = 0; wx < width_kernel_conv_CNN; wx++) {
							ppdelta_dst[wy * width_image_S2_CNN + wx] += *ppw++ * ppdelta_src;
						}
					}
				}
			}
		}
	}

	for (int i = 0; i < num_neuron_S2_CNN; i++) {
		delta_neuron_S2[i] *= activation_function_tanh_derivative(neuron_S2[i]);
	}

	// accumulate dw
	for (int inc = 0; inc < num_map_S2_CNN; inc++) {
		for (int outc = 0; outc < num_map_C3_CNN; outc++) {
			if (!tbl[inc][outc]) continue;

			for (int wy = 0; wy < height_kernel_conv_CNN; wy++) {
				for (int wx = 0; wx < width_kernel_conv_CNN; wx++) {
					int addr1 = get_index(wx, wy, inc, width_image_S2_CNN, height_image_S2_CNN, num_map_S2_CNN);
					int addr2 = get_index(0, 0, outc, width_image_C3_CNN, height_image_C3_CNN, num_map_C3_CNN);
					int addr3 = get_index(wx, wy, num_map_S2_CNN * outc + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_S2_CNN * num_map_C3_CNN);
					
					double dst = 0.0;
					const double* prevo = &neuron_S2[0] + addr1;
					const double* delta = &delta_neuron_C3[0] + addr2;

					for (int y = 0; y < height_image_C3_CNN; y++) {
						dst += dot_product(prevo + y * width_image_S2_CNN, delta + y * width_image_C3_CNN, width_image_C3_CNN);
					}

					delta_weight_C3[addr3] += dst;
				}
			}
		}
	}

	// accumulate db
	for (int outc = 0; outc < len_bias_C3_CNN; outc++) {
		int addr1 = get_index(0, 0, outc, width_image_C3_CNN, height_image_C3_CNN, num_map_C3_CNN);
		const double* delta = &delta_neuron_C3[0] + addr1;

		for (int y = 0; y < height_image_C3_CNN; y++) {
			for (int x = 0; x < width_image_C3_CNN; x++) {
				delta_bias_C3[outc] += delta[y * width_image_C3_CNN + x];
			}
		}
	}

	return true;
}

(6)、C1層：計算C1層神經元誤差、S2層權值誤差、S2層偏置誤差；通過S2層權值乘以S2層神經元誤差，求和。結果再乘以C1層神經元的tanh激活函數的導數(即1-C1神經元的平方)，然后再乘以1/4，獲得C1層每個神經元誤差；通過C1層神經元乘以S2神經元誤差，求和。再乘以1/4，獲得S2層權值誤差；通過S2層神經元誤差求和，來獲得S4層偏置誤差。

代碼段例如以下：

bool CNN::Backward_C1()
{
	init_variable(delta_neuron_C1, 0.0, num_neuron_C1_CNN);
	init_variable(delta_weight_S2, 0.0, len_weight_S2_CNN);
	init_variable(delta_bias_S2, 0.0, len_bias_S2_CNN);

	double scale_factor = 1.0 / (width_kernel_pooling_CNN * height_kernel_pooling_CNN);

	assert(in2wo_C1.size() == num_neuron_C1_CNN);
	assert(weight2io_C1.size() == len_weight_S2_CNN);
	assert(bias2out_C1.size() == len_bias_S2_CNN);

	for (int i = 0; i < num_neuron_C1_CNN; i++) {
		const wo_connections& connections = in2wo_C1[i];
		double delta = 0.0;

		for (int j = 0; j < connections.size(); j++) {
			delta += weight_S2[connections[j].first] * delta_neuron_S2[connections[j].second];
		}

		delta_neuron_C1[i] = delta * scale_factor * activation_function_tanh_derivative(neuron_C1[i]);
	}

	for (int i = 0; i < len_weight_S2_CNN; i++) {
		const io_connections& connections = weight2io_C1[i];
		double diff = 0.0;

		for (int j = 0; j < connections.size(); j++) {
			diff += neuron_C1[connections[j].first] * delta_neuron_S2[connections[j].second];
		}

		delta_weight_S2[i] += diff * scale_factor;
	}

	for (int i = 0; i < len_bias_S2_CNN; i++) {
		const std::vector<int>& outs = bias2out_C1[i];
		double diff = 0;

		for (int o = 0; o < outs.size(); o++) {
			diff += delta_neuron_S2[outs[o]];
		}

		delta_bias_S2[i] += diff;
	}

	return true;
}

(7)、輸入層：計算輸入層神經元誤差、C1層權值誤差、C1層偏置誤差；通過C1層權值乘以C1層神經元誤差。求和。結果再乘以輸入層神經元的tanh激活函數的導數(即1-輸入層神經元的平方)，獲得輸入層每個神經元誤差；通過輸入層層神經元乘以C1層神經元誤差，求和。獲得C1層權值誤差；C1層偏置誤差即為C1層神經元誤差和。

bool CNN::Backward_input()
{
	init_variable(delta_neuron_input, 0.0, num_neuron_input_CNN);
	init_variable(delta_weight_C1, 0.0, len_weight_C1_CNN);
	init_variable(delta_bias_C1, 0.0, len_bias_C1_CNN);

	// propagate delta to previous layer
	for (int inc = 0; inc < num_map_input_CNN; inc++) {
		for (int outc = 0; outc < num_map_C1_CNN; outc++) {
			int addr1 = get_index(0, 0, num_map_input_CNN * outc + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_C1_CNN);
			int addr2 = get_index(0, 0, outc, width_image_C1_CNN, height_image_C1_CNN, num_map_C1_CNN);
			int addr3 = get_index(0, 0, inc, width_image_input_CNN, height_image_input_CNN, num_map_input_CNN);

			const double* pw = &weight_C1[0] + addr1;
			const double* pdelta_src = &delta_neuron_C1[0] + addr2;
			double* pdelta_dst = &delta_neuron_input[0] + addr3;

			for (int y = 0; y < height_image_C1_CNN; y++) {
				for (int x = 0; x < width_image_C1_CNN; x++) {
					const double* ppw = pw;
					const double ppdelta_src = pdelta_src[y * width_image_C1_CNN + x];
					double* ppdelta_dst = pdelta_dst + y * width_image_input_CNN + x;

					for (int wy = 0; wy < height_kernel_conv_CNN; wy++) {
						for (int wx = 0; wx < width_kernel_conv_CNN; wx++) {
							ppdelta_dst[wy * width_image_input_CNN + wx] += *ppw++ * ppdelta_src;
						}
					}
				}
			}
		}
	}

	for (int i = 0; i < num_neuron_input_CNN; i++) {
		delta_neuron_input[i] *= activation_function_identity_derivative(data_single_image[i]/*neuron_input[i]*/);
	}

	// accumulate dw
	for (int inc = 0; inc < num_map_input_CNN; inc++) {
		for (int outc = 0; outc < num_map_C1_CNN; outc++) {
			for (int wy = 0; wy < height_kernel_conv_CNN; wy++) {
				for (int wx = 0; wx < width_kernel_conv_CNN; wx++) {
					int addr1 = get_index(wx, wy, inc, width_image_input_CNN, height_image_input_CNN, num_map_input_CNN);
					int addr2 = get_index(0, 0, outc, width_image_C1_CNN, height_image_C1_CNN, num_map_C1_CNN);
					int addr3 = get_index(wx, wy, num_map_input_CNN * outc + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_C1_CNN);

					double dst = 0.0;
					const double* prevo = data_single_image + addr1;//&neuron_input[0]
					const double* delta = &delta_neuron_C1[0] + addr2;

					for (int y = 0; y < height_image_C1_CNN; y++) {
						dst += dot_product(prevo + y * width_image_input_CNN, delta + y * width_image_C1_CNN, width_image_C1_CNN);
					}

					delta_weight_C1[addr3] += dst;
				}
			}
		}
	}

	// accumulate db
	for (int outc = 0; outc < len_bias_C1_CNN; outc++) {
		int addr1 = get_index(0, 0, outc, width_image_C1_CNN, height_image_C1_CNN, num_map_C1_CNN);
		const double* delta = &delta_neuron_C1[0] + addr1;

		for (int y = 0; y < height_image_C1_CNN; y++) {
			for (int x = 0; x < width_image_C1_CNN; x++) {
				delta_bias_C1[outc] += delta[y * width_image_C1_CNN + x];
			}
		}
	}

	return true;
}

5.        更新各層權值、偏置：通過之前計算的各層權值、各層權值誤差。各層偏置、各層偏置誤差以及學習率來更新各層權值和偏置。

代碼段例如以下：

void CNN::update_weights_bias(const double* delta, double* e_weight, double* weight, int len)
{
	for (int i = 0; i < len; i++) {
		e_weight[i] += delta[i] * delta[i];
		weight[i] -= learning_rate_CNN * delta[i] / (std::sqrt(e_weight[i]) + eps_CNN);
	}
}

bool CNN::UpdateWeights()
{
	update_weights_bias(delta_weight_C1, E_weight_C1, weight_C1, len_weight_C1_CNN);
	update_weights_bias(delta_bias_C1, E_bias_C1, bias_C1, len_bias_C1_CNN);

	update_weights_bias(delta_weight_S2, E_weight_S2, weight_S2, len_weight_S2_CNN);
	update_weights_bias(delta_bias_S2, E_bias_S2, bias_S2, len_bias_S2_CNN);

	update_weights_bias(delta_weight_C3, E_weight_C3, weight_C3, len_weight_C3_CNN);
	update_weights_bias(delta_bias_C3, E_bias_C3, bias_C3, len_bias_C3_CNN);

	update_weights_bias(delta_weight_S4, E_weight_S4, weight_S4, len_weight_S4_CNN);
	update_weights_bias(delta_bias_S4, E_bias_S4, bias_S4, len_bias_S4_CNN);

	update_weights_bias(delta_weight_C5, E_weight_C5, weight_C5, len_weight_C5_CNN);
	update_weights_bias(delta_bias_C5, E_bias_C5, bias_C5, len_bias_C5_CNN);

	update_weights_bias(delta_weight_output, E_weight_output, weight_output, len_weight_output_CNN);
	update_weights_bias(delta_bias_output, E_bias_output, bias_output, len_bias_output_CNN);

	return true;
}

6.        測試准確率是否達到要求或已達到循環次數：依次循環3至5中操作，依據訓練集數量。每循環60000次時，通過計算的權值和偏置。來對10000個測試集進行測試，假設准確率達到0.985或者達到迭代次數上限100次時。保存權值和偏置。

代碼段例如以下：

bool CNN::train()
{
	out2wi_S2.clear();
	out2bias_S2.clear();
	out2wi_S4.clear();
	out2bias_S4.clear();
	in2wo_C3.clear();
	weight2io_C3.clear();
	bias2out_C3.clear();
	in2wo_C1.clear();
	weight2io_C1.clear();
	bias2out_C1.clear();

	calc_out2wi(width_image_C1_CNN, height_image_C1_CNN, width_image_S2_CNN, height_image_S2_CNN, num_map_S2_CNN, out2wi_S2);
	calc_out2bias(width_image_S2_CNN, height_image_S2_CNN, num_map_S2_CNN, out2bias_S2);
	calc_out2wi(width_image_C3_CNN, height_image_C3_CNN, width_image_S4_CNN, height_image_S4_CNN, num_map_S4_CNN, out2wi_S4);
	calc_out2bias(width_image_S4_CNN, height_image_S4_CNN, num_map_S4_CNN, out2bias_S4);
	calc_in2wo(width_image_C3_CNN, height_image_C3_CNN, width_image_S4_CNN, height_image_S4_CNN, num_map_C3_CNN, num_map_S4_CNN, in2wo_C3);
	calc_weight2io(width_image_C3_CNN, height_image_C3_CNN, width_image_S4_CNN, height_image_S4_CNN, num_map_C3_CNN, num_map_S4_CNN, weight2io_C3);
	calc_bias2out(width_image_C3_CNN, height_image_C3_CNN, width_image_S4_CNN, height_image_S4_CNN, num_map_C3_CNN, num_map_S4_CNN, bias2out_C3);
	calc_in2wo(width_image_C1_CNN, height_image_C1_CNN, width_image_S2_CNN, height_image_S2_CNN, num_map_C1_CNN, num_map_C3_CNN, in2wo_C1);
	calc_weight2io(width_image_C1_CNN, height_image_C1_CNN, width_image_S2_CNN, height_image_S2_CNN, num_map_C1_CNN, num_map_C3_CNN, weight2io_C1);
	calc_bias2out(width_image_C1_CNN, height_image_C1_CNN, width_image_S2_CNN, height_image_S2_CNN, num_map_C1_CNN, num_map_C3_CNN, bias2out_C1);

	int iter = 0;
	for (iter = 0; iter < num_epochs_CNN; iter++) {
		std::cout << "epoch: " << iter + 1;

		for (int i = 0; i < num_patterns_train_CNN; i++) {
			data_single_image = data_input_train + i * num_neuron_input_CNN;
			data_single_label = data_output_train + i * num_neuron_output_CNN;

			Forward_C1();
			Forward_S2();
			Forward_C3();
			Forward_S4();
			Forward_C5();
			Forward_output();

			Backward_output();
			Backward_C5();
			Backward_S4();
			Backward_C3();
			Backward_S2();
			Backward_C1();
			Backward_input();

			UpdateWeights();
		}

		double accuracyRate = test();
		std::cout << ",    accuray rate: " << accuracyRate << std::endl;
		if (accuracyRate > accuracy_rate_CNN) {
			saveModelFile("E:/GitCode/NN_Test/data/cnn.model");
			std::cout << "generate cnn model" << std::endl;
			break;
		}
	}

	if (iter == num_epochs_CNN) {
		saveModelFile("E:/GitCode/NN_Test/data/cnn.model");
		std::cout << "generate cnn model" << std::endl;
	}

	return true;
}

double CNN::test()
{
	int count_accuracy = 0;

	for (int num = 0; num < num_patterns_test_CNN; num++) {
		data_single_image = data_input_test + num * num_neuron_input_CNN;
		data_single_label = data_output_test + num * num_neuron_output_CNN;

		Forward_C1();
		Forward_S2();
		Forward_C3();
		Forward_S4();
		Forward_C5();
		Forward_output();

		int pos_t = -1;
		int pos_y = -2;
		double max_value_t = -9999.0;
		double max_value_y = -9999.0;

		for (int i = 0; i < num_neuron_output_CNN; i++) {
			if (neuron_output[i] > max_value_y) {
				max_value_y = neuron_output[i];
				pos_y = i;
			}

			if (data_single_label[i] > max_value_t) {
				max_value_t = data_single_label[i];
				pos_t = i;
			}
		}

		if (pos_y == pos_t) {
			++count_accuracy;
		}

		Sleep(1);
	}

	return (count_accuracy * 1.0 / num_patterns_test_CNN);
}

7.        對輸入的圖像數據進行識別：載入已保存的權值和偏置，對輸入的數據進行識別。過程相當於前向傳播。

代碼段例如以下：

int CNN::predict(const unsigned char* data, int width, int height)
{
	assert(data && width == width_image_input_CNN && height == height_image_input_CNN);

	const double scale_min = -1;
	const double scale_max = 1;

	double tmp[width_image_input_CNN * height_image_input_CNN];
	for (int y = 0; y < height; y++) {
		for (int x = 0; x < width; x++) {
			tmp[y * width + x] = (data[y * width + x] / 255.0) * (scale_max - scale_min) + scale_min;
		}
	}

	data_single_image = &tmp[0];

	Forward_C1();
	Forward_S2();
	Forward_C3();
	Forward_S4();
	Forward_C5();
	Forward_output();

	int pos = -1;
	double max_value = -9999.0;

	for (int i = 0; i < num_neuron_output_CNN; i++) {
		if (neuron_output[i] > max_value) {
			max_value = neuron_output[i];
			pos = i;
		}
	}

	return pos;
}

GitHub： https://github.com/fengbingchun/NN_Test