一、tf.nn.dynamic_rnn ：函数使用和输出
官网：https://www.tensorflow.org/api_docs/python/tf/nn/dynamic_rnn
使用说明：
Args:
cell:
An instance of RNNCell. //自己定义的cell 内容：BasicLSTMCell，BasicRNNCell，GRUCell 等，，，
- inputs：
If time_major == False (default), this must be a Tensor of shape: [batch_size, max_time, ...], or a nested tuple of such elements.
If time_major == True, this must be a Tensor of shape:[max_time, batch_size, ...], or a nested tuple of such elements.
#如果是time_major=True，input的维度是[max_time, batch_size, input_size]，反之就是[batch_size, max_time, input_zise]；
time_major:
The shape format of the inputs and outputs Tensors. If true, these Tensors must be shaped [max_time, batch_size, depth]. If false, these Tensors must be shaped [batch_size, max_time, depth].
Using time_major = True is a bit more efficient because it avoids transposes at the beginning and end of the RNN calculation. However, most TensorFlow data is batch-major, so by default this function accepts input and emits output in batch-major form.
return：
- outputs: The RNN output Tensor.
If time_major == False (default), this will be a Tensor shaped: [batch_size, max_time, cell.output_size].
If time_major == True, this will be a Tensor shaped: [max_time, batch_size, cell.output_size].
#如果是time_major=True，output的维度是[max_time, batch_size, Hidden_size]，反之就是[batch_size, max_time, Hidden_size]；
- state: The final state.
If cell.state_size is an int, this will be shaped [batch_size, cell.state_size].
If it is a TensorShape, this will be shaped [batch_size] + cell.state_size. If it is a (possibly nested) tuple of ints or TensorShape, this will be a tuple having the corresponding shapes.
If cells are LSTMCells state will be a tuple containing a LSTM StateTuple for each cell.
返回值：
然后经过LSTM或者GRU产生两个输出，output和state；
output里面，包含了所有时刻的输出 H；
如果cell 是 LSTM：
state里面，单层rnn的时候包含了最后一个时刻的输出 H 和 C； 多层的时候，包含，所有层的 H 和 C；
如果cell 是 GRU：
state里面，单层rnn的时候包含了最后一个时刻的输出 H；多层的时候，包含所有层的H；
#注意，执行两次run的时候相当于执行了两次图，所以得到的结果不同；
示例代码：
import tensorflow as tf batch_size = 4 input = tf.random_normal(shape=[3, batch_size, 6], dtype=tf.float32) cell = tf.nn.rnn_cell.BasicLSTMCell(10, forget_bias=1.0, state_is_tuple=True) init_state = cell.zero_state(batch_size, dtype=tf.float32) output, final_state = tf.nn.dynamic_rnn(cell, input, initial_state=init_state, time_major=True) #time_major如果是True，就表示RNN的steps用第一个维度表示，建议用这个，运行速度快一点。 #如果是False，那么输入的第二个维度就是steps。 #如果是True，output的维度是[steps, batch_size, depth]，反之就是[batch_size, max_time, depth]。就是和输入是一样的 #final_state就是整个LSTM输出的最终的状态，包含c和h。c和h的维度都是[batch_size， n_hidden] with tf.Session() as sess: sess.run(tf.global_variables_initializer()) #print(sess.run(output)) #print(sess.run(final_state)) print(sess.run([output,final_state]))
输出比较：
由输出可以看出来，这个output 包含了隐含层所有时刻的输出，如果加层的话，那么这个output 的每个时刻，就作为下一层每个时刻的输入；
LSTM 中 state的输出包含了C和H，两者代表的都是当前层的最后一时刻的输出，H和output的最后一个时刻值一样；
输出如下：
[array([[[ 0.11201711, 0.05266698, 0.12750182, 0.03627545, 0.02706259, -0.11562401, 0.08477378, 0.06157489, 0.07341921, 0.15011263], [-0.09552816, -0.17314027, -0.0895469 , -0.26399866, -0.36308575, 0.10537394, -0.09443868, -0.31130335, 0.0132737 , -0.12810872], [-0.00719012, 0.04438379, -0.03804718, -0.06637346, -0.02082551, 0.132549 , -0.05982352, 0.11778613, -0.09206182, 0.02547247], [ 0.14723007, 0.05410767, 0.06571447, 0.06775881, -0.03286515, 0.31600857, 0.03567648, 0.10357846, -0.0679171 , -0.00785992]], [[ 0.06683166, -0.05883167, 0.10910213, 0.05030679, 0.17738451, 0.00631482, -0.00457612, -0.03694798, 0.17743434, 0.06658468], [-0.03385706, -0.20001511, -0.05247132, -0.14611273, -0.17433529, 0.14970839, -0.07725038, -0.32652032, 0.09670977, -0.17828827], [ 0.03988864, -0.03118243, -0.09246919, 0.1831698 , -0.01006366, 0.01672944, 0.01009638, 0.10943947, -0.00420897, -0.0054652 ], [ 0.16778645, 0.08699884, 0.12571299, 0.12276714, 0.04879797, 0.10517071, 0.10341848, 0.15480027, -0.04619027, 0.11167715]], [[ 0.14293307, -0.10649989, 0.09144076, -0.03020415, 0.18182378, 0.22111537, -0.02275194, -0.14586878, 0.19310513, -0.02283864], [-0.0553881 , -0.16710383, -0.09584018, -0.06020959, -0.11862611, 0.05812657, -0.05461238, -0.21729217, 0.08961426, -0.1420837 ], [ 0.03053934, 0.02213254, -0.11577073, 0.08933022, -0.08349261, 0.044699 , 0.01332499, 0.14753158, -0.12446564, 0.00095996], [ 0.21244884, 0.11677884, 0.15352076, 0.04703464, 0.07084017, 0.04610508, 0.09713535, 0.12495688, 0.00218641, 0.17711937]]], dtype=float32),
LSTMStateTuple(
c=array([[ 0.264239 , -0.16139928, 0.25842854, -0.05938458, 0.38918033, 0.37621742, -0.06394874, -0.263255 , 0.32704324, -0.04286532], [-0.11041687, -0.3316248 , -0.21551779, -0.12425021, -0.2452825 , 0.12507899, -0.11451716, -0.40844095, 0.20570038, -0.28551656], [ 0.0634905 , 0.05425977, -0.19805768, 0.15730162, -0.14432296, 0.09046975, 0.02406704, 0.34546444, -0.22364679, 0.00243504], [ 0.40725306, 0.25660557, 0.3873769 , 0.11941462, 0.16212168, 0.10613891, 0.1803763 , 0.26139545, 0.00540481, 0.31761324]], dtype=float32),
h=array([[ 0.14293307, -0.10649989, 0.09144076, -0.03020415, 0.18182378, 0.22111537, -0.02275194, -0.14586878, 0.19310513, -0.02283864], [-0.0553881 , -0.16710383, -0.09584018, -0.06020959, -0.11862611, 0.05812657, -0.05461238, -0.21729217, 0.08961426, -0.1420837 ], [ 0.03053934, 0.02213254, -0.11577073, 0.08933022, -0.08349261, 0.044699 , 0.01332499, 0.14753158, -0.12446564, 0.00095996], [ 0.21244884, 0.11677884, 0.15352076, 0.04703464, 0.07084017, 0.04610508, 0.09713535, 0.12495688, 0.00218641, 0.17711937]], dtype=float32))]
二、tf.nn.bidirectional_dynamic_rnn 的输出：
官网：https://www.tensorflow.org/api_docs/python/tf/nn/bidirectional_dynamic_rnn
示例代码在最下面：
函数输入定义：
tf.nn.bidirectional_dynamic_rnn( cell_fw, cell_bw, inputs, sequence_length=None, initial_state_fw=None, initial_state_bw=None, dtype=None, parallel_iterations=None, swap_memory=False, time_major=False, scope=None )
输出包含两部分：A tuple （outputs，output_state）
include two tuple--->> ( (fw_outputs,bw_outputs),(fw_final_state,bw_final_state) )
输出的shape：
output, final_state = tf.nn.bidirectional_dynamic_rnn(cell_fw，cell_bw, inputs, time_major=True) final_output = tf.concat(outputs, 2) #如果只需要输出，不需要state： '''
output, final_state =
(
(fw_outputs,bw_outputs),
(fw_final_state,bw_final_state)
) = tf.nn.bidirectional_dynamic_rnn(cell_fw，cell_bw, inputs, time_major=True)
# 伪代码 输出shape if time_major == False(default) （ 不同的的设置shape 不一样）； outputs is A tuple ：（output_fw,output_bw）-->>shape ([batch_size,max_time,cell_fw.output_size]，[batch_size,max_time,cell_fw.output_size]） output_state is A tuple ：（output_state_fw,output_state_bw）-->> shape ([batch_size,max_time,cell_bw.output_size]，[batch_size,max_time,cell_bw.output_size]） output_state_fw：shape -->> shape ([batch_size, cell_fw.state_size]，[batch_size, cell_fw.state_size]) output_state_bw：shape -->> shape ([batch_size, cell_bw.state_size]，[batch_size, cell_bw.state_size]) else: outputs is A tuple ：（output_fw，output_bw）-->>shape ([max_time，batch_size，cell_fw.output_size]，[max_time，batch_size，cell_fw.output_size]） output_state is A tuple ：（output_state_fw,output_state_bw）-->> shape ([batch_size,max_time,cell_bw.output_size]，[batch_size,max_time,cell_bw.output_size]） output_state_fw：shape -->> shape ([batch_size, cell_fw.state_size]，[batch_size, cell_fw.state_size]) output_state_bw：shape -->> shape ([batch_size, cell_bw.state_size]，[batch_size, cell_bw.state_size]) '''
测试<bidirectional_dynamic_rnn>的代码和代码的输出如下：
import tensorflow as tf
batch_size = 4 input = tf.random_normal(shape=[3, batch_size, 6], dtype=tf.float32) fw_cell = tf.nn.rnn_cell.BasicLSTMCell(10, forget_bias=1.0, state_is_tuple=True) bw_cell = tf.nn.rnn_cell.BasicLSTMCell(10, forget_bias=1.0, state_is_tuple=True) init_state = fw_cell.zero_state(batch_size, dtype=tf.float32) # bi bi dynamic rnn output, final_state = tf.nn.bidirectional_dynamic_rnn(fw_cell,bw_cell, input,dtype = tf.float32, time_major=True) #time_major如果是True，就表示RNN的steps用第一个维度表示，建议用这个，运行速度快一点。 # bi dynamic rnn #如果是False，那么输入的第二个维度就是steps。 #如果是True，output的维度是[max_time, batch_size, depth]，反之就是[batch_size, max_time, depth]。就是和输入是一样的 #final_state就是整个LSTM输出的最终的状态，包含c和h。c和h的维度都是[batch_size， n_hidden] with tf.Session() as sess: sess.run(tf.global_variables_initializer()) #print(sess.run(output)) #print(sess.run(final_state)) print(sess.run([output,final_state]))
输出比较：
由输出可以看出来，这个output 包含了隐含层所有时刻前向和后向的输出，如果加层的话，那么这个output 的每个时刻，就作为下一层每个时刻的输入；
注：GRU 和 LSTM 的 state 输出不同，但是都是tuple；
如果cell 是 LSTM：
state在单层rnn中，输出包含了最后一层的，前向和后向C和H，两者代表的都是最后层的最后一时刻的输出，前后向的H，对应的output的最后一个时刻值；
state在多层rnn中，输出包含了所有层的的，前向和后向C和H，两者代表的都是每一层的最后一时刻的输出，前后向的H，对应的output每一层的最后一个时刻值（output 只保存最后一层的输出，state 保存所有层的 H和C）；
如果cell 是 GRU：
state在单层rnn中，输出包含了最后一层的，前向和后向的H，两者代表的都是最后层的最后一时刻的输出，前后向的H，对应的output的最后一个时刻值；
state在多层rnn中，输出包含了所有层的，前向和后向的H，H代表的是每一层的最后一时刻的输出，前后向的H，对应的output每一层的最后一个时刻值（output 只保存最后一层的输出，state 保存所有层的 H）；
测试的输出：
[(array([[[ 0.14106834, -0.04528632, -0.02010263, -0.04126427,
-0.0332197 , -0.00106247, 0.13835369, -0.09588441, 0.03510131, 0.05169246], [ 0.06133147, -0.0232582 , -0.05024356, 0.09498103, 0.0086002 , -0.10530724, 0.18253753, -0.07942553, 0.07153056, 0.00886667], [ 0.14753884, -0.04079055, -0.01406685, 0.13480215, 0.03796537, -0.09043769, 0.2314299 , -0.08740149, 0.06912782, -0.01267859], [ 0.06526165, 0.01146618, -0.23638128, -0.05581092, -0.11822883, -0.16410625, -0.05027798, -0.08647367, -0.08076127, 0.04061119]], [[-0.08403834, 0.00833149, 0.03032847, 0.14006507, -0.04024323, 0.17318085, -0.13764451, -0.10830184, 0.12852332, -0.07746515], [-0.09468611, -0.01417435, -0.10501526, 0.18880928, -0.01667678, -0.17388701, -0.08902372, -0.12834042, 0.08464789, -0.06793084], [ 0.19070639, -0.01584721, -0.01068342, 0.02016271, 0.07228621, -0.10173664, 0.06089543, -0.02589199, -0.10167402, 0.0052557 ], [ 0.06274109, -0.00049266, -0.08755676, 0.09687787, -0.08074448, -0.25040433, -0.01494961, -0.19038169, -0.01472963, -0.03207526]], [[ 0.08066446, -0.08768935, 0.03135343, 0.096322 , -0.03832811, 0.11857887, 0.03208668, -0.2740993 , 0.15332787, -0.1054797 ], [-0.12622547, -0.01234146, -0.053387 , 0.17855036, 0.0279602 , -0.16127944, -0.12577067, -0.10588194, 0.1634436 , -0.05892743], [ 0.17111342, -0.01833202, -0.1570774 , 0.00630365, -0.06089298, -0.23136179, -0.08443928, -0.38194925, -0.20987682, -0.02470726], [ 0.02754376, -0.03641582, 0.06693614, 0.28510216, 0.01096512, -0.19997968, 0.01686571, -0.18401314, 0.13415188, -0.11835594]]], dtype=float32),
array([[[-4.71892580e-02, 1.24903403e-01, -1.30026698e-01, 1.35595098e-01, 1.79924071e-01, -2.22253352e-02, 8.44292119e-02, -3.16204876e-02, 1.23763248e-01, -1.41852632e-01], [ 1.13655411e-01, 2.14847624e-01, 1.42422020e-01, -2.18600005e-01, 1.82050109e-01, -1.60371423e-01, -1.40698269e-01, -1.89826444e-01, -1.82936639e-01, -1.77186489e-01], [-1.81219578e-01, -1.18457131e-01, -1.19064420e-01, 1.67499259e-01, 5.41930646e-03, 5.22245020e-02, -1.44038424e-01, -1.43634990e-01, 5.55445813e-02, -1.90527022e-01], [-7.35785663e-02, -7.54526583e-03, 7.17180362e-03, -2.84532481e-03, -9.26728696e-02, -1.73879474e-01, -1.37960657e-01, -1.92156255e-01, -6.91511333e-02, -1.93823442e-01]], [[ 2.35215947e-02, 2.12839022e-01, 1.19708128e-01, 9.60118312e-04, 5.18597253e-02, -1.06155628e-03, -1.04444381e-02, 4.71496470e-02, 9.75937350e-04, -5.14771827e-02], [ 4.95758429e-02, 2.43057415e-01, 2.12201729e-01, -1.59496874e-01, 2.85923164e-02, -1.15410425e-01, -9.19163823e-02, -5.79124615e-02, -1.70543984e-01, -7.04976842e-02], [-3.71870756e-01, -3.13842781e-02, -2.12511465e-01, 4.09433812e-01, -4.14451808e-01, -4.04715101e-04, -4.27415073e-02, 3.08825504e-02, 2.47091308e-01, -1.90000907e-01], [ 1.59666702e-01, -9.14119035e-02, 2.77409274e-02, -3.35936815e-01, 9.16254967e-02, -2.70533301e-02, -1.54404104e-01, -6.44175261e-02, -1.75929859e-01, -2.02384084e-01]], [[-4.12032045e-02, 4.64685149e-02, -1.03556208e-01, 1.14612505e-01, 1.44457981e-01, 4.20663096e-02, 4.07745019e-02, -1.11312367e-01, 5.78245446e-02, -8.26596469e-02], [ 7.64411911e-02, 9.69330519e-02, 8.21158811e-02, -1.07323572e-01, 9.91004631e-02, -5.17336503e-02, -4.19655181e-02, -1.74307302e-02, -5.12403548e-02, -4.96065766e-02], [-2.42707521e-01, 1.06276445e-01, -6.91626742e-02, 4.17545497e-01, -2.72490084e-01, -1.08139351e-01, -2.92627905e-02, -1.17304660e-01, 2.38732815e-01, -2.74660379e-01], [ 1.69804603e-01, -1.16702899e-01, 2.23863665e-02, -2.18844891e-01, 7.04917312e-02, 8.52173045e-02, -1.40099138e-01, 5.31176142e-02, -8.26682746e-02, -9.91851762e-02]]], dtype=float32)),
(LSTMStateTuple(c=array([[ 0.15401003, -0.1852682 , 0.07581142, 0.2094832 , -0.10182755, 0.25072512, 0.05835846, -0.44341922, 0.30883613, -0.22787577], [-0.24870059, -0.02492736, -0.11864171, 0.35445976, 0.06311206, -0.30257928, -0.21532504, -0.22805184, 0.35281387, -0.13972323], [ 0.26539534, -0.04326108, -0.21563095, 0.02033684, -0.22373867, -0.72543144, -0.23284285, -0.6351244 , -0.38619697, -0.07513228], [ 0.06854046, -0.08422969, 0.19689892, 0.5770679 , 0.02262445, -0.48846442, 0.04160193, -0.38153723, 0.2185685 , -0.40132868]], dtype=float32), h=array([[ 0.08066446, -0.08768935, 0.03135343, 0.096322 , -0.03832811, 0.11857887, 0.03208668, -0.2740993 , 0.15332787, -0.1054797 ], [-0.12622547, -0.01234146, -0.053387 , 0.17855036, 0.0279602 , -0.16127944, -0.12577067, -0.10588194, 0.1634436 , -0.05892743], [ 0.17111342, -0.01833202, -0.1570774 , 0.00630365, -0.06089298, -0.23136179, -0.08443928, -0.38194925, -0.20987682, -0.02470726], [ 0.02754376, -0.03641582, 0.06693614, 0.28510216, 0.01096512, -0.19997968, 0.01686571, -0.18401314, 0.13415188, -0.11835594]], dtype=float32)), LSTMStateTuple(c=array([[-0.0744614 , 0.26128462, -0.2831493 , 0.26449782, 0.31356773, -0.05635487, 0.18961594, -0.06494538, 0.32900015, -0.21728653], [ 0.27273336, 0.3635042 , 0.32611233, -0.40245444, 0.40640068, -0.3703317 , -0.35593987, -0.3990611 , -0.40341747, -0.39981088], [-0.34268516, -0.20780411, -0.30328298, 0.24868692, 0.00983843, 0.1082218 , -0.2859917 , -0.24331218, 0.13357104, -0.36969435], [-0.19263096, -0.01261891, 0.01626399, -0.00499403, -0.22496712, -0.32415336, -0.4642058 , -0.39442265, -0.10230929, -0.42570713]], dtype=float32), h=array([[-0.04718926, 0.1249034 , -0.1300267 , 0.1355951 , 0.17992407, -0.02222534, 0.08442921, -0.03162049, 0.12376325, -0.14185263], [ 0.11365541, 0.21484762, 0.14242202, -0.2186 , 0.18205011, -0.16037142, -0.14069827, -0.18982644, -0.18293664, -0.17718649], [-0.18121958, -0.11845713, -0.11906442, 0.16749926, 0.00541931, 0.0522245 , -0.14403842, -0.14363499, 0.05554458, -0.19052702], [-0.07357857, -0.00754527, 0.0071718 , -0.00284532, -0.09267287, -0.17387947, -0.13796066, -0.19215626, -0.06915113, -0.19382344]], dtype=float32)))]
三、encoder 的输出转化为 decoder 的输入；
如果decoder_num_layer 不等于encoder_num_layer，可以扩展encoder的输入，或加入全连接层，再生成多个 encoder_state；
补充注：在encoder2decoder模型中，如果单向rnn 作为decoder输入的时候；state 是直接作为输入的，不需要转化； 必须保证encoder_num_layer == decoder_num_layer
补充注：在encoder2decoder模型中，如果双向rnn 作为decoder输入的时候，
是需要把前向和后向结合在一起的：
示例代码如下：
转化为decoder输入的示例代码：
以下是双向rnn 进行decoder 的时候；
output, final_state = ( (encoder_fw_outputs,encoder_bw_outputs), (encoder_fw_final_state,encoder_bw_final_state) )= tf.nn.bidirectional_dynamic_rnn(cell, input, initial_state=init_state,time_major=True) #encoder 的结果作为 decoder 输入的时候，只需要 encoder 的最后一层的隐藏层（state）； # bi_dynamic_rnnencoder 的output，双向rnn 的输入 encoder_outputs = tf.concat((encoder_fw_outputs, encoder_bw_outputs), 2) # encoder_outputs shape :[batch_size,cell_fw.size + cell_bw.size] ''' 代码不能直接用 encoder and decoder is LSTM # 需要传入 decoder 的 encoder_state encoder_final_state_c = tf.concat( (encoder_fw_final_state.c, encoder_bw_final_state.c), 1) encoder_final_state_h = tf.concat( (encoder_fw_final_state.h, encoder_bw_final_state.h), 1) # encoder_final_state is a tuple shape: ([batch_size,cell_fw_c.state_size + cell_bw_c.state_size],[batch_size,cell_fw_h.state_size+cell_bw_h.state_size]) encoder_final_state = tf.contrib.rnn.LSTMStateTuple( c=encoder_final_state_c, h=encoder_final_state_h ) ''' # 以下的代码可直接使用，num_layers=encoder_layers，encoder 和decoder 层数的设置；必须保证层数如下；
#单向encoder的时候：encoder_layers_num == decoder_layers_num #双向encoder的时候: encoder_layers_num == decoder_layers_num // 2 encoder_state = [] for i in range(num_layers): encoder_state.append(bi_state[0][i]) # forward encoder_state.append(bi_state[1][i]) # backward encoder_state = tuple(encoder_state) # 2 tuple, 2 tuple(c & h), batch_size, hidden_size
return encoder_outputs,encoder_state