Tensorflow API解讀

目錄

---

 class tf.Graph

 class tf.train.RMSPropOptimizer

 class tf.GraphKeys

 class tf.train.ExponentialMovingAverage

---

 tf.get_variable

 tf.train.global_step

 tf.train.exponential_decay

 tf.get_collection

 tf.global_variables_initializer()

 tf.group tf.tuple

 tf.ConfigProto

 tf.train.get_checkpoint_state

 tf.concat()

 tf.reduce_mean()

 tf.expand_dims

 tf.gfile.Glob

 tf.sparse_to_dense()

---

 

 

 

 

 

 

 

 

 

Class Graph     返回目錄

---

 定義

 tensorflow是數據流圖，所有的運算都要預先定義在圖上

若沒有指定圖，程序會有一個默認的注冊圖，可以通過 tf.get_Default_graph 得到當前的默認圖，若要在當前默認圖中加上新的運算，只需要簡單地進行定義就好了

 tf.Graph.as_default() 是一個上下文管理器，這個函數會覆蓋當前的默認圖在指定的上下文中重新定義默認圖

 

在圖上定義的節點可以通過collection一次性取出變量集合

 

Properties

collections　　Returns the names of the collections known to this graph.

Methods

add_to_collection(name,value)  　　　Stores value in the collection with the given name, Note that collections are not sets, so it is possible to add a value to a collection several times.

　　

add_to_collections(names,value)  　　 把value加入到多個collection里面

as_default()　　returns a context manager that makes this Graph the default graph

　　

 

control_dependencies(control_inputs)　　returns a context manager that specifies control dependencies

　　

device(device_name_or_function)  　　returns a context manager that specifies the default device to use

　　

　　None起到了消除當前設置的作用

finalize()　　Finalizes this graph, making it reaad-only, no new oprations can be added to graph

get_all_collection_keys()　　Returns a list of collections used in this graph

get_collection(name,scope=None)　　Returns a list of values in the collection with the given name

　　

get_name_scope()　　Returns the current name scope

name_scope(name)　　Returns a context manager that creates hierarchical names for operations

　　

Utility functions (Wrapper)

• tf.device()        tf.Graph().device()
• tf.name_scope       tf.Graph().name_scope()
• tf.control_dependencies()　　　tf.Graph().control_dependencies()
• tf.get_default_graph()      
• tf.add_to_collection()　　 tf.Graph().add_to_collection()
• tf.get_collection()　　　　tf.Graph().get_collection()
• tf.GraphKeys()   　　　　

 

 

 

 

 

tf.get_variable()   返回目錄

---

 

 Gets an existing variable with these parameters or create a new one, This function prefixes the name with the current variable scope and performs reuse checks

對於reuse=True的變量可以通過名字獲取該變量，如果沒有名字的話就會報錯，想要用get_variable創建新變量的話就需要在reuse=False的環境下

def foo():
    with tf.variable_scope("foo",reuse=True):
        v = tf.get_variable("v",[1])
    return v
v1 = foo()    

def foo():
    with tf.variable_scope("foo",reuse=False):
        v = tf.get_variable("v",[1])
    return v
v1 = foo()
v2 = foo()       

• 當tf.get_variable_scope().reuse = = False時，作用域就是為創建新變量所設置的
• 當tf.get_variable_scope().reuse = = True時，作用域是為重用變量所設置

想要解決上面的問題，由兩種方法，一個自動改，一個手動改

在Tensorflow1.4有定義reuse=tf.AUTOREUSE 就可以同時兼顧這兩種情況

 

不用上面的方法，就要手動修改reuse的狀態

import tensorflow as tf                        
                                               
def foo():                                     
    with tf.variable_scope("foo"):             
        v = tf.get_variable("v",[1])           
    return v                                   
v1 = foo()                                     
tf.get_variable_scope().reuse_variables()      
v2 = foo()                                     
print(v1==v2)                                  
print(v1.name,v2.name)                         
                                               
def foo2():                                    
    with tf.variable_scope("foo2",reuse=True): 
        v = tf.Variable(3,name="v1")           
    return v                                   
v1 = foo2()                                    
v2 = foo2()                                    
print(v1==v2)                                  
print(v1.name,v2.name)                         
                                               
v1 = tf.Variable(3,name="v")                   
v2 = tf.Variable(3,name="v")                   
print(v1==v2)                                  
print(v1.name,v2.name)                         

 

上面這種方式會使tf.get_variable_scope().reuse_variables()后面所有的作用域都可以reuse變量，所有不可取，通常都是在一個上下文管理器內部使用，如下：

with tf.variable_scope("image_filters") as scope:
    result1 = my_image_filter(image1)
    scope.reuse_variables()
    result2 = my_image_filter(image2)

 tf.Variable初始化如下：

conv1_weights = tf.Variable(tf.random_normal([5, 5, 32, 32]),name="conv1_weights")
conv1_biases = tf.Variable(tf.zeros([32]), name="conv1_biases")

 

tf.get_variable()初始化是要在參數里設置初始化器的：

• tf.constant_initializer(value) initializes everything to the provided value, 常量初始化
• tf.random_uniform_initializer(a, b) initializes uniformly from [a, b], 均勻分布
• tf.random_normal_initializer(mean, stddev) initializes from the normal distribution with the given mean and standard deviation.正態分布

weights = tf.get_variable("weights", kernel_shape,initializer=tf.random_normal_initializer())
biases = tf.get_variable("biases", bias_shape,initializer=tf.constant_initializer(0.0))

 

 

 

 

tf.train.global_step()     返回目錄

---

global_step refer to the number of batches seen by the graph. Everytime a batch is provided, the weights are updated in the direction that minimizes the loss. global_step just keeps track of the number of batches seen so far. 

先要創建一個變量來存放global_step的值，這時，用tf.Variable創建就可以，10是初始值， trainable一定要設置成False，這樣梯度傳播時就不會修改global_step,注意name='global_step'一定不能變，因為這已經放在了系統變量中，global_step_tensor取名是可以隨意取的，一般都是直接取名為global_step;

 tf.train.global_step()用來獲取當前sess的global_step值

建立global_step變量的方法如下：

global_step = tf.Variable(0,trainable=False,name='global_step')
global_step = tf.get_variable('global_step',[],initializer=tf.constant_initializer(0),trainable=False)

 本來是需要手動對global_step進行+1操作，但是在實際應用中，操作會在'tf.train.Optimizer.apply_gradients'內部完成   如果是使用minimize，注意要將global_step作為參數傳入minimize

global_step = tf.Variable(0,trainable=False)
increment_op = tf.assign_add(global_step,tf.constant(1))
sess = tf.Session()
init = tf.global_variables_initializer()
sess.run(init)
for step in range(0,10):
    ....
    sess.run(increment_op)

 

 

 

tf.train.exponential_decay()   返回目錄

---

指數衰減公式如下：

staircase=True 表示 global_step/decay_steps 是整數操作，所以衰減圖像時階梯狀的   decay_steps表示經過多少步衰減一次

 

這個唯一問題就是只能等距衰減

要想不等距衰減學習率，只能手動設置if-else賦值了

 

 

 

 

class tf.train.RMSPropOptimizer()    返回目錄

---

 

 

This method simply combines calls compute_gradients() and apply_gradients(). If you want to process the gradient before applying them call compute_gradients() and apply_gradients() explicitly instead of using this function.

minimize = compute_gradients() + apply_gradients()

 

This is the first part of minimize(). It returns a list of (gradient, variable) pairs where "gradient" is the gradient for "variable". Note that "gradient" can be a Tensor, an IndexedSlices, or None if there is no gradient for the given variable.

This is the second part of minimize(). It returns an Operation that applies gradients.

 

 

 

 

 

tf.get_collection()   返回目錄

---

 

Wrapper for Graph.get_collection() using the default graph

Return a list of values in the collection with the given name.

就是把圖中是屬於key這一類的變量一塊取出來，常見的key有tf.GraphKeys這個類里面的

 

tf.GraphKeys  返回目錄

---

 Standard names to use for graph collections

• tf.GraphKeys.GLOBAL_VARIABLES:  所有 Variable objects
• tf.GraphKeys.LOCAL_VARIABLES： local to each machine, subset of Variable
• tf.GraphKeys.MODEL_VARIABLES:  Variable objects used in the model for inference( feed forward)
• tf.GraphKeys.TRAINABLE_VARIABLES:  Variable objects that will be trained by an optimizer (tf.Variable(trainable=True) default is True)
• tf.GraphKeys.SUMMARIES:  the summary tensor objects that have been created in the graph, 在圖上定義的所有的summary對象    tf.summary.merge_all(key=tf.GraphKeys.SUMMARIES)
• tf.GraphKeys.MOVING_AVERAGE_VARIABLES: all variables that maintain their moving averages, if ExponetialMovingAverage object is created and apply() method is called on a list of variables,these variables will be added to the collection
• tf.GraphKeys.REGULARIZATION_LOSSES: regularization losses collected during graph construction
• tf.GraphKeys.LOSSES:   tf.losses.add_loss(loss,loss_collection=tf.GraphKeys.LOSSES)    tf.losses.get_losses(scope=None,loss_collection=tf.GraphKeys.LOSSES)
• tf.GraphKeys.UPDATE_OPS   batch_norm中的moving_mean and moving_variance的更新操作(滑動平均)是要tf.add_to_collection(tf.GraphKeys.UPDATE_OPS,update_moving_mean),然后作為train_op的denpendency

 可以通過wrapper方便獲取以上collection  

• tf.trainable_variables() :  tf.GraphKeys.TRAINABLE_VARIABLES   訓練時，更新其中所有變量
• tf.moving_average_variables() :  tf.GraphKeys.MOVING_AVERAGE_VARIABLES  當ExponentialMovingAverage 的apply方法應用到變量上時，那些變量都會自動加入到MOVING_AVERAGE_VARIABLES中

還可以自定義collection的名字，比如

tf.add_to_collection("losses",l1)
tf.add_to_collection("losses",l2)
losses = tf.get_collection("losses")
loss_total = tf.add_n(losses)

 

tf.global_variables_initializer()   返回目錄

---

在整個session運行之前，圖中所有的變量Variable必須被初始化

sess = tf.Session()
init = tf.global_variables_initializer()
sess.run(init)

在執行初始化后，Variable中的值生成完畢，不會再變化

 

 

 

class tf.train.ExponentailMovingAverage   返回目錄

---

 

When training a model, it is often beneficial to maintain moving averages of the trained parameters. Evaluations that use averaged parameters sometimes produce significantly better results than the final trained values.

The apply() method adds shadow copies of trained variables and add ops that maintain a moving average of the trained variables in their shadow copies. It is used when building the training model. The ops that maintain moving averages are typically run after each training step. The average() and average_name() methods give access to the shadow variables and their names. They are useful when building an evaluation model, or when restoring a model from a checkpoint file. They help use the moving averages in place of the last trained values for evaluations.

就是說對某些變量在訓練過程中就創建一個影子變量，訓練過程不斷更新它的結果，之后用於test階段（就是保存變量的統計平均值）

shadowVariable = decay*shadowVariable + (1-decay)*variable

shadow variables are created with trainalbe=False  用其來存放ema的值 

有兩種在測試階段使用moving averages的方法

• use the average() method which returns the shadow variable for a given variable
• load the checkpoint files by shadow variable names: using average_name()

checkpoint文件中保存的變量名是shadow_var0_name,從文件中找到該變量然后載入到var0變量中

tf.train.ExponentialMovingAverage.variables_to_restore(moving_avg_variables=None)

Returns a map of names to Variables to restore.

Args:

moving_avg_variables: a list of variables that require to use of the moving variable name to be restored. If None, it will default to variables.moving_average_variables() + variables.trainable_variables()

 

 

tf.group()  tf.tuple()  返回目錄

---

    

當有很多tensor或op想要一起run，這兩個函數很方便

w = tf.Variable(1)
mul = tf.multiply(w, 2)
add = tf.add(w, 2)
group = tf.group(mul, add)
tuple = tf.tuple([mul, add])

# sess.run(group)和sess.run(tuple)都會求Tensor(add)，Tensor(mul)的值。

區別是，tf.group()返回的是`op`。tf.tuple()返回的是list of tensor。

#這樣就會導致，sess.run(tuple)的時候，會返回 Tensor(mul),Tensor(add)的值.而 sess.run(group)不會

可以認為group只是將多個op結點連接到一個結點上，當run這個結點時，其他的結點都要run，不會返回所有結點的結果

而tuple只是作為一個列表進行同時運行

注意： tf.group(*[mul,add])需要把括號去掉， tf.tuple([mul,add])需要括號

 

 

 

tf.ConfigProto()   返回目錄

---

# Using the `close()` method.
sess = tf.Session()
sess.run(...)
sess.close()

# Using the context manager.
with tf.Session() as sess:
  sess.run(...)

 

 

 

 

tf.train.get_checkpoint_state()     返回目錄

---

 

Returns CheckpointState proto from the "checkpoint" file.

If the "checkpoint" file contains a valid CheckpointState proto, returns it.

ckpt = tf.train.get_checkpoint_state(FLAGS.pretrained_model_checkpoint_path)
if os.path.isabs(ckpt.model_checkpoint_path):
    saver.restore(sess,ckpt.model_checkpoint_path)
else:         
5 　　saver.restore(sess,os.path.join(FLAGS.pretrained_model_checkpoint_path,ckpt.model_checkpoint_path))

tf.train.get_checkpoint_state是個class，它的屬性ckpt.model_checkpoint_path是在checkpoint文件的第一行

其實直接saver.restore(sess,"/home/ling.hu/imagenet_model/inception_log/lsun_train/model.ckpt-129999")

 

 

 

 

tf.concat()     返回目錄

---

 

concat的 axis方向維度concat后增加，非axis方向維度不變

 

 

 

tf.reduce_mean()   返回目錄

---

 

對axis方向所有的元素進行求平均，axis方向維度變為1，其他方向不變

keep_dims=False, the rank is reduced by 1;  True的話，retains reduced dimensions with length 1

 

 

 

 

 

tf.expand_dims()   返回目錄

---

 對tensor進行增加維度，但是不增加元素，  要增加元素的增加維度就用concat

Inserts a dimension of 1 into a tensor's shape

 

 

 

 

 

tf.gfile.Glob()   返回目錄

---

 

 def data_files(self):                                                      
   """Returns a python list of all (sharded) data subset files.             
                                                                            
   Returns:                                                                 
     python list of all (sharded) data set files.                           
   Raises:                                                                  
     ValueError: if there are not data_files matching the subset.           
   """                                                                      
   tf_record_pattern = os.path.join(FLAGS.data_dir, '%s-*' % self.subset)   
   data_files = tf.gfile.Glob(tf_record_pattern)                            
   if not data_files:                                                       
     print('No files found for dataset %s/%s at %s' % (self.name,           
                                                       self.subset,         
                                                       FLAGS.data_dir))                                                                
   return data_files                                                        

 

 tf.sparse_to_dense    返回目錄

---

sparse_to_dense(
    sparse_indices,
    output_shape,
    sparse_values,
    default_value=0,
    validate_indices=True,
    name=None
)

# If sparse_indices is scalar
dense[i] = (i == sparse_indices ? sparse_values : default_value)

# If sparse_indices is a vector, then for each i
dense[sparse_indices[i]] = sparse_values[i]

# If sparse_indices is an n by d matrix, then for each i in [0, n)
dense[sparse_indices[i][0], ..., sparse_indices[i][d-1]] = sparse_values[i]

Args:

• sparse_indices: A 0-D, 1-D, or 2-D Tensor of type int32 or int64. sparse_indices[i] contains the complete index where sparse_values[i] will be placed.
• output_shape: A 1-D Tensor of the same type as sparse_indices. Shape of the dense output tensor.
• sparse_values: A 0-D or 1-D Tensor. Values corresponding to each row of sparse_indices, or a scalar value to be used for all sparse indices.
• default_value: A 0-D Tensor of the same type as sparse_values. Value to set for indices not specified in sparse_indices. Defaults to zero.
• validate_indices: A boolean value. If True, indices are checked to make sure they are sorted in lexicographic order and that there are no repeats.
• name: A name for the operation (optional).

 

常用於將數字標簽轉換成onehot標簽

為什么要使用tf.expand_dims擴維：

t1=tf.constant([1,2,3])  
t2=tf.constant([4,5,6])  
#concated = tf.concat(1, [t1,t2])這樣會報錯  
t1=tf.expand_dims(tf.constant([1,2,3]),1)  
t2=tf.expand_dims(tf.constant([4,5,6]),1)  
concated = tf.concat(1, [t1,t2])#這樣就是正確的  

使用tf.expand_dims(vectore,1)將行向量vector轉換成一個nx1的列向量

BATCHSIZE=6  
label=tf.expand_dims(tf.constant([0,2,3,6,7,9]),1)
index=tf.expand_dims(tf.range(0, BATCHSIZE),1)  
concated = tf.concat(1, [index, label])  

結果如下：

onehot_labels = tf.sparse_to_dense(concated, tf.pack([BATCHSIZE,10]), 1.0, 0.0)  

 

 最后一步，調用tf.sparse_to_dense輸出一個onehot標簽的矩陣，輸出的shape就是行數為BATCHSIZE，列數為10的矩陣，指定元素值為1.0，其余元素值為0.0