碼上歡樂
相關內容    簡體    繁體

Tensorflow2.0

本文轉載自   查看原文   2019-12-23 17:33   238    tensorflow/ 深度學習

Tensorflow2.0

  1. Tensorflow 簡介

    1. Tensorflow是什么

      1. Google開源軟件庫

        1. 采用數據流圖,用於數值計算
        2. 支持多平台 GPU CPU 移動設備
        3. 最初用於深度學習,變得通用

      2. 數據流圖

        1. 節點---處理數據
        2. 線---節點之間的輸入輸出關系
        3. 線上運輸張量
        4. 節點被分配到各種計算設備上運行

      3. 特性

        1. 高度的靈活性
        2. 真正的可移植性
        3. 產品與科研結合
        4. 自動求微分
        5. 多語言支持
        6. 性能最優化

    2. 歷史

      1. 歷史版本

        1. 2015年11月:首次發布
        2. 2015年12月:支持GPU python3.3(v0.6)
        3. 2016年4月:分布式tensorflow(v0.8)
        4. 2016年11月:支持windows(v0.11)
        5. 2017年2月:性能改進,API穩定性(v1.0)
        6. 2017年4月:Keras集成(v1.1)
        7. 2017年8月:高級API,預算估計器,更多模型,初始TPU支持(v1.3)
        8. 2017年11月:Eager execution和Tensorflow Lite(v1.5)
        9. 2018年3月:TF hub(與訓練的庫),Tensorflow.js Tensorflow Extended(TFX)
        10. 2018年5月:新入門內容 Cloud TPU模塊與管道(v1.6)
        11. 2018年6月:新的分布式策略API:概率編程工具,Tensorflow Probability(v1.8)
        12. 2018年8月:Cloud Big Table 集成(v1.10)
        13. 2018年10月:側重可用性的API改進(v1.12)
        14. 2019年:tensorflow2.0

      2. Tensorflow1.0--------------主要特性

        1. XLA:Accelerate Linear Algebra
          1. 提升速度58倍
          2. 可以在移動設備運行
        2. 引入更高級的API-------tf.layers/tf.metrics/tf.losses/tf.keras
        3. Tensorflow 調試器
        4. 支持docker,引入tensorflow serving服務

      3. Tensorflow1.0---------------------架構

        Kears,Estimator,Datasets,Layers,Distribution engine

      4. Tensorflow2.0----------------------主要特性

        1. 使用tf.keras和eager mode進行更加簡單的模型構建
        2. 魯棒的跨平台部署
        3. 強大的研究實驗
        4. 清除了不推薦使用的API和減少了重復來簡化PI

      5. Tensorflow2.0------------------------架構

      6. Tensorflow2.0--------簡化的模型開發流程

        1. 使用tf.data加載數據
        2. 使用tf.keras構建模型,也可以使用premade estimator來驗證模型
          1. 使用tensorflow hub 進行遷移學習
        3. 使用eager mode 進行運行和調試
        4. 使用分發策略進行分布式訓練
        5. 導出到SavedModel
        6. 使用Tensorflow Serve,Tensorflow Lite,Tensorflow.js部署模型

      7. Tensorflow2.0-----------強大的跨平台能力

        1. Tensorflow服務
          1. 直接通過Http/REST或GRPC/協議緩沖區
        2. Tensorflow Lite----------可部署在Android ios和其他嵌入式設備
        3. Tensorflow.js------可在javascript中部署模型
        4. 其他語言
          1. c,java,go,c#等

      8. Tensorflow2.0------------強大的研究實現

        1. keras功能和子類API,允許創建一些復雜的拓撲結構
        2. 自定義訓練邏輯,使用tf.GradientTape和tf.custom_gradient進行更細力度的控制
        3. 低層API自始至終可以與高層結合使用,完全的可定制
        4. 高級擴展:Ragged Tensors,Tensor2Tensor等

    3. Tensorflow vs Pytorch

      1. 入門時間

        1. Tensorflow 1.*

          1. 靜態圖
          2. 學習額外概念
            1. 圖,會話,變量,占位符等
          3. 寫樣板代碼

        2. Tensorflow 2.*

          1. 動態圖
          2. Eager mode避免1.0缺點,直接集成在python中

        3. Pytorch

          1. 動態圖
          2. Numpy的擴展,直接集成在python中

        4. 靜態圖效率高,動態圖容易調試

        5. 代碼示例 1+\(\frac{1}{2}\)+ \(\frac{1}{2^2}\)+.....+\(\frac{1}{2^{50}}\)

          1. # python
x = 0
y = 1
for iteration in range(50):
    x = x + y
    y = y / 2
print(x)

          2. # Pytorch
import torch
x = tf.constant(0.)
y = tf.constant(1.)
for iteration in range(50):
    x = x + y
    y = y / 2
print(x)

          3. # tensorflow1.*
import tensorflow as tf
x = tf.constant(0.)
y = tf.constant(1.)
add_op = x.assgin(x+y)
div_op = y.assgin(y/2)
with tf.Session() as sess:
    sess.run(tf.golbal_variables_initializers())
    for iteration in range(50):
        sess.run(add_op)
        sess.run(dic_op)
     print(x.eval()) # sess.eval(x)

          4. # tensorflow2.*
import tensorflow as tf
x = tf.constant(0.)
y = tf.constant(1.)
for iteration in range(50)
	x = x + y
    y = y / 2
print(x.numpy())

      2. 圖創建和調試

        1. Tensorflow 1.*
          1. 靜態圖,難以調試,學習tfdbg調試
        2. Tensorflow 2.*與pytorch
          1. 動態圖,python自帶的調試工具

      3. 全面性

        1. Pythorch缺少
          1. 沿維翻轉張量(np.flip,np.flipud,np.fliplr)
          2. 檢查無窮與非數值張量(np.is_nan,np.is_inf)
          3. 快速傅里葉變換(np.fft)
        2. 隨着時間變化,越來越接近

      4. 序列化和部署

        1. Tensorflow支持更加廣泛
        2. 圖保存為protocol buffer
        3. 跨語言
        4. 跨平台
        5. Pytorch支持比較簡單

    4. 環境配置

      1. 本地配置

        1. Virtualenv 安裝或anaconda3
        2. GPU環境配置

      2. 雲端配置

        1. 為什么在雲端配置

          1. 規格統一,節省自己的機器
          2. 有直接配置好的鏡像

        2. 雲環境

          1. Google Cloud,Amazon

          2. 實戰

            1. 從0配置

              # ubuntu 18.04
sudo apt-get install python3 # 安裝python3
sudo apt-get install python # 安裝python3
sudo apt-get install software-properties-common
sudo apt-add-repository universe  # 獎pip所在源加入ubuntu
sudo apt-get update
sudo apt-get install python-pip
sudo apt-get install python3-pip
sudo pip3 install -U virtualenv # 或使用anaconda3
# 創建虛擬環境
mkdir environment && cd environment
virtualenv --system-site-packages -p python3 ./tf_py3
source tf_py3/bin/activate # 激活環境
pip install tensorflow
# pip install tensorflow==2.0.0
pip install numpy pandas matplotlib sklearn jupyter
# 配置jupyter
# 修改為靜態ip配置,端口
jupyter notebook --generate-config # 家目錄生成配置文件 .jupyter
c = get_config()
c.NotebookApp.ip = "*"
c.NotebookAPP.open_borwser = False
c.NotebookAPP.post = 6006
c.NotebookApp.allow_remote_access = True
# 可使用token設置密碼 供以后使用
deactivate # 退出環境


# 配置gpu環境
# https://tensorflow.google.cn/install/gpu
# 找到ubuntu18.04 復制為安裝腳本執行即可
# 可能會出現版本號不對應錯誤 libnvinfer-dev 將腳本中最后一條命令修改后運行libnvinfer-dev-5.1.5-1+cuda10.1 即可成功
nvidia-smi # 查看GPU信息
# tensorflow-gpu安裝與cpu版類似

      tf.test.is_gpu_available() # 判斷gpu可用與否
      ```

           2. 從鏡像配置

        ```shell
        # 雲端的系統鏡像直接有開發環境
        # 升級tensorflow 版本
        pip install --upgrade tensorflow-gpu==2.0.0
        pip3 install --upgrade tensorflow-gpu==2.0.0
        ```

  2. Tensorflow keras 實戰

    1. keras是什么

      1. 基於python的高級神經網絡API
      2. Francois Chollet2014-2015編寫
      3. 以Tensorflow CNTK,或Theano為后端運行,keras必須有后端才可以
        1. 后端可以切換,現在多用tensorflow
      4. 及方便用於快速試驗

    2. tf.keras是什么

      1. Tensorflow 對kerasAPI規范的實現
      2. 相對於tensorflow為后端的keras,Tensorflow-keras與tensorflow結合更緊密
      3. 實現在tf.keras下

    3. Tf-keras與keras的

      1. 聯系

        1. 基於同一套API
          1. keras程序可轉化為tf.keras
          2. 反之可能不成立,tf.keras有其它特性
        2. 相同的JSON與hdf5模型序列化格式和語義

      2. 區別

        1. Tf.keras全面支持eager mode
          1. 只使用keras.Sequential和keras.Model時沒影響
          2. 自定義Model內部運算邏輯時會有影響
            1. T低層API可以使用keras的model.fit等抽象法
            2. 適用於研究人員
        2. Tf.keras支持基於tf.data的模型訓練
        3. Tf.keras支持TPU訓練
        4. Tf.keras支持tf.distribution中的分布式策略
        5. 其他特性
          1. Tf.keras可以與Tensorflow中的estimator集成
          2. Tf.keras可以保存為SavedModel

    4. 知識點

      1. 分類問題與回歸問題

        1. 分類問題:輸出類型是概率分布
        2. 回歸問題:輸出是一個是數值

      2. 目標函數

        1. 參數逐步調整

        2. 目標函數幫助衡量模型好壞

        3. 分類問題

          1. 要衡量目標類別與當前預測的差距

            1. 三分類問題輸出:[0.2,0.7,0.1]
            2. 真是類別:2->ont_hot->[0,0,1]

          2. One_hot編碼,把正整數變為向量表達

            1. 生成一個長度不小於正整數的向量,只有正整數的位置處為1,其余位置都為0

          3. 平方差損失

            \[\frac{1}{n} \sum_{x,y}\frac{1}{2}(y-Model(x))^2 \]

          4. 交叉熵損失

            \[\frac{1}{n}\sum_{x,y}yln(Model(x)) \]

        4. 回歸問題

          1. 預測值與真實值的差距
          2. 平方差損失
          3. 絕對值損失

    5. Tf框架:keras,回調函數

      1. keras搭建模型

        1. # coding:utf-8
# file: tf_keras_classification_model.py
# author: Dean
# contact: 1028968939@qq.com
# time: 2019/12/17 11:47
# desc: keras模型搭建

import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import sklearn
import pandas as pd
import os,sys,time
import tensorflow as tf
from tensorflow import keras
# import keras
def showVersion():
    print((tf.__version__))
    print(sys.version_info)
    for module in mpl, np, pd, sklearn, tf, keras:
        print(module.__name__,module.__version__)

class_names = ['T-shirt','Trouser','Pullover','Dress',
               'Coat', 'Sandal', 'Shirt', 'Sneaker',
               'Bag', 'Ankle boot'
               ]

fashion_mnist = keras.datasets.fashion_mnist
(x_train_all,y_train_all),(x_test,y_test) = fashion_mnist.load_data()
x_valid, x_train = x_train_all[:5000], x_train_all[5000:]
y_valid, y_train = y_train_all[:5000], y_train_all[5000:]
def showDataShape():
    print(x_valid.shape, y_valid.shape)
    print(x_train.shape, y_train.shape)
    print(x_test.shape, y_test.shape)
def show_single_image(img_arr):
     plt.imshow(img_arr,cmap="binary")
     plt.show()
def show_images(n_rows, n_cols, x_data, y_data, class_names):
    assert len(x_data) == len(y_data)
    assert n_rows * n_cols < len(x_data)
    plt.figure(figsize = (n_cols * 1.4, n_rows * 1.6))
    for row in range(n_rows):
        for col in range(n_cols):
            index = n_cols * row + col
            plt.subplot(n_rows, n_cols, index+1)
            plt.imshow(x_data[index], cmap="binary",interpolation="nearest")
            plt.axis("off")
            plt.title(class_names[y_data[index]])
    plt.show()
def nn():
    model = keras.models.Sequential()   #  keras.Sequential() 好像與此一樣
    model.add(keras.layers.Flatten(input_shape=[28,28]))
    model.add(keras.layers.Dense(300,activation="relu"))
    model.add(keras.layers.Dense(100,activation="relu"))  # relu : y = max(0,x)
    model.add(keras.layers.Dense(10,activation="softmax"))
    """
    或
    model = keras.models.Sequential([
        keras.layers.Flatten(input_shape=[28,28]),
        keras.layers.Dense(300,activation="relu"),
        keras.layers.Dense(100,activation="relu"),
        keras.layers.Dense(10,activation="softmax")
    ])
    """
    # softmax : x = [x1, x2, x3],
    #   y = [e^x1/sum, e^x2/sum,e^x3/sum]   sum = e^x1+e^x2+e^x3
    model.compile(loss="sparse_categorical_crossentropy",
                  optimizer = "adam",  # 若loss太低,可能是算法的問題,換用優化過的梯度下降算法
                  metrics = ['accuracy'])
    # model.summary()  # 顯示模型信息
    history = model.fit(x_train,y_train,epochs=10,validation_data=(x_valid,y_valid))
    # history.history  # 中間結果 json
    return history
def plot_learning_curves(history):
    pd.DataFrame(history.history).plot(figsize=(8,5))
    plt.grid(True)
    plt.gca().set_ylim(0,1)
    plt.show()
if __name__ =="__main__":
    pass
    # show_single_image(x_train[0])
    # showVersion()
    # show_images(3,5,x_train,y_train,class_names)
    history = nn()
    plot_learning_curves(history)

        2. # coding:utf-8
# file: tf_keras_classification_model_normalizer.py
# author: Dean
# contact: 1028968939@qq.com
# time: 2019/12/17 11:47
# desc: 歸一化提高准確率

import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import sklearn
import pandas as pd
import os,sys,time
import tensorflow as tf
from tensorflow import keras
def showVersion():
    print((tf.__version__))
    print(sys.version_info)
    for module in mpl, np, pd, sklearn, tf, keras:
        print(module.__name__,module.__version__)

class_names = ['T-shirt','Trouser','Pullover','Dress',
               'Coat', 'Sandal', 'Shirt', 'Sneaker',
               'Bag', 'Ankle boot'
               ]

fashion_mnist = keras.datasets.fashion_mnist
(x_train_all,y_train_all),(x_test,y_test) = fashion_mnist.load_data()
x_valid, x_train = x_train_all[:5000], x_train_all[5000:]
y_valid, y_train = y_train_all[:5000], y_train_all[5000:]
# 歸一化處理 x = (x-u) / std   x-減去均值/方差  : 均值是0方差是1的正態分布
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
# x_train:[None,28,28] ---->[None, 784] ------>[None, 28, 28]
x_train_scaled = scaler.fit_transform(x_train.astype(np.float32).reshape(-1,1)).reshape(-1,28,28)
x_valid_scaled = scaler.transform(x_valid.astype(np.float32).reshape(-1,1)).reshape(-1,28,28)  # 使用訓練集的均值,方差
x_test_scaled = scaler.transform(x_test.astype(np.float32).reshape(-1,1)).reshape(-1,28,28)

def nn():
    model = keras.models.Sequential()   #  keras.Sequential() 好像與此一樣
    model.add(keras.layers.Flatten(input_shape=[28,28]))
    model.add(keras.layers.Dense(300,activation="relu"))
    model.add(keras.layers.Dense(100,activation="relu"))  # relu : y = max(0,x)
    model.add(keras.layers.Dense(10,activation="softmax"))

    model.compile(loss="sparse_categorical_crossentropy",
                  optimizer = "adam",  # 若loss太低,可能是算法的問題,換用優化過的梯度下降算法
                  metrics = ['accuracy'])
    history = model.fit(x_train_scaled,y_train,epochs=10,validation_data=(x_valid_scaled,y_valid))
    model.evaluate(x_test_scaled,y_test)  # 驗證集驗證
    return history

def plot_learning_curves(history):
    pd.DataFrame(history.history).plot(figsize=(8,5))
    plt.grid(True)
    plt.gca().set_ylim(0,1)
    plt.show()
if __name__ =="__main__":
    pass
    history = nn()
    plot_learning_curves(history)

      2. 回調函數

        1. module: tf.keras.callbacks(本文中只有重要的幾個)

          • EarlyStopping:提起終止訓練

          • ModelCheckpoint:每隔一段時間保存模型

          • TensorBoard:可在訓練過程中圖形化顯示

            # coding:utf-8
# file: tf_keras_classification_model_callbacks.py
# author: Dean
# contact: 1028968939@qq.com
# time: 2019/12/17 11:47
# desc: 回調函數

import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import sklearn
import pandas as pd
import os,sys,time
import tensorflow as tf
from tensorflow import keras
def showVersion():
    print((tf.__version__))
    print(sys.version_info)
    for module in mpl, np, pd, sklearn, tf, keras:
        print(module.__name__,module.__version__)

class_names = ['T-shirt','Trouser','Pullover','Dress',
               'Coat', 'Sandal', 'Shirt', 'Sneaker',
               'Bag', 'Ankle boot'
               ]

fashion_mnist = keras.datasets.fashion_mnist
(x_train_all,y_train_all),(x_test,y_test) = fashion_mnist.load_data()
x_valid, x_train = x_train_all[:5000], x_train_all[5000:]
y_valid, y_train = y_train_all[:5000], y_train_all[5000:]

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()

x_train_scaled = scaler.fit_transform(x_train.astype(np.float32).reshape(-1,1)).reshape(-1,28,28)
x_valid_scaled = scaler.transform(x_valid.astype(np.float32).reshape(-1,1)).reshape(-1,28,28)  # 使用訓練集的均值,方差
x_test_scaled = scaler.transform(x_test.astype(np.float32).reshape(-1,1)).reshape(-1,28,28)

def nn():
    model = keras.models.Sequential()   #  keras.Sequential() 好像與此一樣
    model.add(keras.layers.Flatten(input_shape=[28,28]))
    model.add(keras.layers.Dense(300,activation="relu"))
    model.add(keras.layers.Dense(100,activation="relu"))  # relu : y = max(0,x)
    model.add(keras.layers.Dense(10,activation="softmax"))

    model.compile(loss="sparse_categorical_crossentropy",
                  optimizer = "adam",  # 若loss太低,可能是算法的問題,換用優化過的梯度下降算法
                  metrics = ['accuracy'])

    # Tensorboard,EarlyStopping,ModelCheckpoint
    # 相對路徑可能會報錯，另外注意tf2.0以上版本，使用tensorboard時命令行路徑不能有中文，最好是進入事件文件所在文件夾 --logdir=. 報錯可能小，一般報錯都是因為中文路徑問題，還有就是--logdir的問題，相對，絕對路徑問題(好像是這樣，出錯時多試試即可，宗旨，少使用中文，logdir=.)
    logdir = r"D:\desktop\Workspace\PythonWorkSpace\Tensorflow2.0\Tensorflow2.0_谷歌\callbacks"
    if not os.path.exists(logdir):
        os.mkdir(logdir)
    output_model_file = os.path.join(logdir,"fashion_mnist_model.h5")
    callbacks = [
        keras.callbacks.TensorBoard(logdir),
        keras.callbacks.ModelCheckpoint(output_model_file,save_best_only = True),  # 默認保存最近一次訓練,True表示保存效果最好的
        keras.callbacks.EarlyStopping(patience = 5, min_delta = 1e-3)  # 提前結束 當閾值低於1e-3時記錄一次,5次后停止
    ]
    history = model.fit(x_train_scaled,y_train,epochs=10,
                        validation_data=(x_valid_scaled,y_valid),
                        callbacks = callbacks)
    model.evaluate(x_test_scaled,y_test)  # 驗證集驗證
    return history

def plot_learning_curves(history):
    pd.DataFrame(history.history).plot(figsize=(8,5))
    plt.grid(True)
    plt.gca().set_ylim(0,1)
    plt.show()
if __name__ =="__main__":
    history = nn()
    plot_learning_curves(history)

    6. 圖像分類,房價預測

    7. 知識點總結

      1. 分類問題,回歸問題,損失函數

      2. 神經網絡,激活函數,批量歸一化,Dropout

        1. Min-max 歸一化: \(x*=\frac{x-min}{max-min}\)

        2. Z-score歸一化: \(x*= \frac{x-\mu}{\sigma}\)

        3. 批歸一化:將輸入的歸一化擴展到每層激活值上,每層的輸出是下一層的輸入,都做歸一化

        4. 歸一化可以加速訓練,一定程度上緩解梯度消失

          # coding:utf-8
# file: tf_keras_classification_model_dnn.py
# author: Dean
# contact: 1028968939@qq.com
# time: 2019/12/17 11:47
# desc: 深度神經網絡

import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import sklearn
import pandas as pd
import os,sys,time
import tensorflow as tf
from tensorflow import keras
def showVersion():
    print((tf.__version__))
    print(sys.version_info)
    for module in mpl, np, pd, sklearn, tf, keras:
        print(module.__name__,module.__version__)

class_names = ['T-shirt','Trouser','Pullover','Dress',
               'Coat', 'Sandal', 'Shirt', 'Sneaker',
               'Bag', 'Ankle boot'
               ]

fashion_mnist = keras.datasets.fashion_mnist
(x_train_all,y_train_all),(x_test,y_test) = fashion_mnist.load_data()
x_valid, x_train = x_train_all[:5000], x_train_all[5000:]
y_valid, y_train = y_train_all[:5000], y_train_all[5000:]

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()

x_train_scaled = scaler.fit_transform(x_train.astype(np.float32).reshape(-1,1)).reshape(-1,28,28)
x_valid_scaled = scaler.transform(x_valid.astype(np.float32).reshape(-1,1)).reshape(-1,28,28)  # 使用訓練集的均值,方差
x_test_scaled = scaler.transform(x_test.astype(np.float32).reshape(-1,1)).reshape(-1,28,28)

def nn():
    model = keras.models.Sequential()
    model.add(keras.layers.Flatten(input_shape=[28,28]))
    for _ in range(20):  # 循環生成網絡
        model.add(keras.layers.Dense(100,activation="relu"))  # 若使用"selu" 是一個自帶歸一化的激活函數,使用它不用自己歸一化
        model.add(keras.layers.BatchNormalization())
        """
        對於先激活還是先歸一化,不確定,好像都可以,因此下邊寫法也對
        model.add(keras.layers.Dense(100))
        model.add(keras.layers.BatchNormalization())
        model.add(keras.layers.Activation('relu'))
        """
    model.add(keras.layers.AlphaDropout(rate=0.5))  # dropout層,更強大的dropout,drop后均值,方差不變,歸一化性質不變  因此可與selu一起使用
    # model.add(keras.layers.Dropout(rate=0.5))
    model.add(keras.layers.Dense(10,activation="softmax"))

    model.compile(loss="sparse_categorical_crossentropy",
                  optimizer = "adam",
                  metrics = ['accuracy'])
    logdir = r"D:\desktop\Workspace\PythonWorkSpace\Tensorflow2.0\Tensorflow2.0_谷歌\dnn-callbacks"
    if not os.path.exists(logdir):
        os.mkdir(logdir)
    output_model_file = os.path.join(logdir,"fashion_mnist_model.h5")
    callbacks = [
        keras.callbacks.TensorBoard(logdir),
        keras.callbacks.ModelCheckpoint(output_model_file,save_best_only = True),
        keras.callbacks.EarlyStopping(patience = 5, min_delta = 1e-3)
    ]
    history = model.fit(x_train_scaled,y_train,epochs=10,
                        validation_data=(x_valid_scaled,y_valid),
                        callbacks = callbacks)
    model.evaluate(x_test_scaled,y_test)  # 驗證集驗證
    return history

def plot_learning_curves(history):
    pd.DataFrame(history.history).plot(figsize=(8,5))
    plt.grid(True)
    plt.gca().set_ylim(0,3)
    plt.show()
if __name__ =="__main__":
    history = nn()
    plot_learning_curves(history)

      3. Wide,deep模型 超參數搜索

        1. 稀疏特征

          1. 離散值特征
          2. One-hot表示,則是稀疏特征
          3. Eg: 詞表={'你好','你',....} 你={0,1,0,0,0,0...}
          4. 可以叉乘
          5. 優點:有效,應用於工業界
          6. 缺點:
            1. 需要人工設計
            2. 可能過擬合

        2. 密集特征

          1. 向量表達
            1. Eg: 詞表={人工智能,他,你,慕課網}
            2. 他=[0.3,0.5,0.3,(維向量)]
          2. Word2vec工具
          3. 優點:
            1. 帶有語義信息,不同向量之間有相關性
            2. 兼容沒有出現過的特征組合
            3. 更少的人工參與
          4. 絕點:過度泛化

        4. 子類API

        5. 功能API(函數式API)

        6. 多輸入與多輸出

          # coding:utf-8
# file: tf_keras_regression_wide_deep.py
# author: Dean
# contact: 1028968939@qq.com
# time: 2019/12/17 11:47
# desc: wide && deep

import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import sklearn
import pandas as pd
import os,sys,time
import tensorflow as tf
from tensorflow import keras
def showVersion():
    print((tf.__version__))
    print(sys.version_info)
    for module in mpl, np, pd, sklearn, tf, keras:
        print(module.__name__,module.__version__)
from sklearn.datasets import fetch_california_housing
housing = fetch_california_housing()

from sklearn.model_selection import train_test_split
(x_train_all,x_test,y_train_all,y_test) = train_test_split(
    housing.data,housing.target,random_state=7
)
x_train, x_valid,y_train, y_valid = train_test_split(
    x_train_all,y_train_all,random_state=11
)

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()

x_train_scaled = scaler.fit_transform(x_train)
x_valid_scaled = scaler.transform(x_valid)
x_test_scaled = scaler.transform(x_test)

def nn():
    pass
    """
    # 函數式API
    input = keras.layers.Input(shape=x_train.shape[1:])
    hidden1 = keras.layers.Dense(30,activation="relu")(input)
    hidden2 = keras.layers.Dense(30,activation="relu")(hidden1)

    concat = keras.layers.concatenate([input,hidden2])  # 拼接   input 直接到輸出,input經過hidden后輸出   此時input 是相同的
    output = keras.layers.Dense(1)(concat)
    model = keras.models.Model(inputs=[input],outputs=[output])
    """
    """
    # 子類API
    class WideDeepModel(keras.models.Model):
        def __init__(self):
            super(WideDeepModel,self).__init__()
            # 定義模型的層次
            self.hidden1_layer = keras.layers.Dense(30,activation="relu")
            self.hidden2_layer = keras.layers.Dense(30,activation="relu")
            self.output_layer = keras.layers.Dense(1)
        def call(self,input):
            # 完成模型的正向計算
            hidden1 = self.hidden1_layer(input)
            hidden2 = self.hidden2_layer(hidden1)
            concat = keras.layers.concatenate([input,hidden2])
            output = self.output_layer(concat)
            return output
    model = WideDeepModel()  # model = keras.models.Sequential([WideDeepModel(),])
    model.build(input_shape=(None,8))
    """
    """
    # 多輸入
    input_wide = keras.layers.Input(shape=[5])
    input_deep = keras.layers.Input(shape=[6])
    hidden1 = keras.layers.Dense(30,activation="relu")(input_deep)
    hidden2 = keras.layers.Dense(30,activation="relu")(hidden1)
    concat = keras.layers.concatenate([input_wide,hidden2])   # 拼接input是不同的
    output = keras.layers.Dense(1)(concat)
    model = keras.models.Model(inputs=[input_deep,input_wide],
                              outputs=[output])
    x_train_scaled_wide = x_train_scaled[:, :5]
    x_train_scaled_deep = x_train_scaled[:, 2:]
    x_valid_scaled_wide = x_valid_scaled[:, :5]
    x_valid_scaled_deep = x_valid_scaled[:, 2:]
    x_test_scaled_wide = x_test_scaled[:, :5]
    x_test_scaled_deep = x_test_scaled[:, 2:]
    # fit ,evaluate 中換對應數據即可 實現多輸入x_train_scaled --->[x_train_scaled_wide,x_train_scaled_deep] 還要驗證輸入,測試輸入
    """
    # 多輸出
    input_wide = keras.layers.Input(shape=[5])
    input_deep = keras.layers.Input(shape=[6])
    hidden1 = keras.layers.Dense(30, activation="relu")(input_deep)
    hidden2 = keras.layers.Dense(30, activation="relu")(hidden1)
    concat = keras.layers.concatenate([input_wide, hidden2])  # 拼接input是不同的
    output = keras.layers.Dense(1)(concat)   # 此時有兩個輸出
    output2 = keras.layers.Dense(1)(hidden2)
    model = keras.models.Model(inputs=[input_deep, input_wide],
                               outputs=[output,output2])
    # deep,wide 同上進行分割,注意此時fit時 y_train----->[y_train,y_train]  還有驗證y_valid y_test
    # ===================================================
    model.summary()
    model.compile(loss="mean_squared_error", optimizer="adam")  # 若出現梯度爆炸 可換用梯度下降算法
    callbacks = [keras.callbacks.EarlyStopping(patience=5, min_delta=1e-4)]

    history = model.fit(x_train_scaled,y_train,epochs=100,
                        validation_data=(x_valid_scaled,y_valid),
                        callbacks=callbacks)
    model.evaluate(x_test_scaled,y_test)  # 驗證集驗證
    return history

def plot_learning_curves(history):
    pd.DataFrame(history.history).plot(figsize=(8,5))
    plt.grid(True)
    plt.gca().set_ylim(0,1)
    plt.show()
if __name__ =="__main__":
    history = nn()
    plot_learning_curves(history)

        7. 超參數搜索

          1. 神經網絡訓練過程不變的參數
            1. 網絡結構參數:幾層,每層寬度,每層激活函數等
          2. 訓練參數:batch_size,學習率,學習衰減算法等
          3. 手工試耗費人力
          4. 搜索策略
            1. 網格搜索
              • 定義n維方格
              • 每個方格對應一組超參數
              • 一組一組嘗試
              • 例如,學習率定義m個,dropout n個,交叉m*n 次運算
            2. 隨機搜索
              • 在網格搜索的網格中,隨機搜索,次數變多,
            3. 遺傳算法搜索
              1. 對自然界模擬
              2. A,初始化參數集合->訓練->得到模型指標作為生存概率
              3. B,選擇->交叉->變異->產生下一代集合
              4. C,重新到A
            4. 啟發式搜索
              1. 研究熱點-AutoML
              2. 使用循環神經網絡生成參數
              3. 使用強化學習來進行反饋,使用模型來訓練生成參數
          # coding:utf-8
# file: tf_keras_regression_hp_search.py
# author: Dean
# contact: 1028968939@qq.com
# time: 2019/12/17 11:47
# desc: 超參數搜索

import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import sklearn
import pandas as pd
import os,sys,time
import tensorflow as tf
from tensorflow import keras
def showVersion():
    print((tf.__version__))
    print(sys.version_info)
    for module in mpl, np, pd, sklearn, tf, keras:
        print(module.__name__,module.__version__)
from sklearn.datasets import fetch_california_housing
housing = fetch_california_housing()

from sklearn.model_selection import train_test_split
(x_train_all,x_test,y_train_all,y_test) = train_test_split(
    housing.data,housing.target,random_state=7
)
x_train, x_valid,y_train, y_valid = train_test_split(
    x_train_all,y_train_all,random_state=11
)

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()

x_train_scaled = scaler.fit_transform(x_train)
x_valid_scaled = scaler.transform(x_valid)
x_test_scaled = scaler.transform(x_test)

def nn():
    # 超參數搜索
    """
    # 自實現超參數搜索,本例簡單,順序運行,參數單一
    # learning_rate:[1e-3,3e-4,1e-4,3e-3,1e-2,3e-2]
    learning_rate = [1e-3,3e-4,1e-4,3e-3,1e-2,3e-2]
    # w = w + learning_rate * grad
    historys =[]
    for lr in learning_rate:
        model = keras.models.Sequential([
            keras.layers.Dense(30,activation="relu",input_shape=x_train.shape[1:]),
            keras.layers.Dense(1)
        ])
        optimizer = keras.optimizers.Adam(lr) # lr應該是根據不同的策略會逐漸衰減的
        model.compile(loss="mean_squared_error", optimizer=optimizer)
        callbacks = [keras.callbacks.EarlyStopping(patience=5, min_delta=1e-2)]
        history = model.fit(x_train_scaled,y_train,epochs=100,
                            validation_data=(x_valid_scaled,y_valid),
                            callbacks=callbacks)
        historys.append(history)
    """
    # RandomizedSearchCV
    # 1,轉化為sklear的model
    #   tf.keras.wrappers.scikit_learn.KerasRegressor
    #   tf.keras.wrappers.scikit_learn.KerasClassifier
    # 2,定義參數集合
    # 3,搜索參數
    def build_model(hidden_layers = 1,layer_size = 30, learning_rate = 3e-3):
        model = keras.models.Sequential()
        model.add(keras.layers.Dense(layer_size,activation="relu",input_shape=x_train.shape[1:]))
        for _ in range(hidden_layers - 1):
            model.add(keras.layers.Dense(layer_size,activation="relu"))
        model.add(keras.layers.Dense(1))
        optimizer = keras.optimizers.Adam(learning_rate)
        model.compile(loss="mean_squared_error", optimizer=optimizer)
        return model
    sklearn_model = keras.wrappers.scikit_learn.KerasRegressor(build_model) # 傳入函數名

    from scipy.stats import reciprocal
    # f(x) = 1/(x*log(b/a))  a<=x<=b
    param_distribution = {
        "hidden_layers": [1, 2, 3, 4],
        "layer_size": np.arange(1, 100),  # [1,2,3....100]
        "learning_rate": reciprocal(1e-4, 1e-2)  # 按照某種分布生成
    }
    
    from sklearn.model_selection import RandomizedSearchCV
    random_search_cv = RandomizedSearchCV(sklearn_model,
                                          param_distribution,
                                          cv = 5,  # 交叉驗證 份數
                                          n_iter=10,  # 隨機尋找參數組合的數量，默認值為10。
                                          n_jobs=1)  # 並行計算時使用的計算機核心數量，默認值為1。當n_jobs的值設為-1時，則使用所有的處理器。
    callbacks = [keras.callbacks.EarlyStopping(patience=5, min_delta=1e-2)]
    history = random_search_cv.fit(x_train_scaled, y_train, epochs=100,   # 還使用fit
                            validation_data=(x_valid_scaled, y_valid),
                            callbacks=callbacks)
    # cross_validation:交叉驗證 ,訓練集分為n份 n-1份訓練,1份測試
    random_search_cv.best_params_  # 最好的參數
    random_search_cv.best_score_  # 最好的參數對應的分數
    model = random_search_cv.best_estimator_.model  # 最好的模型
    model.evaluate(x_test_scaled,y_test)  # 測試
    
    return historys

def plot_learning_curves(history):
    pd.DataFrame(history.history).plot(figsize=(8,5))
    plt.grid(True)
    plt.gca().set_ylim(0,1)
    plt.show()
if __name__ =="__main__":
    historys = nn()
    for history in historys:
        print()
        plot_learning_curves(history)

  3. Tensorflow基礎API使用

    1. 概要

      1. Tf框架:基礎數據類型,自定義模型與損失函數,自定義求導,tf.function,圖結構\
      2. 項目:圖像分類,房價預測

    2. 知識點

      1. 基礎API
      2. 基礎API與keras的集成
        1. 自定義損失函數
        2. 自定義層次
      3. @tf.function的使用(2.0專有:將python轉化為圖結構)
      4. 自定義求導

    3. @tf.function

      1. 將python函數編譯為圖
      2. 易於將模型導出為GraphDef+checkpoint 或者SavedModel
      3. 使得eager execution可以默認打開
      4. 1.0的代碼可以通過tf.function在2.0繼續使用
        1. 代替session

    4. API

      1. 基礎數據類型

        1. Tf.constant,tf.string

          1. # 0維 一個數 shape=()
# 1維 列表 shape=(n)
# 2維 二維數組 shape=(m,n)
t = tf.constant([
    [1,2,3],
    [4,5,6]
])  # 2.0中可以直接獲取值
print(t)
print(t[:,1:])
print(t[...,2]) # 一維向量 tf.Tensor([3 6], shape=(2,), dtype=int32)
print(t+10)  # 都加上10
print(tf.square(t))  # 每個數平方
print(t @ tf.transpose(t))  # 返回t與它轉置的乘積 2*2
print(t.numpy())  # 直接轉化為numpy對象 2*3矩陣

          2. t = tf.constant("tensorflow")
print(t)  # tf.Tensor(b'tensorflow', shape=(), dtype=string)
print(tf.strings.length(t))  # tf.Tensor(10, shape=(), dtype=int32)
print(tf.strings.length(t,unit="UTF8_CHAR"))  # tf.Tensor(10, shape=(), dtype=int32)
print(tf.strings.unicode_decode(t,"utf8"))  # tf.Tensor([116 101 110 115 111 114 102 108 111 119], shape=(10,), dtype=int32)
t = tf.constant(['cafe','coffee','咖啡'])  #
print(tf.strings.length(t,unit="UTF8_CHAR"))  # tf.Tensor([4 6 2], shape=(3,), dtype=int32)
print(tf.strings.unicode_decode(t,"utf8"))  # <tf.RaggedTensor [[99, 97, 102, 101], [99, 111, 102, 102, 101, 101], [21654, 21857]]>

        2. tf.ragged.constant,tfSpareTensor

          1. v = tf.Variable([[1,2,3,4],[5,6,7,8]])
print(v)  # Variable 對象
print(v.value())  # tensor
print(v.numpy())  # numpy矩陣
# 重新賦值,不能用等於號
v.assign(2*v)
v[0,1].assign(42)
v[1].assign([9,10,11,12])

          2. t = tf.ragged.constant([[11,12],[13,14,15]])
print(t)  # <tf.RaggedTensor [[11, 12], [13, 14, 15]]>
print(t[1])  # tf.Tensor([13 14 15], shape=(3,), dtype=int32)
print(t[1:2])  # <tf.RaggedTensor [[13, 14, 15]]>
print(tf.concat([t,t],axis=0))  # 按照行拼接變為4行
print(tf.concat([t,t],axis=1))  # 若要按照列拼接,首先行數要相同
print(t.to_tensor())  # 變為普通tensor 空的位置補為0,0都在正常值后邊

          3. t = tf.SparseTensor(indices=[[0,1],[1,0],[2,3]],  # 注意必須先[0,1] 再[0,2] 否則to_dense 會報錯,若必須不按順序 tf.sparse.reorder(t) 即可使用
                    values=[1,2,3],
                    dense_shape=[3,4])
print(t)  # 存儲稀疏矩陣,指定值位置,數值,shape即可
print(tf.sparse.to_dense(t))  # 轉換為普通tensor
# 不能加法
t2 = tf.constant("..")  # 普通的4*3 Tensor
tf.sparse.sparse_dense_matmul(t,t2)  # 得到的是3*3普通Tensor

      2. 自定義損失函數------Tf.reduce_mean

         def customized_mse(y_true, y_pred):
        return tf.reduce_mean(tf.square(y_pred - y_true))
 model.compile(loss=customized_mse, optimizer="adam",metrics=["mean_squared_error"])

      3. 自定義層次-----Keras.layers.Lambda和繼承法

        1. layer = tf.keras.layers.Dense(100,input_shape=[None,5])
layer(tf.zero([10,5]))   # 輸出[10,100] 二維
# [10,5] * w  +b = [10,100]
# w  [5:100]
layer.trainable_variables # 獲得kernel 與bias 可以查看
# ---------------------------------------------
 class CustomizedDenseLayer(keras.layers.Layer):
        def __init__(self,units,activation=None,**kwargs):
            self.units = units
            self.activation = keras.layers.Activation(activation)
            super(CustomizedDenseLayer, self).__init__(**kwargs)
        def build(self,input_shape):
            """構建需要的參數"""
            # x * w +b [None,a] w[a,b] [None,b]
            self.kernel = self.add_weight(name="kernel",
                                          shape=(input_shape[1],self.units),
                                          initilizer="uniform", # 定義隨機初始化kernel的方法:此處使用均勻分布
                                          trainable=True)
            self.bias = self.add_weight(name="bias",
                                        shape=(self.units,),
                                        initilizer="zeros",
                                        trainable=True)
            super(CustomizedDenseLayer,self).build(input_shape)
        def call(self,x):
            """完整的正向計算"""
            return self.activation(x @ self.kernel + self.bias)  # @ 表示矩陣乘法

    model = keras.models.Sequential([
        CustomizedDenseLayer(30, activation="relu", input_shape=x_train.shape[1:]),
        CustomizedDenseLayer(1)
    ])

        2.  # 定義簡單的層次,激活函數層,dropout層等
    #   eg:tf.nn.softplus: log(1+e^x)
    # customized_softplus = keras.layers.Dense(1,activation="softplus") = keras.layers.Dense(1),keras.layers.Activation("softplus")
    customized_softplus = keras.layers.Lambda(lambda x:tf.nn.softplus(x))

      4. Tf.function

        1. Tf.fucntion,tf.autograph.to_code,

          1. # python ----->圖
#   法一
def scaled_elu(z,scale=1.0,alpha=1.0):   # python函數
    # z>=0?scale *z :scale * alpha * tf.nn.elu(z)
    is_positive = tf.greater_equal(z,0.0)
    return scale * tf.where(is_positive,z,alpha * tf.nn.elu(z))
print(scaled_elu(tf.constant(-3)))   #   常量
print(scaled_elu(tf.constant([-3,-2.5])))  # 列表向量  都何以全部接受並處理
scaled_elu_tf = tf.function(scaled_elu)
scaled_elu_tf.python_function  # 返回原來的python函數
print(scaled_elu(tf.constant(-3)))
print(scaled_elu(tf.constant([-3,-2.5]))) # 與上邊結果相同,轉換的作用是速度加快
#   法二
#   1+1/2+....1/2^n
@tf.function
def converge_to(n_iters):
    total = tf.constant(0.)
    increment = tf.constant(1.)
    for _ in range(n_iters):
        total += increment
        increment /= 2.0
    return total
print(converge_to(20))


          2. def display_tf_code(func):
    # 中間代碼
    # python代碼轉為tf代碼, 圖就是通過此轉換的
    code = tf.autograph.to_code(func)   # 還有to_graph 是將代碼轉為圖的
    from IPython.display import display,Markdown
    display(Markdown('```python\n{}\n```'.format(code)))
display_tf_code(scaled_elu)

          3. var = tf.Variable(0.)
@tf.function
def add_21():
    return var.assign_add(21)
print(add_21())  # 結果返回21的tensor   ,若var在內部會報錯, 神經網絡中大多是變量,需要在外邊初始化

@tf.function(input_signature=[tf.TensorSpec([None],tf.int32,name='x')])
def cube(z):  # 可接收浮點數,整數  使用輸入簽名后會限制只能輸入int32
    return tf.pow(z,3)
print(cube(tf.constant([1.2,2.6])))
print(cube(tf.constant([1,2])))

          4. # 只有經過輸入簽名的才能保存為Saved_Model,在這個過程中使用get_concrete_function,把tf.function標注的轉換為有圖建議的函數
cube_func_int32 = cube.get_concrete_function(tf.TensorSpec([None],tf.int32))

print(cube_func_int32 is cube.get_concerte_function(tf.TensorSpec([2],tf.int32)))

      5. GraphDef

        1. get_operations,get_operation_by_name

        2. get_tensorf_by_name,as_graph_def

          cube_func_int32.graph  # 圖
cube_func_int32.graph.get_operations()  # 獲取操作
cube_func_int32.graph.get_operations()[2]  # 獲取某一個操作
cube_func_int32.graph.get_operations()[2].xxx  # 獲取某一個操作中的某個屬性

cube_func_int32.graph.get_operation_by_name("operationName")  # 通過operation名字獲取
cube_func_int32.graph.get_tensor_by_name("x:0")  # 通過tensor名字獲取

cube_func_int32.graph.as_graph_def()  # 顯示圖的結構信息

# 主要用來保存圖結構,模型,恢復圖結構,模型

      6. 自動求導

        1. 普通求導方法

          def f(x):
    return 3. * x ** 2 + 2. * x -1
def approximae_derivative(f,x,eps= 1e-3):  # 求函數 f 在 x 點的導數
    # x點向右eps, 向左eps 中間直線的斜率近似導數, eps足夠小,導數足夠接近
    return (f(x+eps)-f(x-eps))/(2*eps)
# print(approximae_derivative(f,1))


def g(x1,x2):
    return (x1 + 5)*(x2 ** 2)
def approximae_gredient(g,x1,x2,eps=1e-3):
    dg_x1 = approximae_derivative(lambda x:g(x,x2),x1,eps)
    dg_x2 = approximae_derivative(lambda x:g(x1,x),x2,eps)
    return dg_x1,dg_x2
# print(approximae_gredient(g,2,3))

        2. Tf.GrandientTape

          1. x1 = tf.Variable(2.0)
x2 = tf.Variable(3.0)
with tf.GradientTape(persistent = True) as tape:
    z = g(x1,x2)

dz_x1 = tape.gradient(z,x1)  # 9.0
dz_x2 = tape.gradient(z,x2)   # tape 只能使用一次,本次會報錯 GradientTape(persistent=Ture) 表示不釋放tape就可以多次使用,但使用完畢后需要手動釋放那個資源
print(dz_x1,dz_x2)
del tape

          2. with tf.GradientTape(persistent = True) as tape: # 不用persistent
    z = g(x1,x2)
dz_x1,dz_x2 = tape.gradient(z,[x1,x2])   # 也可以這樣一次求出兩個

          3. # 當x1,x2 為tf.constant()時,返回值為None
with tf.GradientTape(persistent = True) as tape:  # 這樣修改即可
    tape.watch(x1)
    tape.watch(x2)
    z = g(x1,x2)

          4. x = tf.Variable(5.0)
with tf.GradientTape() as tape:
    z1 = 3 * x
    z2 = x **2
tape.gradient([z1,z2],x)  # z1對於x的導數加上z2對於x的導數

          5. # 二階導數
x1 = tf.Variable(2.0)
x2 = tf.Variable(3.0)
with tf.GradientTape(persistent=True) as outer_tape:
    with tf.GradientTape(persistent=True) as inner_tape:
        z = g(x1,x2)
    inner_grads = inner_tape.gradient(z,[x1,x2])
outer_grads = [outer_tape.gradient(inner_grad,[x1,x2])
               for inner_grad in inner_grads]
print(outer_grads)
del inner_tape
del outer_tape

          6. # 簡單梯度下降
learning_rate = 0.1
x = tf.Variable(0.0)
for _ in range(100):
    with tf.GradientTape() as tape:
        z = 3. * x ** 2 + 2. * x -1
    dz_dx = tape.gradient(z,x)
    x.assign_sub(learning_rate*dz_dx)
print(x) # -0.333333

        3. Optimizer.apply_gradients

          1. # 優化梯度下降
learning_rate = 0.1
x = tf.Variable(0.0)
optimizer = keras.optimizers.SGD(lr = learning_rate)
for _ in range(100):
    with tf.GradientTape() as tape:
        z = 3. * x ** 2 + 2. * x -1
    dz_dx = tape.gradient(z,x)
    optimizer.apply_gradients([(dz_dx,x)])  # 與上邊不同
print(x) # -0.333333

          2. def test():
    metric = keras.metrics.MeanSquaredError()  # 第一個形參是真實值,第二個是預測值  返回的是平均 均方誤差
    print(metric([5],[2]))  # 9
    print(metric([0],[1]))  # 5
    print(metric.result())  # 5
    metric.reset_statues()  # 不再累加
    print(metric([1],[3]))  # 4

          3. def nn(): 
    # fit 的內容
    #   1,batch 遍歷訓練集 metric
    #       1.1,自動求導
    #   2,epoch結束 驗證集 metric
    epochs = 100
    batch_size = 32
    steps_per_epoch = len(x_train_scaled) // batch_size
    optimizer = keras.optimizers.SGD()
    metric = keras.metrics.MeanSquaredError()

    def random_batch(x,y,batch_size=32):
        idx = np.random.randint(0,len(x),size=batch_size)  # 在0到 len(x) 中隨機取batch_size個數
        return x[idx],y[idx]
    model = keras.models.Sequential([
        keras.layers.Dense(30,activation="relu",input_shape=x_train.shape[1:]),
        keras.layers.Dense(1),
    ])
    for epoch in range(epochs):
        metric.reset_states()
        for step in range(steps_per_epoch):
            x_batch,y_batch = random_batch(x_train_scaled,y_train,batch_size)  # 獲取數據
            with tf.GradientTape() as  tape:
                y_pred = model(x_batch)  # 計算預測
                loss = tf.reduce_mean(keras.losses.mean_squared_error(y_batch,y_pred))  # 定義損失函數
                metric(y_batch,y_pred)
            grads = tape.gradient(loss,model.variables)
            grads_and_vars = zip(grads,model.variables)   # 每個參數對應他的梯度
            optimizer.apply_gradients(grads_and_vars)  # 將梯度的變化應用到變量上
            print("\rEpoch",epoch,"train mse:",metric.result().numpy(),end="")
        y_valid_pred = model(x_valid_scaled)
        valid_loss = tf.reduce_mean(
            keras.losses.mean_squared_error(y_valid_pred,y_valid)
        )
        print("\t","valid mse",valid_loss.numpy())

  4. Tensorflow dataset使用

    1. 基礎知識

      1. Tf框架:

        1. 基礎api

          1. tf.data.Dateset.from_tensor_slics

            dataset = tf.data.Dataset.from_tensor_slices((np.arange(10)))
print(dataset)  # <TensorSliceDataset shapes: (), types: tf.int32>    # 里邊的元素每一個為一組

          2. repeat,batch,interleave,map,shuffle,list_files

            for item in dataset.batch(2):  # 還是10打個數,但此時每組2個,5組
     print(item)

            dataset = dataset.repeat(3)  # dataset 變為30個數,30組
dataset =dataset.batch(7)   # 每組7個數 5組
for item in dataset:
    print(item)

            # interleave: 將dataset中的每一個數據處理后,合並返回
# case:dataset 中是一系列文件的名字,通過他,遍歷名字讀取內容,使用它合並
dataset2 = dataset.interleave(
    lambda v:tf.data.Dataset.from_tensor_slices(v),  # map_fn:處理函數
    cycle_length=5, # cycle_length:並行處理個數
    block_length=5 # block_length: 從處理的元素中每次取多少個出來 [0,1,2,3,4,5,6]  只取[0,1,2,3,4]  最后不夠的時候 從第一個中未取到的取
)
for item in dataset2:
    print(item)

            x = np.array([[1,2],[3,4],[5,6]])
y = np.array(['cat','dog','fox'])

dataset3 = tf.data.Dataset.from_tensor_slices((x,y))
for item_x,item_y in dataset3:
    print(item_x.numpy(),item_y.numpy())
"""
[1 2] b'cat'
[3 4] b'dog'
[5 6] b'fox'
"""
dataset4 = tf.data.Dataset.from_tensor_slices({'feature':x,"label":y})
for item in dataset4:
    print(item['feature'].numpy(),item['label'].numpy())  # 同上

        2. csv文件,

          1. 生成csv

            import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import sklearn
import pandas as pd
import os,sys,time
import tensorflow as tf
from tensorflow import keras
def showVersion():
    print((tf.__version__))
    print(sys.version_info)
    for module in mpl, np, pd, sklearn, tf, keras:
        print(module.__name__,module.__version__)
from sklearn.datasets import fetch_california_housing
housing = fetch_california_housing()

from sklearn.model_selection import train_test_split
(x_train_all,x_test,y_train_all,y_test) = train_test_split(
    housing.data,housing.target,random_state=7
)
x_train, x_valid,y_train, y_valid = train_test_split(
    x_train_all,y_train_all,random_state=11
)

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()

x_train_scaled = scaler.fit_transform(x_train)
x_valid_scaled = scaler.transform(x_valid)
x_test_scaled = scaler.transform(x_test)
output_dir = r"generate_csv"
if not os.path.exists(output_dir):
    os.mkdir(output_dir)
def save_to_csv(output_dir,data,name_prefix,header=None,n_parts=10):
    path_format = os.path.join(output_dir,"{}_{:02d}.csv")  # 文件名格式:第一個表示是train還是test 第二個表示為2位的整數
    filenames= []
    for file_idx,row_indics in enumerate(  # 將數據分為索引加數據 通過idx indics獲取
            np.array_split( # 將索引分為n_parts部分  [array(1,2,3,4,), array(5,6,7,8..)]
                np.arange(len(data)),  # 生成和data一樣長的數組,當索引
                         n_parts)   # 將索引分為n_patrs
            ):
        """
        file_idx row_indics
        0       [0,1,2,3,4..]
        1       [5,6,7,8,9..]
        2       ...
        """
        print(name_prefix,file_idx)
        part_csv = path_format.format(name_prefix,file_idx)
        filenames.append(part_csv)
        with open(part_csv,"wt",encoding="utf-8") as f:
            if header is not None:
                f.write(header+"\n")  # 寫入header
            for row_index in row_indics:
                f.write(",".join([repr(col) for col in data[row_index]]))
                f.write('\n')
    return filenames
train_data = np.c_[x_train_scaled,y_train]  # 將數據和標簽合並
valid_data = np.c_[x_valid_scaled,y_valid]
test_data = np.c_[x_test_scaled,y_test]
header_cols = housing.feature_names + ['MidianHouseValue']
header_str = ','.join(header_cols)
train_filenames = save_to_csv(output_dir,train_data,"train",header_str,n_parts=10)

          2. tf.data.TextLineDataset

          3. tf.io.decode_csv

            # 1. filename ---> dataset
# 2. read file --> dataset -> datasets ->merge
# 3, parse csv
filename_dataset = tf.data.Dataset.list_files(train_filenames)
n_readers =5
dataset = filename_dataset.interleave(
    lambda filename:tf.data.TextLineDataset(filename).skip(1),  # 根據文件名按照行讀取文件內容, 並跳過header
    cycle_length=n_readers
)
for line in dataset.take(15):  # 讀取數據的前15行
    print(line.numpy())  #  是一個整字符串
# tf.io.decode_csv(str,record_defaults)
sample_str = '1,2,3,4,5'
records_defaults = [tf.constant(0,dtype=tf.int32)]*5 # 要解析的類型
records_defaults = [tf.constant(0,dtype=tf.int32),0,np.nan,"hello",tf.constant([])] # 要解析的類型
parsed_fields = tf.io_csv(sample_str,records_defaults)
print(parsed_fields)

def parse_csv_line(line,n_fileds=9):
    defs = [tf.constant(np.nan)] * n_fileds
    parsed_fields = tf.io.decode_csv(line,record_defaults=defs)
    x = tf.stack(parsed_fields[0:-1])
    y = tf.stack(parsed_fields[-1:])
    return x,y

            def csv_reader_dataset(filenames,n_readers=5,
                       batch_size=32,n_parse_threads=5,  # 解析時並行數
                       shuffle_buffer_size=10000):  # buffer大小
    dataset = tf.data.Dataset.list_files(filenames)
    dataset = dataset.repeat()  # 重復無限次
    dataset = dataset.interleave(
        lambda filename: tf.data.TextLineDataset(filename).skip(1),
        cycle_length= n_readers,
    )
    dataset.shuffle(shuffle_buffer_size)  # 混排
    dataset = dataset.map(parse_csv_line,num_parallel_calls=n_parse_threads)  # 將數據經過處理后返回 與interleave類似
    dataset = dataset.batch(batch_size)
    return dataset
train_set = csv_reader_dataset(train_filenames,batch_size=32)
valid_set = csv_reader_dataset(train_filenames,batch_size=32)  # 換位valid_filenames
model = keras.models.Sequential([
        keras.layers.Dense(30, activation="relu", input_shape=x_train.shape[8]),
        keras.layers.Dense(1)
    ])
model.compile(loss='mse', optimizer="adam")
callbacks = [keras.callbacks.EarlyStopping(patience=5, min_delta=1e-4)]
history = model.fit(train_set, epochs=100,
                    steps_per_epoch = 1160//32,  # 指定每個epoch 的次數
                    validation_steps =3870//32,
                    validation_data=valid_set,
                    callbacks=callbacks)
# model.evaluate(test_set,steps = 5160//32)

        3. tfrecord文件

          1. tf.train.FloatList, tf.train.Int64List, tf.train.bytesList

            # tfrecord 文件格式
# -> tf.train.Example
#   -> tf.train.Features ->{'key':tf.train.Feature}

          ->tf.train.Feature ->{tf.train.ByteList/FloatList/Int64List}

          favorite_books = [name.encode("utf-8") for name in ["machine learning","cc150"]]
          favorite_books_bytelist = tf.train.BytesList(value = favorite_books)
          """
          value: "machine learning"
          value: "cc150"
          """
          hours_floatlist = tf.train.FloatList(value=[15.5,9.0,7.0,8.0])
          age = tf.train.Int64List(value=[64])

          


2. tf.train.Feature,    tf.train.Features,    tf.train.Example

```python
features = tf.train.Features(
    feature = {
        "favorite_books":tf.train.Feature(bytes_list = favorite_books_bytelist),
        "hours":tf.train.Feature(float_list = hours_floatlist),
        "age":tf.train.Feature(int64_list = age)
    }
)
print(features)  # json格式,顯示每個feature
"""
feature {
  key: "age"
  value {
    int64_list {
      value: 64
    }
  }
}
faeture{....}
"""
example = tf.train.Example(features=features)  # 與features 類似   featrues{feature{}...}

          1. exapmle.SerializeToString

            serizlized_example = example.SerializeToString()  # 壓縮,

          2. tf.io.ParseSingleExample

            output_dir = "tf_tfrecord_basic"
if not os.path.exists(output_dir):
    os.mkdir(output_dir)
filename = "test.tfrecords"
filename_fullpath = os.path.join(output_dir,filename)
# 寫入文件
with tf.io.TFRecordWriter(filename_fullpath) as writer:  # 打開文件
    for i in range(3):  # 寫進去3次
        writer.write(serizlized_example)
# 讀取
dataset = tf.data.TFRecordDataset([filename_fullpath])
for seralized_example in dataset:
    print(serizlized_example)   # 與上邊的serialized_example 類似  壓縮過的

          3. tf.io.VarLenFeature,tf.io.FixedLenFeature

            expected_features = {
    "favorite_books": tf.io.VarLenFeature(dtype=tf.string),   # 變長  一會解析后是sparsetensor
    "hours":tf.io.VarLenFeature(dtype=tf.float32),  # 變長
    "age":tf.io.FixedLenFeature([],dtype=tf.int64),  # 定長,普通的tensor,[] 表示是0維的數,[8] 就表示8特特征
}

          4. tf.data.TFRecordDataset,tf.io.TFRecordOptions

            dataset = tf.data.TFRecordDataset([filename_fullpath])
for serialized_example_tensor in dataset:
    example = tf.io.parse_single_example(
        serialized_example_tensor,
        expected_features
    )
    books = tf.sparse.to_dense(example["favorite_books"],default_value=b"")  # 解析sparsetensor 為tensor  sparsetensor為0的地方 不能轉為字符串 要制定default_value

    print(example)

#  存為壓縮格式
filename_fullpath_zip = filename_fullpath+ ".zip"
options = tf.io.TFRecordOptions(compress_type = "GZIP")
with tf.io.TFRecordWriter(filename_fullpath_zip,options) as writer:
    pass

# 讀取壓縮格式
dataset = tf.data.TFRecordDataset([filename_fullpath_zip],compression_type="GZIP")  # 其余不變即可

      2. 房價csv轉record

        # coding:utf-8
# file: tf_data_generate_tfrecord.py
# author: Dean
# contact: 1028968939@qq.com
# time: 2019/12/22 11:29
# desc:
# coding:utf-8
# file: tf_keras_classification_model_dnn.py
# author: Dean
# contact: 1028968939@qq.com
# time: 2019/12/17 11:47
# desc: csv 文件轉換為record

import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import sklearn
import pandas as pd
import os,sys,time
import tensorflow as tf
from tensorflow import keras
source_dir = "generate_csv"
def get_filenames_by_prefix(source_dir,prefix_name):  # 按照文件名分類
    all_files = os.listdir(source_dir)
    results=[]
    for filename in all_files:
        if filename.startswith(prefix_name):
            results.append(os.path.join(source_dir,filename))
    return results
train_filenames = get_filenames_by_prefix(source_dir,"train")
import pprint
# pprint.pprint(train_filenames)  # 換行打印

def parse_csv_line(line,n_fileds=9):
    defs = [tf.constant(np.nan)] * n_fileds
    parsed_fields = tf.io.decode_csv(line,record_defaults=defs)
    x = tf.stack(parsed_fields[0:-1])
    y = tf.stack(parsed_fields[-1:])
    return x,y
def csv_reader_dataset(filenames,n_readers=5,
                       batch_size=32,n_parse_threads=5,  # 解析時並行數
                       shuffle_buffer_size=10000):  # buffer大小
    dataset = tf.data.Dataset.list_files(filenames)
    dataset = dataset.repeat()  # 重復無限次
    dataset = dataset.interleave(
        lambda filename: tf.data.TextLineDataset(filename).skip(1),
        cycle_length= n_readers,
    )
    dataset.shuffle(shuffle_buffer_size)  # 混排
    dataset = dataset.map(parse_csv_line,num_parallel_calls=n_parse_threads)  # 將數據經過處理后返回 與interleave類似
    dataset = dataset.batch(batch_size)
    return dataset
train_set = csv_reader_dataset(train_filenames,batch_size=32)

# 遍歷數據寫入文件
def serialize_example(x,y):
    """Converts x ,y to tf.train.FloatList and serialize"""
    input_features = tf.train.FloatList(value=x)
    label = tf.train.FloatList(value=y)
    features = tf.train.Features(
        feature={
            "input_features":tf.train.Feature(float_list = input_features),
            "label":tf.train.Feature(float_list = label)
        }
    )
    example = tf.train.Example(features = features)
    return example.SerializeToString()
def csv_dataset_to_tfrecords(base_filename,dataset,n_shards,steps_per_shard,compression_type = None):
    """
    n_shard:文件個數
    steps_per_shard:每個小文件,走多少步
    """
    options = tf.io.TFRecordOptions(compression_type=compression_type)
    all_filenames=[]
    for shard_id in range(n_shards):
        filename_fullpath = '{}_{:05d}-of-{:05d}'.format(
            base_filename,shard_id,n_shards
        )
        with tf.io.TFRecordWriter(filename_fullpath,options) as writer:
            for x_batch,y_batch in dataset.take(steps_per_shard):  #  每個batch都是32個的
                for x_example,y_example in zip(x_batch,y_batch):
                    writer.write(serialize_example(x_example,y_example))
        all_filenames.append(filename_fullpath)
    return all_filenames
n_shards = 20
train_step_per_shard = 11610//32//n_shards
output_dir = "generate_tfreocrds"  # 先創建
train_basename = os.path.join(output_dir,"train")

train_tfrecord_filename = csv_dataset_to_tfrecords(
    train_basename,train_set,n_shards,train_step_per_shard,None
)


# 讀取文件
expected_feature = {
    "input_ferature":tf.io.FixedLenFeature([8],dtype=tf.float32),
    "label":tf.io.FixedLenFeature([1],dtype=tf.float32)
}
def parse_example(serialized_example):
    example = tf.io.parse_single_example(serialize_example,expected_feature)
    return expected_feature["input_ferature"],expected_feature['label']
def tfrecord_reader_dataset(filenames,n_readers=5,
                       batch_size=32,n_parse_threads=5,  # 解析時並行數
                       shuffle_buffer_size=10000):  # buffer大小
    dataset = tf.data.Dataset.list_files(filenames)
    dataset = dataset.repeat()  # 重復無限次
    dataset = dataset.interleave(
        lambda filename: tf.data.TFRecordDataset(filename,compression_type=nONE),
        cycle_length= n_readers,
    )
    dataset.shuffle(shuffle_buffer_size)  # 混排
    dataset = dataset.map(parse_example,num_parallel_calls=n_parse_threads)  # 將數據經過處理后返回 與interleave類似
    dataset = dataset.batch(batch_size)
    return dataset
tfrecords_train = tfrecord_reader_dataset(train_tfrecord_filename,
                                          batch_size=3)
for  x_batch,y_batch in tfrecords_train:
    print(x_batch,y_batch)

  5. Tensorflow Estimator使用與tf1.0

    1. 知識點

      Tf框架:estimator使用,特征列使用,tf1.0基本使用

      項目:泰坦尼克號生存

    2. API

      1. Tf.keras.estimator.to_estimator

        1. Train,evaluate

          # dataset 泰坦尼克號
# https://storage.googleapis.com/tf-datasets/titanic/train.csv
# https://storage.googleapis.com/tf-datasets/titanic/eval.csv

train_file = "csv/train.csv"
eval_file = "csv/eval.csv"

train_df = pd.read_csv(train_file)  # (627,9)
eval_df = pd.read_csv(eval_file)  # (264.9)

y_train = train_df.pop("survived")
y_eval = eval_df.pop("survived")
# print(train_df)
# print(y_train.head())
# print(train_df.describe()) 非離散值的信息
# train_df.age.hist(bins = 20)   畫出age的分布直方圖 bin表示分為多少份
# train_df.sex.value_counts().plot(kind="barh")  # 橫向柱狀圖 "barv"縱向: 統計男性個數,女性個數
# train_df['class'].value_counts().plot(kind="barh") 不同艙位人數
# 男性,女性中獲救比例,
# pd.concat([train_df,y_train],axis=1).groupby('sex').survived.mean()

# 離散值
categorical_columns = ["sex",'n_siblings_spouses','parch','class','deck','embark_town','alone']
# 連續紙
numeric_columns = ['age','fare']
feature_columns =[]
for categorical_column in categorical_columns:
    vocab = train_df[categorical_column].unique()
    feature_columns.append(
        tf.feature_column.indicator_column(
            tf.feature_column.categorical_column_with_vocabulary_list(
                categorical_column,vocab)))
for categorical_column in numeric_columns:
    feature_columns.append(
        tf.feature_column.numeric_column(
            categorical_column,dtype=tf.float32
        )
    )
# 構件dataset的函數
def make_dataset(data_df,label_df,epochs=10,shuffle=True,batch_size=32):
    dataset = tf.data.Dataset.from_tensor_slices(
        (dict(data_df),label_df)
    )
    if shuffle:
        dataset =dataset.shuffle(10000)
    dataset = dataset.repeat(epochs).batch(batch_size)
    return dataset
train_dataset = make_dataset(train_df,y_train,batch_size=5)

# keras.layers.DenseFeature
tf.keras.backend.set_floatx('float64')
# for x,y in train_dataset.take(1):
#     age_column = feature_columns[7]
#     gender_column = feature_columns[0]
#     print(keras.layers.DenseFeatures(age_column)(x).numpy())  # 連續值不變
#     print(keras.layers.DenseFeatures(gender_column)(x).numpy())  # 離散值變為one-hot
# for x,y in train_dataset.take(1):
#     print(keras.layers.DenseFeatures(feature_columns)(x).numpy())

model = keras.models.Sequential([
    keras.layers.DenseFeatures(feature_columns),
    keras.layers.Dense(100,activation="relu"),
    keras.layers.Dense(100,activation="relu"),
    keras.layers.Dense(2,activation="softmax")
])
model.compile(loss="sparse_categorical_crossentropy",
              optimizer=keras.optimizers.SGD(lr=0.01),
              metrics = ['accuracy'])
# 1,model.fit
# 2.model-->estimator ->train
# 第一種
# train_dataset = make_dataset(train_df,y_train,epochs=100)
# eval_dataset = make_dataset(eval_df,y_eval,epochs=1,shuffle=False)
# model.fit(train_dataset,
#           validation_data = eval_dataset,
#           steps_per_epoch = 20,
#           validation_steps=8,
#           epochs=100)
# 第二種
estimator = keras.estimator.model_to_estimator(model)
# input_fn:函數或lambda  return  (feature,label)    或dataset--->(feature,label)
estimator.train(input_fn = lambda :make_dataset(train_df,y_train,epochs=100))    # 會出現bug

      2. Tf.estimator.BaselineClassifier

      3. Tf.estimator.LinerClassifier

      4. Tf.estimator.DNNClassifier

      5. Tf.feature_column

        1. categorical_column_with_vocabulary_list
        2. numeric_column
        3. indicator_column
        4. cross_column

      6. keras.layers.DenseFeatures

        train_file = "csv/train.csv"
eval_file = "csv/eval.csv"
train_df = pd.read_csv(train_file)  # (627,9)
eval_df = pd.read_csv(eval_file)  # (264.9)
y_train = train_df.pop("survived")
y_eval = eval_df.pop("survived")


# 離散值
categorical_columns = ["sex",'n_siblings_spouses','parch','class','deck','embark_town','alone']
# 連續紙
numeric_columns = ['age','fare']
feature_columns =[]
for categorical_column in categorical_columns:
    vocab = train_df[categorical_column].unique()
    feature_columns.append(
        tf.feature_column.indicator_column(
            tf.feature_column.categorical_column_with_vocabulary_list(
                categorical_column,vocab)))
for categorical_column in numeric_columns:
    feature_columns.append(
        tf.feature_column.numeric_column(
            categorical_column,dtype=tf.float32
        )
    )
# 構件dataset的函數
def make_dataset(data_df,label_df,epochs=10,shuffle=True,batch_size=32):
    dataset = tf.data.Dataset.from_tensor_slices(
        (dict(data_df),label_df)
    )
    if shuffle:
        dataset =dataset.shuffle(10000)
    dataset = dataset.repeat(epochs).batch(batch_size)
    return dataset
# =================
# output_dir = "baseline_model"
# if not os.path.exists(output_dir):
#     os.mkdir(output_dir)
# baseline_estimator = tf.estimator.BaselineClassifier(model_dir=output_dir,
#                                                      n_classes=2)
# baseline_estimator.train(input_fn=lambda :make_dataset(train_df,y_train,epochs=100))
# baseline_estimator.evaluate(input_fn=lambda :make_dataset(eval_df,y_eval,epochs=1,shuffle=False,batch_size=20))
# =================
# linear_output_dir = "liner_model"
# if not os.path.exists(linear_output_dir):
#     os.mkdir(linear_output_dir)
# linear_estimator = tf.estimator.LinearClassifier(
#     model_dir=linear_output_dir,
#     n_classes=2,
#     feature_columns=feature_columns
# )
# linear_estimator.train(input_fn=lambda :make_dataset(
#     train_df,y_train,epochs=100))
# =====================================
dnn_output_dir = "dnn_model"
if not os.path.exists(dnn_output_dir):
    os.mkdir(dnn_output_dir)
dnn_estimator = tf.estimator.DNNClass ifier(
    model_dir=dnn_output_dir,
    n_classes=2,
    feature_columns=feature_columns,
    hidden_units=[128,128], # 兩層,都是128
    activation_fn=tf.nn.relu,
    optimizer="Adam"
)
dnn_estimator.train(input_fn= lambda :make_dataset(train_df,y_train,epochs=100))
dnn_estimator.evaluate(input_fn= lambda :make_dataset(
    eval_df,y_eval,epochs=1,shuffle=False))

  6. 卷積神經網絡

    1. 知識點

      1. Tf框架:卷積實現

        卷積網絡:卷積+池化---全連接 ------>分類任務

        全卷積網絡:去掉了最后的全連接層,變為反卷積層(是尺寸變大) ----->物體分割

      2. 項目:圖像分類,kaggle 10monkeys,kaggle cifar10

      3. 理論:卷積,數據增強,遷移學習

      4. 卷積網問題:

        1. 參數太多
          1. 局部連接, 圖像區域性
          2. 參數共享 圖像特征與位置無關
        2. 池化時,剩余的數據會丟棄

    2. keras實現卷積神經網絡

      1. # coding:utf-8
# file: tf_keras_classification_model_cnn.py
# author: Dean
# contact: 1028968939@qq.com
# time: 2019/12/17 11:47
# desc: 深度神經網絡

import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import sklearn
import pandas as pd
import os,sys,time
import tensorflow as tf
from tensorflow import keras
def showVersion():
    print((tf.__version__))
    print(sys.version_info)
    for module in mpl, np, pd, sklearn, tf, keras:
        print(module.__name__,module.__version__)

# class_names = ['T-shirt','Trouser','Pullover','Dress',
#                'Coat', 'Sandal', 'Shirt', 'Sneaker',
#                'Bag', 'Ankle boot'
#                ]

fashion_mnist = keras.datasets.fashion_mnist
(x_train_all,y_train_all),(x_test,y_test) = fashion_mnist.load_data()
x_valid, x_train = x_train_all[:5000], x_train_all[5000:]
y_valid, y_train = y_train_all[:5000], y_train_all[5000:]

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()

x_train_scaled = scaler.fit_transform(x_train.astype(np.float32).reshape(-1,1)).reshape(-1,28,28,1)
x_valid_scaled = scaler.transform(x_valid.astype(np.float32).reshape(-1,1)).reshape(-1,28,28,1)  # 使用訓練集的均值,方差
x_test_scaled = scaler.transform(x_test.astype(np.float32).reshape(-1,1)).reshape(-1,28,28,1)

def nn():
    model = keras.models.Sequential()
    model.add(keras.layers.Conv2D(filters=32,  # 卷積核數量,
                                  kernel_size=3,  # 大小
                                  padding="same",  # 是否填充是的輸入輸出大小一樣
                                  activation="relu",  # 使用selu效果會更好
                                  input_shape=(28,28,1)
                                ))
    model.add(keras.layers.Conv2D(filters=32,kernel_size=3,
                                  padding="same",
                                  activation="relu"))
    model.add(keras.layers.MaxPool2D(pool_size=2))   # 一般步長與大小相同  pool 后的卷積層filter一般會翻倍
    model.add(keras.layers.Conv2D(filters=64, kernel_size=3,
                                  padding="same",
                                  activation="relu"))
    model.add(keras.layers.Conv2D(filters=64, kernel_size=3,
                                  padding="same",
                                  activation="relu"))
    model.add(keras.layers.MaxPool2D(pool_size=2))
    model.add(keras.layers.Conv2D(filters=128,kernel_size=3,
                                  padding="same",
                                  activation="relu"))
    model.add(keras.layers.Conv2D(filters=128, kernel_size=3,
                                  padding="same",
                                  activation="relu"))
    model.add(keras.layers.MaxPool2D(pool_size=2))
    model.add(keras.layers.Flatten())  # 將輸入一維化
    model.add(keras.layers.Dense(128,activation="relu"))
    model.add(keras.layers.Dense(10,activation="softmax"))
    model.summary()
    model.compile(loss="sparse_categorical_crossentropy",
                  optimizer = "adam",
                  metrics = ['accuracy'])
    logdir = "cnn-callbacks"
    if not os.path.exists(logdir):
        os.mkdir(logdir)
    output_model_file = os.path.join(logdir,"fashion_mnist_model.h5")
    callbacks = [
        keras.callbacks.TensorBoard(logdir),
        keras.callbacks.ModelCheckpoint(output_model_file,save_best_only = True),
        keras.callbacks.EarlyStopping(patience = 5, min_delta = 1e-3)
    ]
    history = model.fit(x_train_scaled,y_train,epochs=10,
                        validation_data=(x_valid_scaled,y_valid),
                        callbacks = callbacks)
    model.evaluate(x_test_scaled,y_test)  # 驗證集驗證
    return history

def plot_learning_curves(history):
    pd.DataFrame(history.history).plot(figsize=(8,5))
    plt.grid(True)
    plt.gca().set_ylim(0,3)
    plt.show()
if __name__ =="__main__":
    history = nn()
    plot_learning_curves(history)

    3. keras實現深度可分離卷積

      1. 標准卷積

      2. 卷積核大小(\(D_k\))

      3. 卷積核輸入通道數M

      4. 卷積核個數N,輸出通道N

      5. 圖像大小\((D_F)\)

        \(D_k*D_k*M*N*D_F*D_F\)

      6. 深度可分離卷積

        (此時的卷積核輸入時原圖n*n*3,卷積核為k*k*1,為了計算原圖所有通道,該卷積核個數就是輸入通道數M) \(D_k*D_k*1*M*D_F*D_F+1*1*M*N*D_F*D_F\)

        參考:

        https://blog.csdn.net/xiewenbo/article/details/84315560

        https://blog.csdn.net/qq_21997625/article/details/87106152

      7.  # 代碼同上
 # 第一個Conv2D不變
 # 其余的Conv2D修改為SeparableConv2D即可
 #  計算量少
   
# 兩層 3*3 視野域就是 一個5*5

# 計算量幾乎縮小n倍

# 深度可分離卷積

    4. kera實戰kaggle

      1. 10 monkey,cifar10

        # coding:utf-8
# file: 1_10_monkey_model.py
# author: Dean
# contact: 1028968939@qq.com
# time: 2019/12/23 11:30
# desc: keras_generator 搭建模型
#    model.fit  將數據放入內存訓練
#    model.fit_generator   處理大數據,  使用生成器的數據集

import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import sklearn
import pandas as pd
import os,sys,time
import tensorflow as tf
from tensorflow import keras


train_dir=r"I:\人工智能數據\10_monkey\training\training"
valid_dir=r"I:\人工智能數據\10_monkey\validation\validation"
label_txt=r"I:\人工智能數據\10_monkey\monkey_labels.txt"

labels = pd.read_csv(label_txt,header=0)

height = 128  # 縮放大小
width = 128
channels = 3
batch_size = 64
num_classes = 10

# keras中的數據集:讀取數據,數據分享

train_datagen = keras.preprocessing.image.ImageDataGenerator(
    rescale = 1./255,  # 像素點縮放到0-1
    rotation_range = 40,  # 圖片增強方式,隨機旋轉40
    width_shift_range = 0.2,  # 偏移
    height_shift_range = 0.2,  #
    shear_range = 0.2,  # 剪切強度
    zoom_range = 0.2,  # 縮放程度
    horizontal_flip = True,  # 水平翻轉
    fill_mode = "nearest",  # 放大時填充像素,選擇最近的填充
)

# 讀取圖片,按照上邊方法進行處理
train_generator = train_datagen.flow_from_directory(train_dir,
                                                    target_size=(height,width),
                                                    batch_size=batch_size,
                                                    seed=7,  # 隨機數
                                                    shuffle=True,
                                                    class_mode="categorical")  # label格式 one-hot
valid_datagen = keras.preprocessing.image.ImageDataGenerator(rescale=1./255)
valid_generator = valid_datagen.flow_from_directory(valid_dir,
                                                    target_size=(height,width),
                                                    batch_size=batch_size,
                                                    seed=7,
                                                    shuffle=False,  # 不用訓練不用打亂
                                                    class_mode="categorical")
train_num = train_generator.samples
valid_num = valid_generator.samples
# print(train_num,valid_num) #  1098 272
# x,y = train_generator.next()  # (64,128,128,3) (64,10)

model = keras.models.Sequential([
    keras.layers.Conv2D(filters=32,kernel_size=3,padding="same",
                        activation="relu",input_shape=(width,height,channels)),
    keras.layers.Conv2D(filters=32,kernel_size=3,padding="same",
                        activation="relu"),
    keras.layers.MaxPool2D(pool_size=2),
    keras.layers.Conv2D(filters=64,kernel_size=3,padding="same",
                        activation="relu"),
    keras.layers.Conv2D(filters=64,kernel_size=3,padding="same",
                        activation="relu"),
    keras.layers.MaxPool2D(pool_size=2),
    keras.layers.Conv2D(filters=128,kernel_size=3,padding="same",
                        activation="relu"),
    keras.layers.Conv2D(filters=128,kernel_size=3,padding="same",
                        activation="relu"),
    keras.layers.MaxPool2D(pool_size=2),
    keras.layers.Flatten(),
    keras.layers.Dense(128,activation="relu"),
    keras.layers.Dense(num_classes,activation="softmax")
])
model.compile(loss="categorical_crossentropy",
              optimizer="adam",metrics=['accuracy'])
model.summary()

epochs = 300
# 最后acc 97% 98%
history = model.fit_generator(train_generator,
                              steps_per_epoch= train_num//batch_size,
                              epochs = epochs,
                              validation_data=valid_generator,
                              validation_steps= valid_num//batch_size)
# history.history.keys()  # loss,acc, val_loss,val_acc

def plot_learning_curves(history,label,epochs,min_value,max_value):
    data={}
    data[label] = history.history[label]
    data["val_"+label]=history.history["val_"+label]
    pd.DataFrame(data).plot(figsize=(8.5))
    plt.grid(True)
    plt.axis([0,epochs,min_value,max_value])
    plt.show()
plot_learning_curves(history,"acc",epochs,0,1)
plot_learning_curves(history,"loss",epochs,1.5,2.5)

      2. 數據增強(調整數據 提高准確率)與遷移學習

        1. # model 部分修改即可
resnet50_fine_tune = keras.models.Sequential();
resnet50_fine_tune.add(keras.applications.ResNet50(include_top=False,   # 不包含最后一層
                                                   pooling = 'avg',
                                                   weights="imagenet"))  # None 從頭開始,imagenet接着
resnet50_fine_tune.add(keras.layers.Dense(num_classes,activation="softmax"))
resnet50_fine_tune.layers[0].trainable= False  # 設置第一層的參數不調整
"""
resnet50 = keras.applications.ResNet50(include_top=False,
                                       pooling="avg",
                                       weights="imagenet")
resnet50.summary()
for layer in resnet50.layers[0:-5]:  # 設置resnet50中后0--5層也不可訓練,其余可訓練
    layer.trainable=False
resnet50_new = keras.models.Sequential([
    resnet50,
    keras.layers.Dense(num_classes,activation="softmax"),
])
"""

          # coding:utf-8
# file: 1_cifar10_model.py
# author: Dean
# contact: 1028968939@qq.com
# time: 2019/12/23 11:30
# desc:
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import sklearn
import pandas as pd
import os,sys,time
import tensorflow as tf
from tensorflow import keras
# data https://www.kaggle.com/c/cifar-10/data
class_names = [
    'airplane',
    'automobile',
    'bird',
    'cat',
    'deer',
    'dog',
    'frog',
    'horse',
    'ship',
    'truck'
]
train_lables_file = r'I:\人工智能數據\cifar10\trainLabels.csv'
test_csv_file = r'I:\人工智能數據\cifar10\sampleSubmission.csv'
train_folder = r'I:\人工智能數據\cifar10\train'
test_floder = r'I:\人工智能數據\cifar10\test'
def parse_csv_file(filepath,folder):
    """Paeses csv files into (filename(path),label) format  (xxx.png,'cat')"""
    results =[]
    with open(filepath,"r") as f:
        lines = f.readlines()[1:]  # 去掉header
        for line in lines:
            image_id,label_str = line.strip("\n").split(',')
            image_full_path = os.path.join(folder,image_id+".png")
            results.append((image_full_path,label_str))
    return results
train_lables_info = parse_csv_file(train_lables_file,train_folder)  # 50000
test_csv_info = parse_csv_file(test_csv_file,test_floder)  # 300000

# 制作dataframe
# train_df = pd.DataFrame(train_lables_info)
train_df = pd.DataFrame(train_lables_info[0:45000])
valid_df = pd.DataFrame(train_lables_info[45000:])
test_df = pd.DataFrame(test_csv_info)

# 設置dataframe的列名
train_df.columns = ["filepath",'class']
valid_df.columns = ["filepath",'class']
test_df.columns = ["filepath",'class']


height = 32  # 縮放大小
width = 32
channels = 3
batch_size = 32
num_classes = 10

# keras中的數據集:讀取數據,數據分享

train_datagen = keras.preprocessing.image.ImageDataGenerator(
    rescale = 1./255,  # 像素點縮放到0-1
    rotation_range = 40,  # 圖片增強方式,隨機旋轉40
    width_shift_range = 0.2,  # 偏移
    height_shift_range = 0.2,  #
    shear_range = 0.2,  # 剪切強度
    zoom_range = 0.2,  # 縮放程度
    horizontal_flip = True,  # 水平翻轉
    fill_mode = "nearest",  # 放大時填充像素,選擇最近的填充
)

# 讀取圖片,按照上邊方法進行處理
train_generator = train_datagen.flow_from_dataframe(
    train_df,
    directory="./",
    x_col="filepath",
    y_col="class",
    classes=class_names,
    target_size=(height, width),
    batch_size=batch_size,
    seed=7,
    shuffle=True,
    class_mode="sparse"
)
valid_datagen = keras.preprocessing.image.ImageDataGenerator(rescale=1./255)
valid_generator = valid_datagen.flow_from_dataframe(valid_df,
                                                    directory="./",
                                                    x_col="filepath",
                                                    y_col="class",
                                                    classes=class_names,
                                                    target_size=(height,width),
                                                    batch_size=batch_size,
                                                    seed=7,
                                                    shuffle=False,
                                                    class_mode="sparse")
train_num = train_generator.samples
valid_num = valid_generator.samples
# print(train_num,valid_num) #  1098 272
# x,y = train_generator.next()  # (64,128,128,3) (64,10)

model = keras.models.Sequential([
    keras.layers.Conv2D(filters=128,kernel_size=3,padding="same",
                        activation="relu",input_shape=(width,height,channels)),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2D(filters=128,kernel_size=3,padding="same",
                        activation="relu"),
    keras.layers.BatchNormalization(),
    keras.layers.MaxPool2D(pool_size=2),
    keras.layers.Conv2D(filters=256,kernel_size=3,padding="same",
                        activation="relu"),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2D(filters=256,kernel_size=3,padding="same",
                        activation="relu"),
    keras.layers.BatchNormalization(),
    keras.layers.MaxPool2D(pool_size=2),
    keras.layers.Conv2D(filters=512,kernel_size=3,padding="same",
                        activation="relu"),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2D(filters=512,kernel_size=3,padding="same",
                        activation="relu"),
    keras.layers.BatchNormalization(),
    keras.layers.MaxPool2D(pool_size=2),
    keras.layers.Flatten(),
    keras.layers.Dense(512,activation="relu"),
    keras.layers.Dense(num_classes,activation="softmax")
])
model.compile(loss="sparse_categorical_crossentropy",
              optimizer="adam",metrics=['accuracy'])
model.summary()

epochs = 1
history = model.fit_generator(train_generator,
                              steps_per_epoch= train_num//batch_size,
                              epochs = epochs,
                              validation_data=valid_generator,
                              validation_steps= valid_num//batch_size)

def plot_learning_curves(history,label,epochs,min_value,max_value):
    data={}
    data[label] = history.history[label]
    data["val_"+label]=history.history["val_"+label]
    pd.DataFrame(data).plot(figsize=(8.5))
    plt.grid(True)
    plt.axis([0,epochs,min_value,max_value])
    plt.show()
plot_learning_curves(history,"acc",epochs,0,1)
plot_learning_curves(history,"loss",epochs,0,2)

  7. 循環神經網絡

    1. Tf框架:LSTM實現
    2. 項目:文本分類,文本生成,Kaggle文本分類
    3. 理論:序列式問題,循環網絡,LSTM,雙向LSTM

  8. Tensorflow分布式

    1. 理論部分

      1. GPU設置
        1. 默認用全部GPU並占滿內存
        2. 如何不浪費內存和計算資源
          1. 內存增長
          2. 虛擬設備機制
        3. 多GPU
          1. 虛擬GPU&實際GPU
          2. 手工設置&分布式機制
        4. API
          1. tfdebugging.set_log_device_placement
          2. tf.config.experimental.set_visible_devices 設置可見設備
          3. tf.config.experimental.list_logical_devices 獲取邏輯設備
          4. tf.config.experimental.list_physical_devices 獲取物理設備
          5. tf.config.experimental.set_memory——growth 設置內存自增
          6. tf.config.experimental.VirtualDeviceConfiguration 建立邏輯設備
          7. tf.config.set_soft_device_placement 自動分配任務到設備
      2. 分布式策略

    2. 實戰部分

      1. GPU設置部分
      2. 分布式訓練

  9. Tensorflow模型保存於部署

    1. Tf框架:模型保存,導出tflite,部署

    2. 項目:圖像分類

    3. 模型保存

      1. 文件格式

        1. Ckeckpoint與graphdef (tf1.0)
        2. keras(hdf5),SavedModel(tf2.0)(參數+網絡結構)

      2. 保存的是什么

        1. 參數
        2. 參數+網絡結構

      3. TFLite

        1. TFLite Converter
          1. 模型轉化 將上邊的三種模型轉化為tflite
        2. TFLite Interpreter
          1. 模型加載
          2. 支持android與ios
          3. 支持多種語言
        3. TFLite-FlatBuffer
          1. google開源的跨平台數據序列化庫
          2. 優點
            1. 直接讀取序列化數據
            2. 高效內存使用和速度
            3. 靈活,數據前后向兼容,靈活控制數據結構
            4. 代碼少
            5. 強類型數據
        4. TFlite-量化
          1. 參數從float變為8bit整數
            1. 准確率損失
            2. 模型大小變為1/4
          2. 量化方法
            1. float=(int-歸零點)*因子

      4. 案例

        1. keras-保存參數與保存模型+參數

          keras.callbacks.ModelCheckpoint(out_put_dir,
                               save_best_only=True,
                               save_weights_only=False) # 為false表示只保存參數,True表示都保存
keras.models.load_model(out_put_dir) # 若都保存了,從新加載后和可直接使用

          # 定義模型之后,導入weights 可使用

          # 也可保存權重
model.save_weights("xxxx/xxx.h5") 

          tf.saved_model.save(model,".../xxx_garph")
# assets
# variables checkpoint 信息
# saved_model.pb 

          # 命令行工具查看model信息
!saved_model_cli show --dir "path" --all
# 載入model
model = tf.saved_model.load("path")
inference = model.signatures["serving_default"]
results = inference(tf.constant(x_test_scaled[0:1]))

        2. keras簽名函數保存到SavedModel

          tf_export =tf.Model()
tf_export.cube = 簽名函數
tf.saved_model.save(tf._export,dir)

        3. keras,SavedModel,簽名函數到具體函數

          load_saved_model=keras.models.load_model(".../...h5")
load_saved_model(np.zeros([1,28,28]))

run_model = tf.function(lambda x:load_saved_model(x))
keras_concrete_function = run_model.get_concrete_function(
    tf.TensorSpec(
    	load_keras_model.inputs[0].shape,
        load_keras_model.inputs[0].dtype
    ))
# 測試
keras_concrete_function(tf.constant(np.zeros(1,28,28,dtype=np.float32)))

        4. keras,SavedModel,具體函數到tflite

        5. tflite量化

        6. tensorflow js.android部署

  10. 機器翻譯於tensorflow2tensor使用

    1. Tf框架:transformer實現,tensor2tensor使用
    2. 項目:機器翻譯
    3. 理論:序列道序列模型.注意力機制,可縮放點積注意力,多頭注意力


免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。



猜您在找 tensorflow2.0——LSTM tensorflow2.0編程規范 tensorflow2.0學習筆記（一） tensorflow2.0——可訓練變量 tensorflow2.0第2章 Tensorflow keras實戰 tensorflow2.0——模型保存與加載 （一）tensorflow2.0 - 自定義layer TensorFlow2.0教程1：keras 函數api 【tensorflow2.0】AutoGraph和tf.Module 【tensorflow2.0】優化器optimizers
 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM