TensorFlow服務是一個用於服務機器學習模型的開源軟件庫。它處理機器學習的推斷方面，在培訓和管理他們的生命周期后采取模型，通過高性能，引用計數的查找表為客戶端提供版本化訪問。
可以同時提供多個模型，或者實際上是同一模型的多個版本。這種靈活性有助於新版本，非原子性遷移客戶端到新模型或版本，以及A / B測試實驗模型。
主要用途是高性能生產服務，但是同樣的服務基礎設施也可以用於批量處理（例如地圖縮減）作業以預先計算推理結果或分析模型性能。在這兩種情況下，GPU可以顯着增加推理吞吐量。 TensorFlow服務提供了一個調度程序，可以將單個推理請求分組批量，以便在GPU上進行聯合執行，並配置延遲控制。
TensorFlow服務對TensorFlow模型（自然）具有開箱即用的支持，但它的核心是管理任意版本的項目（servables），並將其傳遞給其本機API。除了經過訓練的TensorFlow模型之外，服務器還可以包括推理所需的其他資產，如嵌入，詞匯和特征轉換配置，甚至非基於TensorFlow的機器學習模型。
TensorFlow Serving is an open-source software library for serving machine learning models. It deals with the inference aspect of machine learning, taking models after training and managing their lifetimes, providing clients with versioned access via a high-performance, reference-counted lookup table.
Multiple models, or indeed multiple versions of the same model, can be served simultaneously. This flexibility facilitates canarying new versions, non-atomically migrating clients to new models or versions, and A/B testing experimental models.
The primary use-case is high-performance production serving, but the same serving infrastructure can also be used in bulk-processing (e.g. map-reduce) jobs to pre-compute inference results or analyze model performance. In both scenarios, GPUs can substantially increase inference throughput. TensorFlow Serving comes with a scheduler that groups individual inference requests into batches for joint execution on a GPU, with configurable latency controls.
TensorFlow Serving has out-of-the-box support for TensorFlow models (naturally), but at its core it manages arbitrary versioned items (servables) with pass-through to their native APIs. In addition to trained TensorFlow models, servables can include other assets needed for inference such as embeddings, vocabularies and feature transformation configs, or even non-TensorFlow-based machine learning models.
http://www.tensorflownews.com/2017/08/09/google-tensorflow-serving-library/