TensorFlow Serving

TensorFlow Serving handles the deployment half of the machine learning lifecycle: taking trained models and exposing them to clients as versioned, production-grade inference services. Developed by Google, the system serves multiple models, or multiple versions of the same model, simultaneously, so new versions roll out without client code changes and canary or A/B experiments run side by side. Requests arrive over gRPC or REST, and an intelligent batch scheduler groups them for efficient GPU execution while keeping latency overhead minimal. A reference-counted lookup table gives clients consistent access to specific model versions during transitions. Although it integrates with TensorFlow SavedModels out of the box, its modular servable abstraction extends to embeddings, vocabularies, feature transformations, and even non-TensorFlow models. Distributed as source and Docker images under the Apache 2.0 license, it is standard infrastructure for teams operating TensorFlow models in production at scale.