Early versions of TensorFlow have major identity crises. So many APIs are proposed with split personalities. Thankfully, the adoption of the Keras API in TensorFlow 2 (TF 2) has changed the picture. More significantly, the API becomes more programmer-friendly, easier to trace, and less tedious. In this article, we show how to use this API to build deep learning models.

Sequential Model with CIFAR10

Keras API provides a high-level built-in class “models.Sequential” to create common deep learning (DL) models. In this section, we will train a CIFAR10 classifier and make predictions from it. The first part of the code below prepares the CIFAR10 dataset for training. Then, we build a CNN classifier and configure a training that uses an Adam optimizer with the cross-entropy loss function (model.compile). We also configure the training to collect data on accuracy. Finally, we call “model.fit” to train the model which returns history that contains the performance metric. …

While Keras provides deep learning layers to create models, it also provides APIs to preprocessing data. For example, preprocessing.Normalization() normalizes features with the feature means and the variances of the training dataset. These parameters are not part of the trainable parameters. Instead, in line 18, we “adapt” the normalization layer to the training samples (“data”). This method will calculate the means and the variances automatically.

In this article, we discuss how to use TensorFlow (TF) Dataset to build efficient data pipelines for training and evaluation. But, if the training data is small, we can fit the data into memory and preprocess them as Numpy ndarry. However, many real-life datasets are too large. To scale the solution, we create data pipelines (tf.data.Dataset) using features like preprocessing, prefetching, and caching in scaling up the solution.

MINST Data Preprocessing with NumPy

Nevertheless, toy datasets remain important in developing novel models and algorithms. So, let’s start our discussion with these small datasets, like MNIST.

Keras provides APIs in loading popular datasets into memory as NumPy ndarray. When a ndarray is passed into a TF operation, it will be converted to a TF Tensor automatically (or vice versa). If possible, TF maintains the same underlying memory representation for Numpy ndarray and Tensors. So the conversion is cheap. Nevertheless, if the Tensor is hosted in GPU, the conversion requires a data copy to the CPU host since a NumPy ndarray always has a copy in the host memory. …

Keras has 3 built-in RNN layers: SimpleRNN, LSTM ad GRU.

LSTM

Starting with a vocabulary size of 1000, a word can be represented by a word index between 0 and 999. For example, the word “side” can be encoded as integer 3.

Eager execution is highly promoted in TF 2. It makes coding and debugging easier. But that is not necessarily suggested for real training or production. In this article, we will talk about the two options: the eager execution mode and the graph mode, as well as its pro and con. In particular, running code in graph mode is not as seamless as it was projected. It can have significant performance impacts if some issues are ignored.

By default, TF operations in 2.x are run in eager execution mode. For example, tf.matmul (matrix multiplication) below executes immediately and returns a tf.Tensor object containing the result [[4.]]. This is what we usually expect in running Python codes. …

Keras API provides built-in classes to save models regularly during model fitting. To save a model and restore it later, we can create a callback ModelCheckPoint passed to model.fit, and the model will be saved regularly.

Keras API can perform backpropagation easily using built-in optimizers and loss functions. However, there are cases where we want to manipulate or apply gradient specifically. For example, to avoid explosive gradient, we may want to clip the gradient.

In this article, we will overview some of the key extensions and libraries in TensorFlow 2.x (TF 2.x). This will include TF Datasets, TF Hub, XLA, model optimization, TensorBoard, TF Probability, Neural Structured Learning, TF Serving, TF Federated, TF Graphics, and MLIR.

TensorFlow Datasets

It supports the loading of many popular datasets. For a complete list, checkout out the TF dataset category. Here is the code sample for loading MNINST data.

import tensorflow as tf
import tensorflow_datasets as tfds
import tensorflow.data as tfd

# Construct a tf.data.Dataset
# Data loaded into ~/tensorflow_datasets/mnist
ds = tfds.load('mnist', split='train', shuffle_files=True)

# Build your input pipeline
ds = ds.shuffle(1024).batch(32).prefetch(tfd.experimental.AUTOTUNE)
for example in ds.take(1):
    image, label = example["image"], example["label"]

assert image.shape == (32, 28, 28, 1)
assert label.shape …