Image for post
Image for post
Photo by Tim Mossholder

Welcome to my articles on Deep Learning, Reinforcement Learning, Meta-Learning, and Machine Learning. The purpose of this article is to index the series and articles I wrote so far. I also include a F.A.Q. for the most common questions and contact methods.

Series

Not part of a series yet

NLP

GPU Computing


Image for post
Image for post

Early versions of TensorFlow have major identity crises. So many APIs are proposed with split personalities. Thankfully, the adoption of the Keras API in TensorFlow 2 (TF 2) has changed the picture. More significantly, the API becomes more programmer-friendly, easier to trace, and less tedious. In this article, we show how to use this API to build deep learning models.

Sequential Model with CIFAR10

Keras API provides a high-level built-in class “models.Sequential” to create common deep learning (DL) models. In this section, we will train a CIFAR10 classifier and make predictions from it. The first part of the code below prepares the CIFAR10 dataset for training. Then, we build a CNN classifier and configure a training that uses an Adam optimizer with the cross-entropy loss function (model.compile). We also configure the training to collect data on accuracy. Finally, we call “model.fit” to train the model which returns history that contains the performance metric. …


Image for post
Image for post

While Keras provides deep learning layers to create models, it also provides APIs to preprocessing data. For example, preprocessing.Normalization() normalizes features with the feature means and the variances of the training dataset. These parameters are not part of the trainable parameters. Instead, in line 18, we “adapt” the normalization layer to the training samples (“data”). This method will calculate the means and the variances automatically.


Image for post
Image for post

In this article, we discuss how to use TensorFlow (TF) Dataset to build efficient data pipelines for training and evaluation. But, if the training data is small, we can fit the data into memory and preprocess them as Numpy ndarry. However, many real-life datasets are too large. To scale the solution, we create data pipelines (tf.data.Dataset) using features like preprocessing, prefetching, and caching in scaling up the solution.

MINST Data Preprocessing with NumPy

Nevertheless, toy datasets remain important in developing novel models and algorithms. So, let’s start our discussion with these small datasets, like MNIST.

Keras provides APIs in loading popular datasets into memory as NumPy ndarray. When a ndarray is passed into a TF operation, it will be converted to a TF Tensor automatically (or vice versa). If possible, TF maintains the same underlying memory representation for Numpy ndarray and Tensors. So the conversion is cheap. Nevertheless, if the Tensor is hosted in GPU, the conversion requires a data copy to the CPU host since a NumPy ndarray always has a copy in the host memory. …


Image for post
Image for post

Keras has 3 built-in RNN layers: SimpleRNN, LSTM ad GRU.

LSTM

Starting with a vocabulary size of 1000, a word can be represented by a word index between 0 and 999. For example, the word “side” can be encoded as integer 3.


Image for post
Image for post

Eager execution is highly promoted in TF 2. It makes coding and debugging easier. But that is not necessarily suggested for real training or production. In this article, we will talk about the two options: the eager execution mode and the graph mode, as well as its pro and con. In particular, running code in graph mode is not as seamless as it was projected. It can have significant performance impacts if some issues are ignored.

By default, TF operations in 2.x are run in eager execution mode. For example, tf.matmul (matrix multiplication) below executes immediately and returns a tf.Tensor object containing the result [[4.]]. This is what we usually expect in running Python codes. …


Image for post
Image for post

Keras API provides built-in classes to save models regularly during model fitting. To save a model and restore it later, we can create a callback ModelCheckPoint passed to model.fit, and the model will be saved regularly.


Image for post
Image for post

Keras API can perform backpropagation easily using built-in optimizers and loss functions. However, there are cases where we want to manipulate or apply gradient specifically. For example, to avoid explosive gradient, we may want to clip the gradient.


Image for post
Image for post

In this article, we will overview some of the key extensions and libraries in TensorFlow 2.x (TF 2.x). This will include TF Datasets, TF Hub, XLA, model optimization, TensorBoard, TF Probability, Neural Structured Learning, TF Serving, TF Federated, TF Graphics, and MLIR.

TensorFlow Datasets

It supports the loading of many popular datasets. For a complete list, checkout out the TF dataset category. Here is the code sample for loading MNINST data.

import tensorflow as tf
import tensorflow_datasets as tfds
import tensorflow.data as tfd

# Construct a tf.data.Dataset
# Data loaded into ~/tensorflow_datasets/mnist
ds = tfds.load('mnist', split='train', shuffle_files=True)

# Build your input pipeline
ds = ds.shuffle(1024).batch(32).prefetch(tfd.experimental.AUTOTUNE)
for example in ds.take(1):
image, label = example["image"], example["label"]

assert image.shape == (32, 28, 28, 1)
assert label.shape …


Image for post
Image for post

CPU provides a generic set of instructions for general computing. To modify or optimize applications, we change the code but the hardware is fixed. Nevertheless, this generalization comes at the cost of complexity in hardware. Without complex hardware optimizations, like speculative execution, it hurts performance. However, these optimizations increase die size, and power consumption.

About

Jonathan Hui

Deep Learning

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store