Discover what GOT Character will die next with TensorFlow

TensorFlow is a an open source library for numerical computation, specializing in machine learning applications. In this codelab, you will learn how to install and run TensorFlow on a single machine, and will predict deaths on Game of Thrones data using the tutorial Wide+Deep Learning Network.

What are we going to be building?

In this lab, we combine the strengths of Wide and Deep Neural Network modelling to produce a prediction model on whether a Game of Thrones character may die.

What You'll Learn

Familiarize yourself with Tensorflow
How to select features for the wide part: choose the sparse base columns and crossed columns you want to use.
Select features for the deep part: choose the continuous columns, the embedding dimension for each categorical column, and the hidden layer sizes.
How to design the Wide+Deep Learning TensorFlow model
Put them all together in a Wide & Deep model (DNNLinearCombinedClassifier).

What You'll Need

A computer connected to the internet with Python2.7 installed
The libraries sklearn, pandas , numpy installed
A basic understanding of python and neural nets
A fast computer running OS X or Linux
A fair amount of time

The Goal

To made a classification model
Predict the next character death probability in Game of Thrones

On a Docker Image

if you were going to perform image processing, GPU enabled installation...?. For more information, visit the Docker installation guideliness in the TensorFlow documentation.

Using virtualenv

As we are not going to process images, we recommend for this codelab virtualenv installation, because it will take you less time – if you were going to perform image processing, GPU enabled installation would be worth it.

With virtualenv the installation process is the following:

Install pip and virtualenv.
Create a virtualenv environment.
Activate the virtualenv environment and install TensorFlow in it.
After the install you will activate the virtualenv environment each time you want to use TensorFlow.

Install pip and virtualenv


# For Ubuntu/Linux 64-bit
sudo apt-get install python-pip python-dev python-virtualenv

# For Mac OS X
sudo easy_install pip
sudo pip install --upgrade virtualenv

Create a Virtualenv environment

Create a virtualenv environment in the directory ~/tensorflow (for Linux & MacOS):

virtualenv --system-site-packages ~/tensorflow

Activate the environment


# if you are using bash
source ~/tensorflow/bin/activate

# if you are using csh
source ~/tensorflow/bin/activate.csh

Install TensorFlow in the environment

Now that you have created the environment, install TensorFlow just as you would for a regular pip installation.

Copy the version that fit's your architecture and python's version:


# Ubuntu/Linux 64-bit, CPU only, Python 2.7
export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.12.0rc0-cp27-none-linux_x86_64.whl

# Ubuntu/Linux 64-bit, GPU enabled, Python 2.7
# Requires CUDA toolkit 8.0 and CuDNN v5. For other versions, see "Installing from sources" below.
export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-0.12.0rc0-cp27-none-linux_x86_64.whl

# Mac OS X, CPU only, Python 2.7:
export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-0.12.0rc0-py2-none-any.whl

# Mac OS X, GPU enabled, Python 2.7:
export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow_gpu-0.12.0rc0-py2-none-any.whl

# Ubuntu/Linux 64-bit, CPU only, Python 3.4
export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.12.0rc0-cp34-cp34m-linux_x86_64.whl

# Ubuntu/Linux 64-bit, GPU enabled, Python 3.4
# Requires CUDA toolkit 8.0 and CuDNN v5. For other versions, see "Installing from sources" below.
export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-0.12.0rc0-cp34-cp34m-linux_x86_64.whl

# Ubuntu/Linux 64-bit, CPU only, Python 3.5
export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.12.0rc0-cp35-cp35m-linux_x86_64.whl

# Ubuntu/Linux 64-bit, GPU enabled, Python 3.5
# Requires CUDA toolkit 8.0 and CuDNN v5. For other versions, see "Installing from sources" below.
export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-0.12.0rc0-cp35-cp35m-linux_x86_64.whl

# Mac OS X, CPU only, Python 3.4 or 3.5:
export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-0.12.0rc0-py3-none-any.whl

# Mac OS X, GPU enabled, Python 3.4 or 3.5:
export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow_gpu-0.12.0rc0-py3-none-any.whl

If your requirement's are listed above, you can also check out the installing from sources section in TensorFlow's installing guideliness.

Finally, install TensorFlow

Once you have setted the $TF_BINARY_URL env variable with the url of the binary that fits your system, simply install it with pip


# Python 2
pip install --upgrade $TF_BINARY_URL

# Python 3
pip3 install --upgrade $TF_BINARY_URL

Wide+Deep Neural Model

This approach of two neuralnets merged as one combines the strengths of memorization and generalization. It's useful for generic large-scale regression and classification problems with sparse input features (e.g., categorical features with a large number of possible feature values). If you're interested in learning more about how Wide & Deep Learning works, please check out google´s research paper.

The wide model responds to logistic regression with sparse features and transformations operations and the deep model stands for feed-forward neural network with an embedding layer and several hidden layers

Categorical and Continuous Variables

Categorical variables are also known as discrete or qualitative variables. Categorical variables can be further categorized as either nominal, ordinal or dichotomous. Nominal variables are variables that have two or more categories, but which do not have an intrinsic order.

The CSV takes an expanded view on character deaths from GOT, including the listed columns that give an overall basic information such as name, title, age and date of Birth. But also dives into the the alive relationships that the character holds, the book in which the character appears

Note that not all columns of the dataset have been selected for this tutorial. There have been some excluded data due to the relevance or not for this exercise.


CATEGORICAL_COLUMNS = ["alive", "title", "male", "culture",
                       "house", "spouse", "isAliveMother", "isAliveFather", "isAliveHeir",
                       "isAliveSpouse", "isMarried", "isNoble", "numDeadRelations",
                       "boolDeadRelations", "isPopular" , "popularity"]

CONTINUOUS_COLUMNS = ["name", "dateOfBirth",  "mother", "father",
                      "heir", "book1", "book2", "book3", "book4", "book5", "age",
                      "isAlive", "house", "title", "numDeadRelations"]

Define Base Feature Columns

First, let's define the base categorical and continuous feature columns that we'll use. These base columns will be the building blocks used by both the wide part and the deep part of the model.


# Generalizing  Transformations 
...
column[i] = tf.contrib.layers.real_valued_column(column_name)
...
age_buckets = tf.contrib.layers.bucketized_column(age,
                                                  boundaries=[
                                                      18, 25, 30, 35, 40, 45,
                                                      50, 55, 60, 65
                                                  ])

The Wide Model: Linear Model with Crossed Feature Columns

The wide model is a linear model with a wide set of sparse and crossed feature columns:


# Wide columns and deep columns.
wide_columns = [name, dateOfBirth, DateoFdeath, mother, father, heir, book1, book2,
                book3, book4, book5, age, isAlive
                tf.contrib.layers.crossed_column([house, title],
                                                 hash_bucket_size=int(1e4)),
                tf.contrib.layers.crossed_column(
                    [age_buckets, house, title],
                    hash_bucket_size=int(1e6)),
                tf.contrib.layers.crossed_column([numDeadRelations, title],
                                                 hash_bucket_size=int(1e4))]

The Deep Model: Neural Network with Embeddings

The deep model is a feed-forward neural network, as shown in the previous figure. Each of the sparse, high-dimensional categorical features are first converted into a low-dimensional and dense real-valued vector, often referred to as an embedding vector. These low-dimensional dense embedding vectors are concatenated with the continuous features, and then fed into the hidden layers of a neural network in the forward pass:


  deep_columns = [
      tf.contrib.layers.embedding_column(title, dimension=8),
      tf.contrib.layers.embedding_column(house, dimension=8),
      tf.contrib.layers.embedding_column(culture, dimension=8),
      tf.contrib.layers.embedding_column(isAliveNoble, dimension=8),
      tf.contrib.layers.embedding_column(numberDeadRelations,
                                         dimension=8),
      tf.contrib.layers.embedding_column(popularity, dimension=8),
      male,
      spouse,
      isPopular,
      spouse,
      isMarried,
  ]

The wide models and deep models are combined by summing up their final output log odds as the prediction, then feeding the prediction to a logistic loss function. All the graph definition and variable allocations have already been handled for you under the hood, so you simply need to create a DNNLinearCombinedClassifier:


import tempfile
model_dir = tempfile.mkdtemp()
m = tf.contrib.learn.DNNLinearCombinedClassifier(
    model_dir=model_dir,
    linear_feature_columns=wide_columns,
    dnn_feature_columns=deep_columns,
    dnn_hidden_units=[100, 50])

This classifier takes two neurons with 100 and 50 hidden units each. We can use several other features for customizing this model that go from the hidden units, activaction function and optimizer. The accuracy of this model is 80 % . Below you can find another customized classifier


import tempfile
model_dir = tempfile.mkdtemp()
m = tf.contrib.learn.DNNLinearCombinedClassifier(
    model_dir=model_dir,
    linear_feature_columns=wide_columns,
    dnn_feature_columns=deep_columns,
    dnn_hidden_units=[100, 50],
    dnn_activation_fn= tf.nn.relu ,
    enable_centered_bias=True))

The accuracy of the model is 80.5 %
How changing the parameters influences the accuracy of the model, training_steps or...
Testing character
Opening the model in Tensorboard