Custom NLP: Chapter 5 - Text Classification - Tensorflow

NLP Oct 11, 2021

In our previous posts we explored What is Custom NLP and a sample implementation of Custom NLP for Entity Recognition using AutoML.

But sometimes Cloud Native may not be the preferred solution or there might be some of us aspiring to be a Deep Learning Engineer and would prefer building their own Custom Neural Networks.

This article is all about helping you understand those building blocks for creating Custom Neural Models for Text Classification using one of the leading Open Source framework - Tensorflow.

Before we dive in deeper lets do a quick recap on what Custom Neural Architecture implies. Kindly find the link below to explain a brief about What Neural Networks and their Architectures are -

Before we start scripting, it is very essential to understand what is the exact problem statement.

What is Text Classification?

Well in simple terms it is broadly categorising a text corpus into pre-defined categories / Labels.

In many instances there are needs to identify user sentiments or categorise Customer reviews and as all of this is based on Text Analysis we would need an AI Solution that can accept Text and classify the corpus in its respective bucket.

So now without wasting much of our precious time, let's get right to it.

Step 0: Install all the necessary Packages

pip3 install pandas
pip3 install numpy
pip3 install tensorflow, tensorflow_hub

Step 1: Import all the necessary Packages

import pandas as pd
import random
import numpy as np
import os
import json

import tensorflow as tf
import tensorflow_hub as hub

As you can see, we leverage a variety of libraries for accomplishing our task. Varying from packages meant for Tabular data processing to Neural Model creation.

Step 2: Download the Dataset and Relax

For today we would be using a dataset curated by Open Source Community.

Download the Dataset and store it on your local system. Be sure to remember the path where you store the Dataset as we will be using it for our training job. Sit back and Relax as this might take some time depending on your internet speed.

Step 3: Read and Store the Data needed

As the data is sitting within a CSV, it is needed that the system can read that data and get it ready for processing.

def data_reader(dataset_path: str):
    data = pd.read_csv(dataset_path)
    all_labels = {}
    all_text = []

    for index, row in data.iterrows():
        text = row[0]
        label = row[1]
        all_text.append((text, label))
        if label not in all_labels:
            all_labels[label] = len(all_labels)

    rev_labels = {}
    for key, labels in all_labels.items():
        rev_labels[labels] = key

    return all_text, all_labels, rev_labels

The above function return the following -

all_text - A list of all the text corpus
all_labels - A Dictionary of all the labels with their corresponding token value
rev_labels - A Dictionary where the Key and Values are reversed of all_labels. This is used for inferencing, if needed in the future.

Step 4: Create a Data Generator

In general if you want to speed up your development process, Keras provides its own Data Generator. But as we are firm that we want to build everything Custom, here is an example of how you can create your own Data Generators.

def data_loader(bs, data, y_lab):
    while True:
        texts = []
        labels = []
        while len(texts) < bs:
            indice = random.randint(0, len(data) - 1)
            target = data[indice][1]
            labels.append(y_lab[target])

            test_text = data[indice][0]
            texts.append(test_text)

        yield np.asarray(texts), np.asarray(labels)

Data Generators are very crucial when you are dealing with massive volumes of Data.

If you fail to use Generator and instead choose to load the complete Dataset in your system memory, it may lead to an overflow and your Training would fail.

The highlight of Data Generators are, they use the YIELD function of Python to only store those data which are a part of the Batch which is being processed, thereby freeing up a lot of memory.

A Data Generator returns the following -

Input Data Array for the Model
Ground Truth Values or Labels in this case for the Model

Step 5: Create the Model

Now that we have processed the Data, it's time for the Neural Expert to rise and shine.

def model_arc(y_labels):
    embedding = "https://tfhub.dev/google/nnlm-en-dim50/2"
    hub_layer = hub.KerasLayer(
        embedding, input_shape=[], dtype=tf.string, trainable=True
    )
    model = tf.keras.Sequential()
    model.add(hub_layer)
    model.add(tf.keras.layers.Dense(16, activation="relu"))
    model.add(tf.keras.layers.Dense(len(y_labels), activation="softmax"))

    model.summary()
    model.compile(
        optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"]
    )
    return model

As you can see, we are not just declaring the model in the previous function but also compiling it with the necessary Loss Functions and Optimisers.

After running the above function, you would notice the Model Summary being printed on your terminal.

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
keras_layer (KerasLayer)     (None, 50)                48190600  
_________________________________________________________________
dense (Dense)                (None, 16)                816       
_________________________________________________________________
dense_1 (Dense)              (None, 7)                 119       
=================================================================
Total params: 48,191,535
Trainable params: 48,191,535
Non-trainable params: 0
_________________________________________________________________

Step 6: Create your Train Function

All that's left to be done is create an entry point function for your code.

def train(dataset_path, batch_size, epochs):
    all_text, all_labels, rev_labels = data_reader(dataset_path=dataset_path)

    print("target_encodings: ", all_labels)
    print("Number of training texts: ", len(all_text))

    with open(os.path.join(output_dir, "labels.json"), "w") as f:
        json.dump(rev_labels, f)

    train_generator = data_loader(bs=batch_size, y_lab=all_labels, data=all_text)

    model = model_arc(y_labels=all_labels)

    model.fit_generator(
        generator=train_generator,
        steps_per_epoch=(len(all_text) // batch_size),
        epochs=epochs,
    )
    tf.keras.models.save_model(model, filepath=os.path.join(output_dir, "models_v2"))

The job of this function is to call the respective steps in the chronological order as expected for Training the Model.

Step 7: Let it Train

dataset_path = "User passes the local path of the Dataset"

output_dir = "User passes the output directory where they want to store the model"
batch_size = 8
epochs = 3
train(
    dataset_path=dataset_path,
    batch_size=batch_size,
    epochs=epochs,
)

# OUTPUT ON THE TERMINAL
Epoch 1/3
1533/1533 [==============================] - 164s 107ms/step - loss: 1.1523 - accuracy: 0.5884
Epoch 2/3
1533/1533 [==============================] - 161s 105ms/step - loss: 0.3029 - accuracy: 0.9047
Epoch 3/3
1533/1533 [==============================] - 161s 105ms/step - loss: 0.1618 - accuracy: 0.9546

You can set the Hyper-parameters to your liking. As a recommendation do try out different parameters to see how it effects the training results and Accuracy.

Enjoy the State of the Art Accuracy with your Custom Model for your Personalised Dataset.

You can find the complete code base in the link mentioned below -

Congratulations ...

If you have followed the above steps then you should have a Custom trained model on your hand.

I hope this article finds you well and was easy to follow. In our further posts we will be looking at how we can create Custom Models using Cloud Native Solutions for working more closely with Industry Requirements.

STAY TUNED for more such content 😁