Air Paradis : Detect bad buzz with deep learning

Context

"Air Paradis" is an airline company who's marketing department wants to be able to detect quickly "bad buzz" on social networks, to be able to anticipate and address issues as fast as possible. They need an AI API that can detect "bad buzz" and predict the reason for it.

The goal here is to evaluate different approaches to detect "bad buzz" :

1. Baseline Model : Logistic Regression
2. Word embedding : Gensim Doc2Vec
3. Azure Cognitive Services : Text Analytics API
4. HuggingFace Transformer Pipeline : Sentiment Analysis
5. HuggingFace : BERT Fine-tuning
6. AzureML Studio : Automated ML
7. AzureML Studio : Designer
8. Custom Models : Neural Networks with Keras
9. AzureML Studio : Notebooks

After exploring our dataset, we will compare the different approaches.

Project modules

The helpers functions and project specific code will be placed in ../src/.

We will use the Python programming language, and present here the code and results in this Notebook JupyterLab file.

We will use the usual libraries for data exploration, modeling and visualisation :

• NumPy and Pandas : for maths (stats, algebra, ...) and large data manipulation
• Plotly : for interactive data visualization

We will also use libraries specific to the goals of this project :

• NLP Natural Language Processing
  • NLTK and Spacy : for text processing

import pickle

# Import custom helper libraries
import os
import sys

src_path = os.path.abspath(os.path.join("../src"))
if src_path not in sys.path:
    sys.path.append(src_path)

import data.helpers as data_helpers
import visualization.helpers as viz_helpers

# Maths modules
from scipy.stats import f_oneway
import pandas as pd

# Viz modules
import plotly.express as px

# Render for export
import plotly.io as pio

pio.renderers.default = "notebook"

Exploratory data analysis (EDA)

We are going to load the data and analyse the distribution of each variable.

Load data

Let's download the data from the Kaggle - Sentiment140 dataset with 1.6 million tweets competition.

# Download and unzip CSV files
!cd .. && make dataset && cd notebooks

>>> Downloading and extracting data files...
Data files already downloaded.
>>> OK.

Now we can load the data.

# Load data from CSV
df = pd.read_csv(
    os.path.join("..", "data", "raw", "training.1600000.processed.noemoticon.csv"),
    names=["target", "id", "date", "flag", "user", "text"],
)

# Reduce memory usage
df = data_helpers.reduce_dataframe_memory_usage(df)

Explore data

Let's display a few examples, find out how many data points are available, what are the variables and what is their distribution.

# Display first few rows
df.head(5)

	target	id	date	flag	user	text
0	0	1467810369	Mon Apr 06 22:19:45 PDT 2009	NO_QUERY	_TheSpecialOne_	@switchfoot http://twitpic.com/2y1zl - Awww, t...
1	0	1467810672	Mon Apr 06 22:19:49 PDT 2009	NO_QUERY	scotthamilton	is upset that he can't update his Facebook by ...
2	0	1467810917	Mon Apr 06 22:19:53 PDT 2009	NO_QUERY	mattycus	@Kenichan I dived many times for the ball. Man...
3	0	1467811184	Mon Apr 06 22:19:57 PDT 2009	NO_QUERY	ElleCTF	my whole body feels itchy and like its on fire
4	0	1467811193	Mon Apr 06 22:19:57 PDT 2009	NO_QUERY	Karoli	@nationwideclass no, it's not behaving at all....

# Diaplay number of rows and colmn types
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1600000 entries, 0 to 1599999
Data columns (total 6 columns):
 #   Column  Non-Null Count    Dtype
---  ------  --------------    -----
 0   target  1600000 non-null  Int8
 1   id      1600000 non-null  UInt32
 2   date    1600000 non-null  string
 3   flag    1600000 non-null  category
 4   user    1600000 non-null  string
 5   text    1600000 non-null  string
dtypes: Int8(1), UInt32(1), category(1), string(3)
memory usage: 48.8 MB

There are 1600000 rows, each composed of 6 columns :

• target: the polarity of the tweet (0 = negative, 2 = neutral, 4 = positive)
• id: The id of the tweet ( 2087)
• date: the date of the tweet (Sat May 16 23:58:44 UTC 2009)
• flag: The query (lyx). If there is no query, then this value is NO_QUERY.
• user: the user that tweeted (robotickilldozr)
• text: the text of the tweet (Lyx is cool)

We are only interrested in the target and text variables. The rest of the columns are not useful for our analysis.

# Drop useless columns
df.drop(columns=["id", "date", "flag", "user"], inplace=True)

# Replace target values with labels
df.target = df.target.map(
    {
        0: "NEGATIVE",
        2: "NEUTRAL",
        4: "POSITIVE",
    }
)

# Display basic statistics
df.describe(include="all")

	target	text
count	1600000	1600000
unique	2	1581466
top	NEGATIVE	isPlayer Has Died! Sorry
freq	800000	210

# Plot target distribution
viz_helpers.histogram(
    df, label_x="target", label_colour="target", title="Target distribution"
)

There are exactly as many (800000) POSITIVE tweets as NEGATIVE tweets. There are no NEUTRAL tweets. The problem is well balanced and there will be no bias towards one class during the training of our models.

# Plot text length distribution
df["text_length"] = df.text.str.len()

p_value = f_oneway(
    df.loc[df["target"] == "NEGATIVE", "text_length"],
    df.loc[df["target"] == "POSITIVE", "text_length"],
)[1]

viz_helpers.histogram(
    df,
    label_x="text_length",
    label_colour="target",
    title=f"Text length distribution / p-value={p_value:.5f}",
    include_boxplot=True,
)

There are no big difference between the POSITIVE and NEGATIVE tweets, but NEGATIVE tweets are slightly longer than POSITIVE tweets. In both classes, there are two modes : ~45 characters and 138 characters (the maximum allowed at some point).

# Plot word count distribution
df["word_count"] = df.text.str.split().str.len()

p_value = f_oneway(
    df.loc[df["target"] == "NEGATIVE", "word_count"],
    df.loc[df["target"] == "POSITIVE", "word_count"],
)[1]

viz_helpers.histogram(
    df,
    label_x="word_count",
    label_colour="target",
    title=f"Word count distribution / p-value={p_value:.5f}",
    include_boxplot=True,
)

There are no big difference between the POSITIVE and NEGATIVE tweets, but NEGATIVE tweets are significatively longer than POSITIVE tweets. In both classes, there are two modes : ~7 words and ~20 words.

Text analysis

We will look more in details at what contains the text variable.

First, we will transform the dataset into a Bag of Words representation with TfIdf (Term Frequency - Inverse Document Frequency) weights. To achieve this, we are going to use th SpaCy tokenizer.

# Vectorizers
from sklearn.feature_extraction.text import TfidfVectorizer

# Tokenizers, Stemmers and Lemmatizers
import nltk
from nltk.corpus import stopwords
import spacy

# Download resources
nltk.download("stopwords")
stopwords = set(stopwords.words("english"))

# Download SpaCy model
try:
    nlp = spacy.load("en_core_web_sm")
except:
    !python -m spacy download en_core_web_sm
    nlp = spacy.load("en_core_web_sm")

# Define tokenizer
tokenizer = lambda text: [  # SpaCy Lemmatizer
    token.lemma_.lower() for token in nlp(text) if token.is_alpha and not token.is_stop
]

2022-02-06 04:40:44.868994: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-02-06 04:40:44.869067: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
[nltk_data] Downloading package stopwords to
[nltk_data]     /home/clement/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!

# Processed data path
processed_data_path = os.path.join("..", "data", "processed")
vectorized_dataset_file_path = os.path.join(
    processed_data_path, "tfidf_spacy_dataset.pkl"
)
vocabulary_file_path = os.path.join(processed_data_path, "tfidf_spacy_vocabulary.pkl")

if os.path.exists(vectorized_dataset_file_path) and os.path.exists(
    vocabulary_file_path
):
    # Load vectorized dataset
    with (open(vectorized_dataset_file_path, "rb")) as f:
        X = pickle.load(f)
    # Load vocabulary
    with (open(vocabulary_file_path, "rb")) as f:
        vocabulary = pickle.load(f)
else:
    # Define vectorizer
    vectorizer = TfidfVectorizer(
        strip_accents="unicode",
        lowercase=True,
        stop_words=stopwords,
        tokenizer=tokenizer,
    )

    # Vectorize text
    X = vectorizer.fit_transform(df.text)

    # Get vocabulary
    vocabulary = vectorizer.get_feature_names_out()

    # Save vectorized dataset as pickle
    with open(vectorized_dataset_file_path, "wb") as f:
        pickle.dump(X, f)

    # Save vocabulary as pickle
    with open(vocabulary_file_path, "wb") as f:
        pickle.dump(vocabulary, f)

Our corpus is now transformed into a BoW representation. We can analyse the words frequencies.

# List words TF-IDF scores
words = pd.Series(X.sum(axis=0).A1, index=vocabulary)

# Top 20 tokens by TfIdf
top_20_words = words.nlargest(20).sort_values(ascending=False)

# Plot top 20 tokens by TfIdf
fig = px.bar(
    top_20_words,
    x=top_20_words.index,
    y=top_20_words.values,
    labels={"x": "Words", "y": "Count", "color": "Count"},
    title=f"Top 20 important words (Tf-Idf) - Vocalbulary size: {len(vocabulary)}",
    color=top_20_words.values,
)
fig.show()

We can see that the most important words actually meaningful and relevant regarding the sentiment associated to each message.

Models comparison

In this section, we are gonig to compare the metrics of the models we have tested in the other Notebooks.

Raw metrics

Each model is built and tested in the corresponding Notebooks (cf. list above). We are only looking at the classification metrics here.

models_results_df = pd.DataFrame(
    data=[
        {
            "Model": "1 - Logistic Regression",
            "True Positives": 108688,
            "True Negatives": 107510,
            "False Positives": 52490,
            "False Negatives": 51312,
            "Average Precision": 0.73,
            "ROC AUC": 0.74,
        },
        {
            "Model": "2 - Word embedding",
            "True Positives": 119134,
            "True Negatives": 105070,
            "False Positives": 54930,
            "False Negatives": 40866,
            "Average Precision": 0.75,
            "ROC AUC": 0.77,
        },
        {
            "Model": "3.1 - Azure Cognitive Service API",
            "True Positives": 788,
            "True Negatives": 673,
            "False Positives": 327,
            "False Negatives": 212,
            "Average Precision": 0.75,
            "ROC AUC": 0.78,
        },
        {
            "Model": "3.2 - Logistic Regression on Azure Cognitive Service",
            "True Positives": 164,
            "True Negatives": 123,
            "False Positives": 77,
            "False Negatives": 36,
            "Average Precision": 0.76,
            "ROC AUC": 0.78,
        },
        {
            "Model": "4 - HuggingFace Sentiment Analysis",
            "True Positives": 622,
            "True Negatives": 798,
            "False Positives": 202,
            "False Negatives": 378,
            "Average Precision": 0.79,
            "ROC AUC": 0.80,
        },
        {
            "Model": "5.1 - HuggingFace BERT Fine-tuning",
            "True Positives": 99585,
            "True Negatives": 17631,
            "False Positives": 82369,
            "False Negatives": 415,
            "Average Precision": 0.822,
            "ROC AUC": 0.883,
        },
        {
            "Model": "5.2 - HuggingFace BERTweet Fine-tuning",
            "True Positives": 89984,
            "True Negatives": 79351,
            "False Positives": 20649,
            "False Negatives": 10016,
            "Average Precision": 0.901,
            "ROC AUC": 0.915,
        },
        {
            "Model": "6.1 - AzureML Automated ML : 1h on CPU",
            "True Positives": 55214,
            "True Negatives": 48811,
            "False Positives": 23189,
            "False Negatives": 16786,
            "Average Precision": 0.79,
            "ROC AUC": 0.797,
        },
        {
            "Model": "6.2 - AzureML Automated ML : 10h on GPU",
            "True Positives": 111449,
            "True Negatives": 110539,
            "False Positives": 16552,
            "False Negatives": 17461,
            "Average Precision": 0.942,
            "ROC AUC": 0.942,
        },
        {
            "Model": "7.1 - AzureML Designer : Feature Hashing",
            "True Positives": 2114,
            "True Negatives": 2125,
            "False Positives": 1075,
            "False Negatives": 1086,
            "Average Precision": 0.663,
            "ROC AUC": 0.726,
        },
        {
            "Model": "7.2 - AzureML Designer : N-Gram Features",
            "True Positives": 2390,
            "True Negatives": 2285,
            "False Positives": 915,
            "False Negatives": 810,
            "Average Precision": 0.723,
            "ROC AUC": 0.811,
        },
        {
            "Model": "8.1 - FFNN on word counts",
            "True Positives": 129323,
            "True Negatives": 129822,
            "False Positives": 30178,
            "False Negatives": 30677,
            "Average Precision": 0.89,
            "ROC AUC": 0.89,
        },
        {
            "Model": "8.2 - FFNN on SpaCy Embedding",
            "True Positives": 126823,
            "True Negatives": 127274,
            "False Positives": 32726,
            "False Negatives": 33177,
            "Average Precision": 0.88,
            "ROC AUC": 0.88,
        },
        {
            "Model": "8.3 - FFNN on Gensim Doc2Vec Embedding",
            "True Positives": 119852,
            "True Negatives": 117321,
            "False Positives": 42679,
            "False Negatives": 40148,
            "Average Precision": 0.82,
            "ROC AUC": 0.82,
        },
        {
            "Model": "8.4 - FFNN with custom Embedding",
            "True Positives": 126888,
            "True Negatives": 130583,
            "False Positives": 29417,
            "False Negatives": 33112,
            "Average Precision": 0.88,
            "ROC AUC": 0.88,
        },
        {
            "Model": "8.5 - FFNN on Bert encoding",
            "True Positives": 128648,
            "True Negatives": 128653,
            "False Positives": 31347,
            "False Negatives": 31352,
            "Average Precision": 0.88,
            "ROC AUC": 0.88,
        },
        {
            "Model": "8.6 - RNN",
            "True Positives": 123295,
            "True Negatives": 134909,
            "False Positives": 25091,
            "False Negatives": 36705,
            "Average Precision": 0.89,
            "ROC AUC": 0.89,
        },
        {
            "Model": "8.7 - LSTM",
            "True Positives": 136020,
            "True Negatives": 127477,
            "False Positives": 32523,
            "False Negatives": 23980,
            "Average Precision": 0.90,
            "ROC AUC": 0.90,
        },
        {
            "Model": "8.8 - Bidirectional-LSTM",
            "True Positives": 130727,
            "True Negatives": 133962,
            "False Positives": 26038,
            "False Negatives": 29273,
            "Average Precision": 0.91,
            "ROC AUC": 0.91,
        },
        {
            "Model": "8.9 - Stacked Bidirectional-LSTM",
            "True Positives": 130822,
            "True Negatives": 133962,
            "False Positives": 26038,
            "False Negatives": 29178,
            "Average Precision": 0.91,
            "ROC AUC": 0.91,
        },
    ]
)

models_results_df["Accuracy"] = (
    models_results_df["True Positives"] + models_results_df["True Negatives"]
) / (
    models_results_df["True Positives"]
    + models_results_df["True Negatives"]
    + models_results_df["False Positives"]
    + models_results_df["False Negatives"]
)

models_results_df["F1"] = (
    2
    * models_results_df["True Positives"]
    / (
        2 * models_results_df["True Positives"]
        + models_results_df["False Positives"]
        + models_results_df["False Negatives"]
    )
)

models_results_df["Precision"] = models_results_df["True Positives"] / (
    models_results_df["True Positives"] + models_results_df["False Positives"]
)

models_results_df["Recall"] = models_results_df["True Positives"] / (
    models_results_df["True Positives"] + models_results_df["False Negatives"]
)

models_results_df["Sensitivity"] = models_results_df["True Positives"] / (
    models_results_df["True Positives"] + models_results_df["False Negatives"]
)

models_results_df["Specificity"] = models_results_df["True Negatives"] / (
    models_results_df["True Negatives"] + models_results_df["False Positives"]
)

for metrics in [
    ("Average Precision", "ROC AUC"),
    ("Accuracy", "F1"),
    ("Precision", "Recall"),
    ("Sensitivity", "Specificity"),
]:
    fig = px.scatter(
        models_results_df,
        x=metrics[0],
        y=metrics[1],
        color="Model",
        hover_name="Model",
    )
    fig.update_traces(marker_size=15)
    fig.show()

Observations

Best model

The best model is (by far) the model obtained by 6.2 - AzureML Automated ML : 10h on GPU :

• fine-tuned pre-trained BERT featurization layer
• MaxAbsScaler
• LightGBM

Pros

• best classification results overall
• training the best model only took ~5minutes
• unbiased : only 1.4% prediction difference between POSITIVE and NEGATIVE classes
• no hyper-parameters

Cons

• running the experiment to find the best model required 10 hours with GPU capability
• not easily explainable : the BERT layer doesn't allow to clearly identify important features

Off-the-shelf models (cloud or pre-trained)

These models (3.1 - Azure Cognitive Service API and 4 - HuggingFace Sentiment Analysis) have produced verage results.

Adding a classification model on top of Azure Cognitive Service prediction did not improve the results a lot (3.2 - Logistic Regression on Azure Cognitive Service).

Pros

• no training required
• quite good baseline
• HuggingFace model is free and easy to integrate and deploy
• Azure Cognitive Service can be used very easily as an REST API

Cons

• the results are not the best (HuggingFace's model is quite biased towards the NEGATIVE class)
• Azure's managed service has rate limits (100req/s , 1000req/min) can become expensive with a large number of requests (from €0.2243 up to €0.8970 per 1,000 text records )

Fine-tuned BERT model

We've seen that a fine-tuned BERT model can be very efficient as a pre-processing layer (cf. 6.2 - AzureML Automated ML : 10h on GPU).

But directly fine-tuning BERT for sentiment analysis (5.1 - HuggingFace : BERT Fine-tuning) proved to be a real challenge : we were only able to obtain average results after ~6.5h of training with GPU capability.

Using a more adapted model (5.2 - HuggingFace : BERTweet Fine-tuning) proved to greatly improve the results, at the cost of a longer training time (11h).

Given the BERT model has more than 109M parameters (134M for BERTweet), our dataset is probably too small to fine-tune the model efficiently.

Pros

• not the worst results, and largely improving with the amount of training data
• using a more adapted model is much more efficient

Cons

• not the best results, given the resources spent on training the model
• requires a LOT of resources (training data, CPU, GPU, time/money)

Custom Neural Networks

Starting from a basic model (8.1 - FFNN on word counts), adding a Embedding layer (8.4 - FFNN with custom Embedding), a Recurrent layers ((8.6 - RNN, 8.7 - LSTM, 8.8 - Bidirectional-LSTM and 8.9 - Stacked Bidirectional-LSTM) and a second stacked LSTM layer showed that more complexity does not mean better results.

Adding the Embedding layer actually reduced the classification results. Adding the Recurrent layer improved the model with Embedding layer. Adding the LSTM layer improved the results further more. Adding the Bidirectional-LSTM layer improved just slightly the results. Adding the Stacking a second Bidirectional-LSTM did not significatively change the results.

Pros

• best results with a fully custom model
• model complexity (and required training resources) can be controlled
• did not require huge amounts of training data and time/cpu/gpu resources

Cons

• Neural architecture search (NAS) is a challenge : there is no single Artificial Neural Network architecture to solve a problem. Building an ANN requires to train and evaluate models to decide the networks depth (number of layers), width (size of each layer), complexity (kind of layers)