An introduction to cosine similarity and sentence vectorisation
Published 2019-09-29
Table of Contents
In this post I will summarise and compare sentence similarity scoring using both bag of words and word embedding representations of the text. I will begin by introducing the idea of cosine similarity, a method for computing the similarity between two sentences.
From here, I will walk through the steps involved (with code) in creating sentence vectors, first using bag of words, then word embeddings. I will discuss each methods pros and cons, compare the results, and recommend ways in which my examples could be improved.
The extended code and dataset used to generate the examples discussed in the post are available on GitHub.
What is cosine similarity?
Cosine similarity is a popular NLP method for approximating how similar two word/sentence vectors are. The intuition behind cosine similarity is relatively straight forward, we simply use the cosine of the angle between the two vectors to quantify how similar two documents are.
From trigonometry we know that the Cos(0) = 1, Cos(90) = 0, and that 0 <= Cos(θ) <= 1. With this in mind, we can define cosine similarity between two vectors as follows:
By way of example, lets imagine we have 3 sentences – A, B, and C and we want to figure out which sentence (B or C) is most similar to A. Let us also assume that these sentences are represented by the following vectors:
-
A = [0.1, 0.7]
-
B = [0.2, 0.6]
-
C = [0.8, 0.1]
Using the code below, we can simply calculate the cosine similarity using the formula defined above to yield cosine_similarity (A, B) = 0.98 and cosine_similarity(A,C) = 0.26. With this result we can say that sentence A is more similar to B than C.
import numpy as np
def cosine_similarity_calc(vec_1,vec_2):
sim = np.dot(vec_1,vec_2)/(np.linalg.norm(vec_1)*np.linalg.norm(vec_2))
return sim
A = np.array([0.1,0.7])
B = np.array([0.2, 0.6])
C = np.array([0.8,0.1])
print('Sentence A and B smilarity:',cosine_similarity_calc(A,B))
print('Sentence A and C smilarity:',cosine_similarity_calc(A,C))Now that you understand cosine similarity, you might be wondering how do we convert a sentence to a vector? Well lucky for you, I am going to summarise, and compare two popular methods for sentence vectorisation.
Method: 1: Bag of words
Bag of words is a popular, and easy to implement method of sentence (and document) vectorisation whereby each sentence is represented by a vector which counts the occurrence of each word contained within the corpus using a document-term matrix.
Consider the following sentences from our sample dataset:
-
Sentence_A = “Hi, how are you?
-
Sentence_B = “Hey, what’s up?”
Using a bag of words based approach to vectorise these sentences would yield the following document-term matrix:
We can implement a bag of words approach very easily using the scikit-learn library, as demonstrated in the code below:
import pandas as pd
import string
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity
# function to remove punctuation from text (input is a string)
def clean_text(sentence):
clean_sentence = "".join(l for l in sentence if l not in string.punctuation)
return clean_sentence
# function to calculate cosine similarity using bow representation (input is a dataframe)
def bow_similarity(sentences_df):
# first lets clean the text by removing punctuation
sentences_df['clean_text'] = sentences_df.apply(lambda row: clean_text(row['sentence_text']), axis=1)
# initialise the bag of words tokeniser and apply it to our clean text
# this will create vector representations for each word
count_vec = CountVectorizer()
dtm = count_vec.fit_transform(sentences_df['clean_text']).toarray()
# calculate similarity, returns an NxN matrix with 1's across diagonal
similarity_df = pd.DataFrame(cosine_similarity(dtm)).reset_index()
# here we are going to unpivot the similairty matrix to return
# the data in a format like:
# | sentence_a | sentence_b | similarity |
# unpivot the similarity df
df_unpiv = pd.melt(similarity_df, id_vars=['index'])
# get unique combinations of sentance_a and sentance_b
df_unpiv_unique = (df_unpiv.loc[pd.DataFrame(np.sort(df_unpiv[['index', 'variable']],1),index=df_unpiv.index)
.drop_duplicates(keep='first')
.index])
# remove instances where sentence a == sentence b
df_unpiv_unique = df_unpiv_unique[df_unpiv_unique['index'] != df_unpiv_unique['variable']]
# now finally join on the original df to get the required o utput
# join the text
df_with_text = pd.merge(df_unpiv_unique, sentences_df.reset_index()
, left_on='index'
, right_on='index')
df_with_text = pd.merge(df_with_text, sentences_df.reset_index()
, left_on='variable'
, right_on='index')
df_with_text = df_with_text.loc[:,['sentence_text_y', 'sentence_text_x', 'value']]
df_with_text.columns = ['sentence_a', 'sentence_b', 'similarity']
return df_with_text
sentences = ['Hi, how are you?', 'Hey what\'s up?']
sentences_df = pd.DataFrame({'sentences':sentences})
print(bow_similarity(sample_sentences))Running this code will create the document-term matrix before calculating the cosine similarity between vectors A = [1,0,1,1,0,0,1], and B = [0,1,0,0,1,1,0] to return a similarity score of 0.00!!!!!
At this point we have stumbled across one of the biggest weaknesses of the bag of words method for sentence similarity…semantics.
While bag of words is intuitive and easy to implement, there is nothing in our document-term matrix which is capable of capturing the semantic similarity between words like “hi” and “hey”, and phrases such as “what’s up” and “how are you”.
As a result of this, we cannot rely on bag of words based systems for sentence similarity scoring. Instead, many state of the art approaches leverage word embeddings.
Method: 2: Word embedding
Word embeddings are vector representations of words which model semantic similarity through each words proximity to other words in the vector space. For example, words such as “hi” and “hello” will have similar coordinates to each other, which in turn will have very different coordinates to the word mathematics.
These embeddings are generated using deep learning architectures somewhat similar autoencoders. These architectures will analyse large corpora and create condensed vector representations of each word contained within the corpora.
This sounds too complicated for us to implement on our simple sentence similarity example right? Wrong, thankfully for us libraries such as spaCy make it very easy for us to generate word vectors for each word in our sample sentences. From here, we can simply take the average word vector for each word in the sentence and use this as our sentence vector.
The code listed below details the steps involved in implementing a simple word embedding based similarity system. Running this code with our two sample sentences will yield a similarity score of 0.83. This, while not perfect certainly captures the semantic similarity of these two sentences better than the bag of words approach.
import pandas as pd
import numpy as np
import string
import itertools
import spacy
# function to remove punctuation from text (input is a string)
def clean_text(sentence):
clean_sentence = "".join(l for l in sentence if l not in string.punctuation)
return clean_sentence
# function to calculate the cosine
def cosine_similarity_calc(vec_1,vec_2):
sim = np.dot(vec_1,vec_2)/(np.linalg.norm(vec_1)*np.linalg.norm(vec_2))
return sim
# function to calculate cosine similarity using word vectors (input is a series)
def embeddings_similarity(sentences):
# first we need to get data into | sentence_a | sentence_b | format
sentence_pairs = list(itertools.combinations(sentences, 2))
sentence_a = [pair[0] for pair in sentence_pairs]
sentence_b = [pair[1] for pair in sentence_pairs]
sentence_pairs_df = pd.DataFrame({'sentence_a':sentence_a, 'sentence_b':sentence_b})
# get unique combinations of sentance_a and sentance_b
sentence_pairs_df = sentence_pairs_df.loc[
pd.DataFrame(
np.sort(sentence_pairs_df[['sentence_a', 'sentence_b']],1),
index=sentence_pairs_df.index
).drop_duplicates(keep='first').index
]
# remove instances where sentence a == sentence b
sentence_pairs_df = sentence_pairs_df[sentence_pairs_df['sentence_a'] != sentence_pairs_df['sentence_b']]
# load word embeddings (will use these to convert sentence to vectors)
# Note you will need to run the following command (from cmd) to download embeddings:
# 'python -m spacy download en_core_web_lg'
embeddings = spacy.load('en_core_web_lg')
# now we are ready to calculate the similarity
sentence_pairs_df['similarity'] = sentence_pairs_df.apply(
lambda row: cosine_similarity_calc(
embeddings(clean_text(row['sentence_a'])).vector,
embeddings(clean_text(row['sentence_b'])).vector),
axis=1
)
return sentence_pairs_df
# calculate similarity for sample sentences
sentences = ['Hi, how are you?', 'Hey what\'s up?']
print(embeddings_similarity(sentences))Closing thoughts
While bag of words makes makes sense intuitively and is easy to implement, it fails to capture the semantic relationships that exists between words. Although, word embeddings help capture these semantic relationships, the simple approach of taking the average word vector across the sentence implemented in this tutorial fails to model the order, and context in which words appear and perhaps generalises the sentences too much. For example, in our full dataset the sentences “Hello, how are things?” and “Three bikers stop in town.” yielded a similarity score of 0.64.
We can of course employ word embeddings within a RNN/LSTM architecture which will analyse each word, and the order in which appear within the sentence to generate more sophisticated, and robust sentence similarity systems. This may be a topic for a future blog post…
See GitHub for full code and dataset