About the Project

In this I will be building a Movie Recommender System based on IMDB Dataset.Recommender System are quite useful and popular these.whether it be an E-commerce company or food Delivery Service,Everyone requires a recommender system.It helps them to understand their customers while it also provides a smooth user flow for the users to go through the website.Throughout this Notebook ,I will be discussing various Approaches to build a recommender system ,common algorithms to build a recommender system and also a small demo of how recommendation will be shown to the user.

Exploring and Visualization

The Dataset we are using for this project can be found here.The Dataset comprises of 2 csv files.Lets import our libraries and see what data we have in the csv file

 
# linear algebra import numpy as np 
# data processing, CSV file I/O (e.g. pd.read_csv) 
import pandas as pd 
# for visualizations and plotting graphs 
import matplotlib.pyplot as plt 
import seaborn as sns 
#setting the style of grids
sns.set_style('whitegrid') 

There are two files tmdb_5000_movies.csv and tmdb_5000_credits.csv.Lets go through them one by one

 
#Load the First csv File 
data_movies = pd.read_csv('Data/tmdb_5000_movies.csv') 
#Load the second csv File 
data_credits = pd.read_csv('Data/tmdb_5000_credits.csv') 

 
#Print the first 1 rows of the data_movies 
data_movies.head(1) 

Now lets understand what each feature is

 data_movies.columns 

RangeIndex: 4803 entries, 0 to 4802
Data columns (total 20 columns):
budget                  4803 non-null int64
genres                  4803 non-null object
homepage                1712 non-null object
id                      4803 non-null int64
keywords                4803 non-null object
original_language       4803 non-null object
original_title          4803 non-null object
overview                4800 non-null object
popularity              4803 non-null float64
production_companies    4803 non-null object
production_countries    4803 non-null object
release_date            4802 non-null object
revenue                 4803 non-null int64
runtime                 4801 non-null float64
spoken_languages        4803 non-null object
status                  4803 non-null object
tagline                 3959 non-null object
title                   4803 non-null object
vote_average            4803 non-null float64
vote_count              4803 non-null int64
dtypes: float64(3), int64(4), object(13)
memory usage: 750.5+ KB
None

genre: states the genre of the film along with the genre id i.e comedy,drama
budget: budget of the movie
homepage: url of the movie website
keywords: indicates the plot of keywords which define the movies
genre: states the genre of the film along with the genre id i.e comedy,drama
original_languages: original language in which movie was produced
popularity : Popularity of movie
vote_avg : Average votes given to that movie
vote_count : total no of votes given to that movie

We will be exploring many of the columns later in the notebook and also understand what they

Exploratory Data Analysis

1.1 Original language

Lets find out what is most popular language in which films are produced

 
fig = plt.figure(figsize=(8,8)) 
sns.countplot(y=data_movies['original_language'],data=data_movies) 

1.2 Popular Genres

In this section we will analyze which genre is most common among movies. i.e on which genre most of the movies are based.

 
import ast genres_movies = dict() 
for movies in data_movies['genres']: 
  movie = ast.literal_eval(movies) 
  for genre in movie: 
    if(genre['name'] not in genres_movies.keys()): 
      genres_movies[genre['name']] = 1 
    else: 
      genres_movies[genre['name']]= genres_movies[genre['name']]+1 

Recommender Systems

Recommendation Systems are of mainly 3 different types.Each one offers a different level of personalization.Most recommender systems take either of two basic approaches: collaborative filtering or content-based filtering. Other approaches (such as hybrid approaches) also exist.

• Collaborative Filtering

  In Collaborative Filtering,recommendations are based on prior user experience.Collaborative filtering uses group knowledge to form a recommendation based on like users.

• Content Based Filtering

  Content-based filtering constructs a recommendation on the basis of a user's behavior.This definitely requires user browser history ,what kinds of content and blogs he is reading and recommends based on the History.Suppose we were to build a blog recommendation and if a particular user reads blogs about linux , the system would recommend blogs like operating system , ubuntu and other unix based operating.This is what user may like.

• Hybrid Approach

  In hybrid based Approach , that combines both the Collaborative and Content based approach is more complex as compared to both above.The Content based filtering could be used if the user data is not mature enough to be compared with other user data and give a cold start to the system but slight shift towards Collaborative approach when the user data grows enough.

Weighted Voting Approach

Weighted Voting Approach is simple common approach which doesn't takes any user activity into account. It simply uses a predefined ranking formula to rate movie based on their votes,views etc.

If we peek into our data,we have a lot of features like Vote Average, Vote Count,Popularity

Building our Simple Engine

The rating formula for IMDB is :

Now lets peek again into our movies data

 
data_movies.head(1) 

Lets calculate the mean vote,average vote

vote_counts = data_movies[data_movies['vote_count'].notnull()]['vote_count'].astype('int') 
# we will be using minimum votes as 95% percentile of the maximum votes received 
#calculating the average votes 
vote_averages = data_movies[data_movies['vote_average'].notnull()]['vote_average'].astype('int') 
# setting minimum vote requirements to be 85 percentile of the votes of the movies 
minimum_votes = votes_count.quantile(0.85) 
print 'minimum_votes :',minimum_votes 

1300

Now we have calculated the minimum vote requirements for the movies to be listed in our chart.Let's extract all the movies from our dataframe which qualifies above criteria

 
qualified = data_movies[(data_movies['vote_count'] >= m) & (data_movies['vote_count'].notnull()) & (data_movies['vote_average'].notnull())][['title', 'year', 'vote_count', 'vote_average', 'popularity', 'genres']] 
qualified['vote_count'] = qualified['vote_count'].astype('int')
qualified['vote_average'] = qualified['vote_average'].astype('int') 
qualified.shape 

(721, 6)

According to our criteria for listing movies we have 721 movies ready to be recommended to our user

Now we will use the standard IMDB rating formula to calculate the weighted rating for all qualified movies.

 
def weighted_rating(x): 
  v = x['vote_count'] 
  R = x['vote_average'] 
  return (v/(v+m) * R) + (m/(m+v) * C)

we have defined our weighted_rating function which calculates weighted rating for movies.

 
qualified['wr'] = qualified.apply(weighted_rating, axis=1) 
qualified = qualified.sort_values('wr', ascending=False).head(100) 

Now we have implemented a basic rating system which gives pretty good recommendations based on various movies metrics.We see that three Christopher Nolan Films, Inception, The Dark Knight and Interstellar occur at the very top of our chart. The chart also indicates a strong bias of IMDB Users towards particular genres and directors.

Now lets build recommendations based on particular movie genres.85% of movie genre ratings will be used to build our movie chart

Genre Based Recommendation

As we have seen above,we calculated the mean_vote and vote percentile for rating and listing movies,we will be using the same strategy to build recommendation

 
# Its returns movies if movie contains the specified genre 
def genre_movies(x,genre): 
  if genre in x['movie_genres']: 
    return x 

Now lets write a function that returns top movies belonging to particular genre

 
# it returns all movies of particular genre if they qualified movie criteria to be listed 
def genre_recommend(genre,percentile): 
  gen_movie = data_movies.apply(lambda x : ret_mov(x,genre),axis=1) 
  gen_movie = gen_movie.dropna() 
  vote_count= gen_movie[gen_movie['vote_count'].notnull()]['vote_count'].astype('int') 
  vote_averages = gen_movie[gen_movie['vote_average'].notnull()]['vote_average'].astype('int') 
  c = vote_averages.mean() 
  m = vote_counts.quantile(percentile) 
  gen_qualified = gen_movie[(gen_movie['vote_count'] >= m) & (gen_movie['vote_count'].notnull()) & (gen_movie['vote_average'].notnull())][['title', 'year', 'vote_count', 'vote_average', 'popularity', 'genres']] 
  gen_qualified['vote_count'] = gen_qualified['vote_count'].astype('int')
  gen_qualified['vote_average'] = gen_qualified['vote_average'].astype('int') 
  gen_qualified['wr'] = gen_qualified.apply(lambda x:weighted_rating(x),axis=1) 
  gen_qualified = gen_qualified.sort_values('wr',ascending=False) 
  return gen_qualified 

Now lets check our recommendations.

 
genre_recommend('Science Fiction',0.85).head(10) 

Now our movie recommender system recommends movies based on a particular genre.this could serve as one of the strategy for recommending movies.But it provides no personalization for users and recommendations are purely genre based.Let's improve our system by using content like movies overview,tagline for recommendations

Content Based Filtering

As we have already seen a couple of techniques for building recommendation engines we will be targeting a content based recommendation engine. we will be utilizing the tagline and overview of the to recommend the Movies.Lets start building our content based recommender system

 # create a shallow copy so that we don't alter the original dataframe 
 content_movies = data_movies.copy(False) 
 

As you can infer we have created a auxiliary dataframe to hold the overview and the tagline.Now lets merge these two columns and join them to create a description column in the content_movies dataframe.

 
# join overview and tagline content_movies
['description'] = content_movies['tagline'] + content_movies['overview'] 
#remove missing data 
content_movies['description'] = content_movies['description'].fillna('') 

We have joined the feature columns and remove useless data.Now we will extract features from the description using Tfidf(Term frequency inverse document frequency)

 
from sklearn.feature_extraction.text import TfidfVectorizer 
tf = TfidfVectorizer(analyzer='word',ngram_range=(1, 2),min_df=0, stop_words='english') tfidf_matrix = tf.fit_transform(content_movies['description']) 

Now lets see our tfidf matrix shape to analyze what what tfidf matrix really is

 
tfidf_matrix.shape 

(4803, 128039)

we have got our Tfidf matrix of shape 4803x128039.the 4803 is total no of movies in our dataframe.128039 is size of feature vector i.e we have got 128039 features for a single movie.Now we have to predict how similar a movie is to other movies using its feature vector.

Cosine Similarity

Cosine Similarity is a measure of similarity between two non zero vectors.It measures the cosine angle between two non zero vectors. cosine 0represents maximum similarity with value of cosine 1. cosine 0 represents least similarity.

Cosine similarity between two vectors can be mathematically represented as

In our case we have calculated the Tfidf matrix where each row represents a feature vector of a particular movie. Since we need to predict similarity between feature vectors of movies in Tfidf matrix ,we need to find the dot-product of tfidf matrix which will give us a cosine similarity score.This score denotes how similar movies are to each other.

We will be using sklearn's linear_kernel which is faster than dot product and gives us the similarity score

 
from sklearn.pairwise import linear_kernel 
cos_sim = linear_kernel(tfidf_matrix) 

 
cos_sim[0] 

array([ 1.        ,  0.00680476,  0.        , ...,  0.        ,
0.00344913,  0.        ])

Now lets build recommendation using our calculated similarity scores.Note that now movies will be recommended using the tagline and movie overview.This is a much better approach as compared to our previous approaches as now recommendations will much be user centric.

 
def improved_recommend(title): 
  idx = indices[title] 
  sim_scores = list(enumerate(cos_sim[idx])) 
  sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True) 
  sim_scores = sim_scores[1:31] movie_indices = [i[0] for i in sim_scores]
  return titles.iloc[movie_indices] 

 
improved_recommend('The Shawshank Redemption').head(20) 

637                     Les Misérables
3785                            Prison
609                        Escape Plan
2868                          Fortress
4727                      Penitentiary
434                   The Longest Yard
3978                          Amnesiac
1187                   Bridge of Spies
642                           Unbroken
891                    Man on the Moon
1779            The 40 Year Old Virgin
791                         Goosebumps
1171                Dumb and Dumber To
1166                          Get Hard
2570                 Ramona and Beezus
1597                          Southpaw
42                         Toy Story 3
2798    The Boy in the Striped Pyjamas
1860                Kiss of the Dragon
3871                 A Christmas Story
Name: title, dtype: object

It seems like recommendations have improved from previous approaches.Movies like The Prison and Escape Plan seems to have a similar story plot as The Shawshank Redemption.

Here we come to the End of the Project.We have build a movie recommendation system that recommends movies based on user choice and similarity of movies.

There are several approaches which could further improve the recommendations like using CastCrew Members,Director in the Description so that user will get much better recommendation.

Also hybrid approaches like using popularity as well as description of the movies to predict similarity .I might also try these approaches later.