Data Wrangling(Gathering)¶
This phase of the project containts the following tasks which need to be done programmatically.
In [1]:
#Importing basic packages needed to get Data
import pandas as pd
import requests
import os
import tweepy
import json
1. Download Data Manually and Read in to check
In [2]:
archive = pd.read_csv('data/twitter-archive-enhanced.csv')
archive.head()
Out[2]:
2. Programmatically download data from a URL
In [3]:
# Here we have the URL provided by UDACITY
url = "https://d17h27t6h515a5.cloudfront.net/topher/2017/August/599fd2ad_image-predictions/image-predictions.tsv"
#fetching Data and saving to disk.
r = requests.get(url)
In [4]:
folderName = 'data'
fName = url.split('/')[-1]
#Creating Folder Named Data
if not os.path.exists(folderName):
os.makedirs(folderName)
In [5]:
#Writing Data to file
with open(os.path.join(folderName,fName),mode = 'wb') as file:
file.write(r.content)
In [6]:
#Reading in Downloaded Data to check if working.
img = pd.read_csv('data/image-predictions.tsv', sep='\t')
img.head()
Out[6]:
3. Downloading Twitter API Data For the Required Values
In [7]:
#Extracting Twitter Id's from the Archive DataFrame.
tweet_id = archive['tweet_id']
In [8]:
#Twiter Auth Data (Remove before sumbission)
consumer_key = 'VjFpwyCsbShxMv2ECEDWu71Uo'
consumer_secret = 'tLKupsqpJlJbGAE595oLptb4zVgyTVe5cGRaRQHOfnDt06w29e'
access_token = '2981974992-nCKD9ib35SsdrNN0HuMHKUNqpBCPvzWYZYtd0PR'
access_token_secret = 'msZMlp6w3mAjAxmiiqhIwgwntJPlyXMHHgX2wc5xgKMOg'
# consumer_key = ''
# consumer_secret = ''
# access_token = ''
# access_token_secret = ''
In [9]:
#Tweety Auth
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth,wait_on_rate_limit=True, wait_on_rate_limit_notify=True)
In [10]:
#Variable Array created to capure Index ID's of Errors using the Twitter API
api_error = []
Writing a loop to download additional data from the twitter api and save them as their respective text files to be used in the futher steps.
Folder Name Being saved to : tweets/
File name: tweet_[ID OF TWEET].txt
In [11]:
#Using a Try Except Block here to access the twitter API, the Error's are logged in the api_error varuiable if needed later.
counter = 0;
for t in tweet_id:
try:
counter = counter+1
fileName = 'data/tweet_json.txt'
tweet = api.get_status(id=t, tweet_mode='extended')
with open(fileName, 'a') as outfile:
json.dump(tweet._json,outfile)
outfile.write(','+'\n')
print(str(counter) +" Success")
except Exception:
print(str(counter)+" ERROR ERROR ERROR")
api_error.append(counter)
pass
In [12]:
#Errors for ID's, They have been not added to the DataFrame.
errors = [2056,1993,1945,1865,1836,1616,1310]
api_error
Out[12]:
In [15]:
#Loading in JSON File
with open('data/tweet_json.txt') as f:
data = json.loads(f.read())
In [16]:
data[0]
#Using this Dumped Data (first Instance Only) to parse the JSON using the following tool : http://json.parser.online.fr/ \
#and understand the structure of the JSON.
Out[16]:
In [18]:
#TEST CODE - Used to only check first Data
#Checking Queries on first Data Value printing them to the console.
Tid = data[0]['id_str']
full_text = data[0]['full_text']
retweet_count = data[0]['retweet_count']
fav_count = data[0]['favorite_count']
url = data[0]["extended_entities"]["media"][0]["url"]
index = data[0]['full_text'].index('/')
numerator = int(data[0]['full_text'][index-2:index])
denominator = int(data[0]['full_text'][index+1:index+3])
name = (data[0]['full_text'].split('.')[0].split(" ")[-1])
val = data[0]['full_text']
if 'doggo' in val:
dog = 'doggo'
elif 'pupper' in val:
dog = 'pupper'
elif 'puppo' in val:
dog = 'puppo'
elif 'floofer' in val:
dog = 'floofer'
else:
dog = None
print(Tid)
print(full_text)
print(retweet_count)
print(fav_count)
print(url)
print(index)
print(numerator)
print(denominator)
print(name)
In [19]:
#Another Check for parsing the JSON (it has quite a complicated Schema.)
data[0]["extended_entities"]["media"][0]["url"]
Out[19]:
Here we are extracting data which will be used to solve various data Quality issues.
These are Dog Name, Numerator, Denominator and the type of dog (doggo, fluffer, etc)
Creating the Final Dataframe : api¶
In [20]:
#Extracting Required feilds from JSON and making a new data frame df
df_list = []
for val in data:
Tid = val['id_str']
full_text = val['full_text']
retweet_count = val['retweet_count']
fav_count = val['favorite_count']
index = val['full_text'].index('/')
numerator = full_text[index-2:index]
denominator = full_text[index+1:index+3]
name = full_text.split('.')[0].split(" ")[-1]
if 'doggo' in full_text:
dog = 'doggo'
elif 'pupper' in full_text:
dog = 'pupper'
elif 'puppo' in full_text:
dog = 'puppo'
elif 'floofer' in full_text:
dog = 'floofer'
else:
dog = None
df_list.append({'tweet_id': int(Tid),
'full_text': full_text,
'retweet_count': int(retweet_count),
'fav_count' : int(fav_count),
'numerator' : numerator, #[Q#1]
'denominator': denominator, #[Q#3]
'pet_name' : name, #[Q#2]
'dog' : dog #[T#1]
})
api = pd.DataFrame(data=df_list)
In [25]:
#Checking our newly created DataFrame.
api.head()
Out[25]:
In [22]:
#Rearanging the DF Columns to make more sense when read in.
api = api[['tweet_id', 'full_text', 'fav_count','retweet_count', 'pet_name', 'dog', 'numerator', 'denominator']]
api.head()
Out[22]:
In [ ]:
#Saving Twitter Data extracted from API as CSV
#api.to_csv('data/twitter_archive_api.csv')
Testing Code is Below Please Check for Understanding how I reached the above solutions¶
In [24]:
#Sample Test Block for the loop used above.
df_api = []
for val in api['full_text']:
#Code for finding numberator and denominator
Tid = api["tweet_id"]
index = val.index('/')
rating_numerator = val[index-2:index]
rating_denominator = val[index+1:index+3]
name = (val.split('.')[0].split(" ")[-1])
if 'doggo' in val:
dog = 'doggo'
elif 'pupper' in val:
dog = 'pupper'
elif 'puppo' in val:
dog = 'puppo'
elif 'floofer' in val:
dog = 'floofer'
else:
dog = None
df_api.append({
'tweet_id' : Tid,
'name' : name,
'rating_numerator' : rating_numerator,
'rating_denominator' : rating_denominator,
'dog' : dog
})
df_api_pd = pd.DataFrame(data=df_api)
df_api_pd.head()
Out[24]:
In [26]:
#Sample code used for Calculating Numerator, Denominator and Pet Name
index = val.index('/')
print(val[index-2:index])
print(val[index+1:index+3])
print(val.split('.')[0].split(" ")[-1])
In [27]:
#Checking the value of variable val which contains full_text from the twitter api
val
Out[27]:
In [28]:
#Re initing Val with a diffrent Data value (without a dog Name)
val = api["full_text"][12]
val
Out[28]:
In [29]:
str(val)
if 'doggo' in val:
dog = ('doggo')
elif 'pupper' in val:
dog = ('pupper')
elif 'puppo' in val:
dog = ('puppo')
elif 'floofer' or 'floof' in val:
dog = ('floofer')
else:
dog = ("None")
Data Wrangling(Assessing)¶
In [30]:
#Importing basic packages needed to get Data
import pandas as pd
import requests
import os
import tweepy
import json
In [31]:
# Assigning agreed upon variable names
archive = pd.read_csv('data/twitter-archive-enhanced.csv')
img = pd.read_csv('data/image-predictions.tsv', sep='\t')
api = pd.read_csv('data/twitter_archive_api.csv')
In [32]:
archive.head()
Out[32]:
In [33]:
val = archive.text[12]
val.split('.')[0].split(" ")[-1]
# print(val.split('/')[1])
val
Out[33]:
In [34]:
index = val.index('/')
print(val[index-2:index])
print(val[index+1:index+3])
In [35]:
#Code for finding numberator and denominator
index = val.index('/')
rating_numerator = val[index-2:index]
rating_denominator = val[index+1:index+3]
In [ ]:
# index = val.index('This is')
# index
In [36]:
list(archive.columns.values)
Out[36]:
In [37]:
archive.count()
Out[37]:
In [38]:
img.head(20)
Out[38]:
In [39]:
img.sample(50)
Out[39]:
In [40]:
list(img.columns.values)
Out[40]:
In [41]:
api.head(30)
Out[41]:
In [42]:
#Listing All Colums Before dropping
list(api.columns.values)
Out[42]:
In [44]:
#Dropping First Column which has Just Numbers
api.drop(columns=['Unnamed: 0'], inplace=True)
#Listing All Colums Before dropping
list(api.columns.values)
Out[44]:
In [43]:
api.head(20)
Out[43]:
Programatic Assessment
In [45]:
archive.info()
In [46]:
archive.describe()
Out[46]:
In [47]:
api.info()
In [48]:
img.info()
In [49]:
api.head()
Out[49]:
In [51]:
# archive.subset(retweeted_status_id != 'NaN')
#archive(archive['retweeted_status_id'] != None)
#patients(patients['city'] == 'New York')
In [52]:
# CHecking for Duplicates throughout the data sets
all_colums = pd.Series(list(archive) + list(api) + list(img))
all_colums[all_colums.duplicated()]
Out[52]:
Detect and document at least eight (8) quality issues and two (2) tidiness issues in your wrangle_act.ipynb Jupyter Notebook.
In [53]:
archive.duplicated(['tweet_id']).sum()
Out[53]:
In [54]:
img.duplicated(['tweet_id']).sum()
Out[54]:
Data Wrangling (Cleaning)¶
In [55]:
#Importing Basic Packages
import pandas as pd
import numpy as np
import math
In [56]:
#Assigning agreed upon variable names (Original Data)
img = pd.read_csv('data/image-predictions.tsv', sep='\t')
api = pd.read_csv('data/twitter_archive_api.csv')
archive = pd.read_csv('data/twitter-archive-enhanced.csv')
In [57]:
#Creating Backups and Working on the *_clean Data.
img_clean = img.copy()
api_clean = api.copy()
archive_clean = archive.copy()
Quality Issues¶
archiveThe numerator needs to be Recalculated as told to us - can be taken from api column (needs to be Cleaned)archiveDog names are incorrect, need to re-extractarchiveDog Ratings are incorrect, need to re-extractarchiveRemove retwetted DataimgThe columns p1,p2,p3 have underscores separating their names. We should add white spacesimgif P1_dog ,P2_dog and P3_dog are all false. The tweet is Invalid and cannot be processed by the neural network and hence must be removed.apiremove stray count column (unnamed 0)
Tidyness Issues¶
achiveandapi: Combine doggo, pupper etc into one column- Merge into two Tables (get rid of
apimerge intoarchivedataset)
Solving Quality Issues¶
#1. archive The numerator Column needs to be recaluclated as mentioned.¶
Define
- This step was noted here, although it was easier to fix it in the gathering stage while the api data was being formed.
- I will leave a tooltip that Quality issue has been solved there. [Q#1]
- The Extra Columns will be Dropped Later in the Other Misc Operations.
Code
Test
#2. archive Dog names are incorret, need to re-extract¶
Define
- Solved in the Data Gathering Stage and columns will be Dropped Later [Q#2]
- The Extra Columns will be Dropped Later in the Other Misc Operations.
Code
Test
#3. archive Dog Ratings are incorrect, need to re-extract¶
Define
- Solved in the Data Gathering Stage and columns will be Dropped Later [Q#3]
- The Extra Columns will be Dropped Later in the Other Misc Operations.
Code
Test
#4. archive Remove retwetted Data
¶
Define
Looking at the Column names 'retweeted_status_id' stands out as the defacto proof that the tweet has been retweeted. We hence Check to see this value should be null throughout the entire column and just keep the values which have 'NAN'
- Remove Any rows containting any other Values other than NaN in the
retweeted_status_idcolumnn - Looking at the Column names 'retweeted_status_id' stands out as the defacto proof that the tweet has been retweeted. We hence Check to see this value should be null throughout the entire column and just keep the values which have 'NAN'
Code
In [58]:
archive_clean = archive[archive.retweeted_status_id.isnull()]
archive_clean.head()
Out[58]:
Test
- Should be blank - This means we have removed all the retweeted Data.
In [59]:
archive_clean[archive_clean.retweeted_status_id.notnull()]
Out[59]:
#5. img The columns p1,p2,p3 have underscores seperating their names¶
Define
- Remove underscores with whitespaces to increase readability
In [60]:
img.head(1)
Out[60]:
Code
In [61]:
#Remvoving Whitespace using str.replace()
img_clean.p1 = img_clean.p1.str.replace('_',' ')
img_clean.p2 = img_clean.p2.str.replace('_',' ')
img_clean.p3 = img_clean.p3.str.replace('_',' ')
Test
In [62]:
#Checking if the above solution worked.
img_clean.head()
Out[62]:
p1, p2 & p3 Columns have whiteSpaces. - Success
#6. img if P1_dog ,P2_dog and P3_dog are all false.¶
Define
- The tweet is Invalid if
p1_dog, p2_dog & p3_dogare allFALSEand cannot be processed by the neural network and hence must be removed.
Code
In [63]:
#looking at the original DF
img.head()
Out[63]:
In [64]:
#Keeping only the True Values
#img_clean = img_clean[~(img_clean.p1_dog & img_clean.p2_dog & img_clean.p3_dog)]
img.count()
Out[64]:
In [65]:
#Keeping only the True Values
img_clean = img_clean[~((img_clean.p1_dog == False) & (img_clean.p2_dog == False) & (img_clean.p3_dog == False))]
img_clean.count()
Out[65]:
Test
In [66]:
img_clean[(img_clean.p1_dog == False) & (img_clean.p2_dog == False) & (img_clean.p3_dog == False)]
Out[66]:
#7 api remove stray count column ['unnamed 0']¶
Define
-When the API csv was imported there was a stray column, we must drop it as it will cause issues later while merging
In [67]:
api.head(0)
Out[67]:
Code
In [68]:
#Dropping Stray Column
api_clean.drop(columns=['Unnamed: 0'], inplace=True)
Test
In [69]:
api_clean.head()
Out[69]:
Solving Tidiness Issues¶
#2. Merge into two Tables (get rid of api merge into archive dataset)¶
Define
As per the final project specification we need to merge these two tables to eliminate redudant data
These Tables are
- archive
- api
Code
In [70]:
#Merge into two Tables (get rid of `api` merge into `archive` dataset)
#Saving new DF as var name 'final'
final = pd.merge(archive_clean,api_clean,how='right',on='tweet_id')
Test
In [71]:
print(final.columns)
#1. archive and api : Combine doggo, pupper etc into one column¶
Define
- We Extracted this column directly in the gathering stage while working with the Twitter JSON
- This value is named
dogin the api table, we can replace the columnsfloofer,pupper,puppo,doggowith the single column dog - The Remaining Columns mayebe Dropped.
- referenced for Operations done during gathering [T#1]
Code
In [72]:
#Solving Tidiness Issue #1
#1. `achive` and `api` : Combine doggo, pupper etc into one column
final.drop(columns=['floofer','pupper','puppo','doggo'], inplace=True)
Test
In [73]:
final.head()
Out[73]:
Other Misc Operations¶
Define
- Dropping Extra Columns which were recalculated Before
names[Q#2] - Dropping
rating_numeratorandrating_denominator[Q#1] [Q#3] - Dropping extra repeated column
text
Code
In [74]:
final.drop(columns=['name','rating_numerator','rating_denominator','text'], inplace=True)
#[Q#2] Solved.
#[Q#1], [Q#3] Solved,
We need to clean the numerator column¶
Define
The code we wrote turned out some errors, we shall manual clean this.
In [75]:
#We Need to check what values should be not there
final.numerator.value_counts()
Out[75]:
Code
In [76]:
#Making a list of which rows have errors - using value_counts() for refrence.
numerator_error = []
numerator_error.append(final[final.numerator == '.9']['tweet_id'])
numerator_error.append(final[final.numerator == '.5']['tweet_id'])
numerator_error.append(final[final.numerator == '\r\n9']['tweet_id'])
numerator_error.append(final[final.numerator == 'ry']['tweet_id'])
numerator_error.append(final[final.numerator == '.8']['tweet_id'])
numerator_error.append(final[final.numerator == '.9']['tweet_id'])
numerator_error.append(final[final.numerator == ';2']['tweet_id'])
numerator_error.append(final[final.numerator == '(8']['tweet_id'])
numerator_error.append(final[final.numerator == 'st']['tweet_id'])
numerator_error.append(final[final.numerator == '\r\n5']['tweet_id'])
numerator_error.append(final[final.numerator == '-5']['tweet_id'])
numerator_error.append(final[final.numerator == ' w']['tweet_id'])
numerator_error
Out[76]:
Here we start to Manually Clean the data.
Steps
- First we check which rows have the error prone data, look at each dataframes
full_text - Using our judgement we manually change the values
- This is although time consuming, since there are just a few errors it is doable.
In [77]:
final[final.numerator == '.9'].full_text
final.loc[[847,1192,1430],'numerator'] = 9
In [78]:
final[final.numerator == '.5']
Out[78]:
In [79]:
final.loc[42].full_text
final.loc[[42],'numerator'] = 13.5
In [80]:
final.loc[1509].full_text
final.loc[[1509],'numerator'] = 9.5
In [81]:
final[final.numerator == '\r\n9']
Out[81]:
In [82]:
final.loc[1492].full_text
final.loc[[1492,2082],'numerator'] = 9
In [83]:
final[final.numerator == 'ry']
Out[83]:
In [84]:
final.loc[2329].full_text
final.loc[[2329],'numerator'] = 11
In [85]:
final[final.numerator == '.8']
Out[85]:
In [86]:
final.loc[1255].full_text
final.loc[[1255],'numerator'] = 8
In [87]:
final[final.numerator == '.9']
Out[87]:
In [88]:
final[final.numerator == ';2']
Out[88]:
In [89]:
final.loc[2066].full_text
final.loc[[2066],'numerator'] = 2
In [90]:
final[final.numerator == '(8']
Out[90]:
In [91]:
final.loc[1473].full_text
final.loc[[1473],'numerator'] = 8
In [92]:
final[final.numerator == 'st']
Out[92]:
In [93]:
final.loc[1635].full_text
final.loc[[1635],'numerator'] = 12
In [94]:
final[final.numerator == '\r\n5']
Out[94]:
In [95]:
final.loc[1912].full_text
final.loc[[1912],'numerator'] = 5
In [96]:
final[final.numerator == '-5']
Out[96]:
In [97]:
final[final.numerator == ' w']
Out[97]:
In [98]:
final.loc[1069].full_text
final.loc[[1069],'numerator'] = 3
Test
- All Stray values eliminated (especially strings)
In [99]:
print(final.numerator.value_counts())
We need to clean the denominator column as well¶
Define
The code we wrote turned out some errors, we shall manual clean this.
We need to eliminate string values
Code
In [100]:
final.denominator.value_counts()
Out[100]:
In [101]:
final[final.denominator == 'pe']
Out[101]:
In [102]:
final.loc[2329].full_text
final.loc[[2329],'denominator'] = 10
In [103]:
final[final.denominator == 'ou']
Out[103]:
In [104]:
final.loc[1069].full_text
final.loc[[1069],'denominator'] = 10
In [105]:
final[final.denominator == 'sw']
Out[105]:
In [106]:
final.loc[1635].full_text
final.loc[[1635],'denominator'] = 10
Test
- All String Values eliminated
In [107]:
final.denominator.value_counts()
Out[107]:
Test
In [ ]:
final.columns
Final Code to Save¶
RUN LAST AND UNCOMMENT
In [ ]:
#Save Files to CSV
#final.to_csv('data/final/twitter_archive_master.csv')
#img_clean.to_csv('data/final/image_predictions.csv')
#print("Saved Successfully")
Data Wrangling (Visualizations)¶
In [108]:
#Import Statements
%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from datetime import datetime
In [109]:
#importing Final Data found in Data/final Folder
archive = pd.read_csv('data/final/twitter_archive_master.csv',index_col=0)
img = pd.read_csv('data/final/image_predictions.csv', index_col=0)
print("Import Successful")
1. Analysing fav_count from archive¶
In [110]:
fav_mean = archive.fav_count.mean()
fav_median = archive.fav_count.median()
fav_max = archive.fav_count.max()
fav_sum = archive.fav_count.sum()
In [111]:
archive.fav_count.count()
Out[111]:
In [112]:
print("Mean Favourite Value is : {}".format(fav_mean))
print("Median Favourite Value is : {}".format(fav_median))
print("Max Favourite Value for an tweet is : {}".format(fav_max))
print("Total Favourite secured for All Tweets : {}".format(fav_sum))
In [113]:
fav_plt = archive.fav_count.hist(alpha=0.8,figsize=(8,8))
plt.xlabel("Fav Counts");
plt.ylabel("Count of Tweets");
plt.title("Favorited Tweets");
plt.savefig('Docs/Viz/1.png');
2. Analysing retweet_count from archive¶
In [114]:
archive
Out[114]:
In [115]:
ass = np.sort(archive.retweet_count)[::-1]
In [116]:
ass
Out[116]:
In [117]:
retweet_mean = archive.retweet_count.mean()
retweet_median = archive.retweet_count.median()
retweet_max = archive.retweet_count.max()
retweet_sum = archive.retweet_count.sum()
In [118]:
archive.retweet_count.hist(alpha=0.8,figsize=(8,8),color = "green")
plt.xlabel("Retweet Counts");
plt.ylabel("Count of Tweets");
plt.title("Re-Tweeted Tweets");
plt.savefig('Docs/Viz/2.png');
In [119]:
print("Mean Retweets Value is : {}".format(retweet_mean))
print("Median Retweets Value is : {}".format(retweet_median))
print("Max Retweets Value for an tweet is : {}".format(retweet_max))
print("Total Retweets secured for All Tweets : {}".format(retweet_sum))
3. Analysing Dog Names from archive¶
In [120]:
pie = archive.dog.value_counts()
pie.plot(kind="pie");
plt.savefig('Docs/Viz/3.png');
In [121]:
dog_val = archive.dog.value_counts()
name_sum = dog_val[0]+dog_val[1]+dog_val[2]+dog_val[3]
pupper_per = (dog_val[0]/name_sum)*100
doggo_per = (dog_val[1]/name_sum)*100
puppo_per = (dog_val[2]/name_sum)*100
floofer_per = (dog_val[3]/name_sum)*100
print("The Percentile Value of Pupper to all dogs is {}%".format(pupper_per))
print("The Percentile Value of Doggo to all dogs is {}%".format(doggo_per))
print("The Percentile Value of Puppo to all dogs is {}%".format(puppo_per))
print("The Percentile Value of Floofer to all dogs is {}%".format(floofer_per))
In [122]:
dog_val.sum()
Out[122]:
4. Analysing numerators & denominators from archive¶
In [123]:
num = archive.numerator
dom = archive.denominator
num_mean = num.mean()
num_median = num.median()
num_max = archive.numerator.max()
dom_mean = dom.mean()
dom_median = dom.median()
In [124]:
print("The mean value of all the numerator of the ratings given is : {}".format(num_mean))
print("The median value of all the numerator of the ratings given is : {}".format(num_median))
print("The mean value of all the denominators of the ratings given is : {}".format(dom_mean))
print("The median value of all the denominators of the ratings given is : {}".format(dom_median))
print("".format())
In [125]:
num_plt = num.plot(figsize=(10,10), kind='hist', color="#ff9960");
plt.ylabel("Tweets")
plt.xlabel("Numerator Values")
plt.title("Numerator Histogram");
# num_plt.axes.get_yaxis().set_visible(False)
plt.savefig('Docs/Viz/4.png');
In [126]:
print("The Maximum Rating Numerator given is {}".format(num_max))
#5 Finding out about the posting habits¶
In [127]:
time_dt = pd.to_datetime(archive.timestamp).dt.date
time_hr = pd.to_datetime(archive.timestamp).dt.hour
time_yr = pd.to_datetime(archive.timestamp).dt.year
In [128]:
time_hr.value_counts()
time_hr.plot(figsize=(8,6), kind='hist');
plt.title("Hourly Posting Graph");
plt.savefig('Docs/Viz/5.png');
In [129]:
time_dt = time_dt.value_counts()
time_dt.plot(figsize=(15,10));
plt.title("Daily Posting Graph");
plt.savefig('Docs/Viz/6.png');
In [130]:
time_dt.value_counts()
Out[130]:
In [131]:
time_yr.value_counts().plot(kind='bar');
plt.title("Yearly Graph Figure");
plt.savefig('Docs/Viz/7.png');
Analyzing the img DataSet¶
In [132]:
#Viewing The DataSet
img.head()
Out[132]:
#6 Calculating the mean values for the images uploaded per post from img¶
In [133]:
#Calulating the mean values for the images uploaded per post
img_uploaded_mean = img.img_num.mean()
img_uploaded_median = img.img_num.median()
In [134]:
print("The Average amount of pictures uploaded per tweet is : {}".format(img_uploaded_mean))
print("The Median amount of the uploaded photos is : {}".format(img_uploaded_median))
Neural Network Analysis
- We can gauge the efficiency of the algorithm by seeing it's prediction strength.
- This is a value from 0.00 to 0.99(or 1)
- Each phase of the neural analysis (p1_conf, p2_conf, p3_conf) has a hit ratio and we can gauge a lot about the network works.
#7 Finding the efficiency of the Prediction inp1_conf,p2_conf & p3_conf¶
In [135]:
# Calulating the mean values for the confidence prediction varibles
Cp1_mean = img.p1_conf.mean()
Cp2_mean = img.p2_conf.mean()
Cp3_mean = img.p3_conf.mean()
In [136]:
print("Calculating the Efficiency of the Neural Network on the diffrent stages p1,p2,p3")
print("The Average Efficiency of Stage one P1 {} ".format(Cp1_mean))
print("The Average Efficiency of Stage one P2 {} ".format(Cp2_mean))
print("The Average Efficiency of Stage one P3 {} ".format(Cp3_mean))
#8 Finding Hit rate of the neural network through the different stages¶
In [137]:
#using counts to get true false valuses of the data
prediction_p1 = img.p1_dog.value_counts()
prediction_p2 = img.p2_dog.value_counts()
prediction_p3 = img.p3_dog.value_counts()
#Finding percentile values of each
p1_per = prediction_p1[1]/ (prediction_p1[0]+prediction_p1[1])*100
p2_per = prediction_p2[1]/ (prediction_p2[0]+prediction_p2[1])*100
p3_per = prediction_p3[1]/ (prediction_p3[0]+prediction_p3[1])*100
In [138]:
#Printing Above found percentiles.
print("P1 Stage Success Hit Rate is {} %".format(p1_per))
print("P2 Stage Success Hit Rate is {} %".format(p2_per))
print("P3 Stage Success Hit Rate is {} %".format(p3_per))
#9 Analyzing which dogs are the most popular through different stages of the neural network.¶
In [139]:
#Anaylzing which dogs are the most popular through diffrent stages of the neural network.
d_p1 = img.p1.value_counts()
d_p2 = img.p2.value_counts()
d_p3 = img.p3.value_counts()
In [140]:
#Dumping the Data to Read and Anaylyze
print("Finding out the most popular Dogs for Each Stage \n")
#print("P1")
print("The Top Popular Dogs for Stage P1 Are :\n{} \n".format(d_p1.head()))
#print("P2")
print("The Top Popular Dogs for Stage P2 Are :\n{} \n".format(d_p2.head()))
#print("P3")
print("The Top Popular Dogs for Stage P3 Are :\n{} \n".format(d_p3.head()))
In [141]:
# Merging all the Data into one Series so i can get a better picutre for joint anaylysis
all_dogs = pd.concat([d_p1, d_p2, d_p3])
d_all = all_dogs.groupby(all_dogs.index).aggregate(sum)
d_all = d_all.sort_values(ascending=False)
print("The Top 10 Dogs overall through all the stages in our DataSet are \n\n{}\n".format(d_all.head(10)))
Thank You for Reading :)