Rych Data

Elizabeth's Experiments in Data Science

About

IPYNB: News API, Google Translate API, Google Natural Language API

// under API Google machine learning python jupyter notebooks

Choose Your News - Full Python Code

The NewsAPI

In [1]:
import requests
import json
import pprint
import pandas as pd 
import os
import newsapi
import matplotlib.pyplot as plt
import seaborn as sns

newsapi offers two classes, Articles and Sources, that redirect to two endpoints, https://newsapi.org/v1/articles and https://newsapi.org/v1/sources, respectively.

Articles

In [2]:
import newsapi
apikey = 'insertkeynumber'
In [3]:
# instantiate an article object
from newsapi.articles import Articles
a = Articles(API_KEY='')
a.get(source="bbc-news")
Out[3]:
{'articles': [{'author': 'BBC News',
   'description': "Amateur footage shows Key Largo, where the sea has been drawn away towards the hurricane's centre.",
   'publishedAt': '2017-09-10T15:33:39Z',
   'title': 'Hurricane Irma: Storm drains bay in Florida Keys',
   'url': 'http://www.bbc.co.uk/news/av/world-us-canada-41220778/hurricane-irma-storm-drains-bay-in-florida-keys',
   'urlToImage': 'https://ichef.bbci.co.uk/news/1024/cpsprodpb/847A/production/_97741933_mediaitem97741932.jpg'},
  {'author': 'BBC News',
   'description': 'BBC correspondents report from the British Virgin Islands, which were ravaged by Hurricane Irma.',
   'publishedAt': '2017-09-10T12:48:09Z',
   'title': 'Hurricane Irma: Troops go to work on devastated Tortola',
   'url': 'http://www.bbc.co.uk/news/uk-41218817',
   'urlToImage': 'https://ichef.bbci.co.uk/news/1024/cpsprodpb/80E1/production/_97739923_boats.jpg'},
  {'author': 'BBC News',
   'description': 'The operating system of unreleased iPhones is leaked to two websites, revealing secret details.',
   'publishedAt': '2017-09-10T17:26:30Z',
   'title': "Apple suffers 'major iPhone X leak'",
   'url': 'http://www.bbc.co.uk/news/technology-41220517',
   'urlToImage': 'https://ichef.bbci.co.uk/news/1024/cpsprodpb/F054/production/_97742516_whatsubject.jpg'},
  {'author': 'BBC News',
   'description': "Footage emerges from Oaxaca state after 90 people died in the country's strongest quake for a century.",
   'publishedAt': '2017-09-10T11:19:08Z',
   'title': 'Mexico earthquake: Drone reveals aftermath',
   'url': 'http://www.bbc.co.uk/news/av/world-latin-america-41219113/mexico-earthquake-drone-reveals-aftermath',
   'urlToImage': 'https://ichef.bbci.co.uk/news/1024/cpsprodpb/E603/production/_97738885_p05fnl3q.jpg'},
  {'author': 'BBC News',
   'description': 'China has started "relevant research" but has not yet decided when the ban would come into force.',
   'publishedAt': '2017-09-10T10:25:24Z',
   'title': 'China looks at plans to ban petrol and diesel cars',
   'url': 'http://www.bbc.co.uk/news/business-41218243',
   'urlToImage': 'https://ichef.bbci.co.uk/news/1024/cpsprodpb/2E6B/production/_97738811_gettyimages-82199575.jpg'},
  {'author': 'BBC News',
   'description': 'A government spokesman said: "We have no policy to negotiate with terrorists."',
   'publishedAt': '2017-09-10T15:11:19Z',
   'title': 'Rohingya crisis: Myanmar rebuffs rebel ceasefire',
   'url': 'http://www.bbc.co.uk/news/world-asia-41220257',
   'urlToImage': 'https://ichef.bbci.co.uk/images/ic/1024x576/p05fp0wn.jpg'},
  {'author': 'BBC News',
   'description': 'The stateless former president of Georgia and governor in Ukraine had been stopped at the border.',
   'publishedAt': '2017-09-10T18:02:34Z',
   'title': 'Former governor Mikheil Saakashvili forces entry to Ukraine',
   'url': 'http://www.bbc.co.uk/news/world-europe-41219773',
   'urlToImage': 'https://ichef.bbci.co.uk/news/1024/cpsprodpb/4D70/production/_97742891_mediaitem97742886.jpg'},
  {'author': 'BBC News',
   'description': "At least 400 children are believed to be buried in a section of St Mary's Cemetery in Lanark.",
   'publishedAt': '2017-09-10T04:52:47Z',
   'title': "Bodies of 'hundreds' of children buried in mass grave",
   'url': 'http://www.bbc.co.uk/news/uk-41200949',
   'urlToImage': 'https://ichef.bbci.co.uk/news/1024/cpsprodpb/6893/production/_97717762_smyllum-editedforiplayer.png'},
  {'author': 'BBC News',
   'description': "The celebrity baker said he went to a 2003 party in a WW2 German uniform as an 'Allo 'Allo character.",
   'publishedAt': '2017-09-10T11:06:02Z',
   'title': 'Bake Off: Paul Hollywood in Nazi outfit apology',
   'url': 'http://www.bbc.co.uk/news/uk-41218525',
   'urlToImage': 'https://ichef.bbci.co.uk/news/1024/cpsprodpb/DD07/production/_97738565_hollywood_alloallo_pa_bbc.jpg'},
  {'author': 'BBC News',
   'description': 'Jens Stoltenberg tells the BBC that Nato and its allies want a peaceful solution to the nuclear threat.',
   'publishedAt': '2017-09-10T14:38:08Z',
   'title': 'North Korea: Nato chief says global response needed',
   'url': 'http://www.bbc.co.uk/news/world-asia-41219767',
   'urlToImage': 'https://ichef.bbci.co.uk/news/1024/cpsprodpb/63A0/production/_97740552_hi041584494.jpg'}],
 'sortBy': 'top',
 'source': 'bbc-news',
 'status': 'ok'}

The .get() method takes three arguments: source (required), sort_by (optional), and attributes_format (optional Default:True). The source is the identifer for the news source or blog you want headlines from. The sort_by argument lets the user specify which type of list they want. The possible options are top, latest, and popular. Note: not all options are available for all sources (default: top).

In [4]:
# shows the functionality of the get method. 
a.get(source="bbc-news")
Out[4]:
{'articles': [{'author': 'BBC News',
   'description': "Amateur footage shows Key Largo, where the sea has been drawn away towards the hurricane's centre.",
   'publishedAt': '2017-09-10T15:33:39Z',
   'title': 'Hurricane Irma: Storm drains bay in Florida Keys',
   'url': 'http://www.bbc.co.uk/news/av/world-us-canada-41220778/hurricane-irma-storm-drains-bay-in-florida-keys',
   'urlToImage': 'https://ichef.bbci.co.uk/news/1024/cpsprodpb/847A/production/_97741933_mediaitem97741932.jpg'},
  {'author': 'BBC News',
   'description': 'BBC correspondents report from the British Virgin Islands, which were ravaged by Hurricane Irma.',
   'publishedAt': '2017-09-10T12:48:09Z',
   'title': 'Hurricane Irma: Troops go to work on devastated Tortola',
   'url': 'http://www.bbc.co.uk/news/uk-41218817',
   'urlToImage': 'https://ichef.bbci.co.uk/news/1024/cpsprodpb/80E1/production/_97739923_boats.jpg'},
  {'author': 'BBC News',
   'description': 'The operating system of unreleased iPhones is leaked to two websites, revealing secret details.',
   'publishedAt': '2017-09-10T17:26:30Z',
   'title': "Apple suffers 'major iPhone X leak'",
   'url': 'http://www.bbc.co.uk/news/technology-41220517',
   'urlToImage': 'https://ichef.bbci.co.uk/news/1024/cpsprodpb/F054/production/_97742516_whatsubject.jpg'},
  {'author': 'BBC News',
   'description': "Footage emerges from Oaxaca state after 90 people died in the country's strongest quake for a century.",
   'publishedAt': '2017-09-10T11:19:08Z',
   'title': 'Mexico earthquake: Drone reveals aftermath',
   'url': 'http://www.bbc.co.uk/news/av/world-latin-america-41219113/mexico-earthquake-drone-reveals-aftermath',
   'urlToImage': 'https://ichef.bbci.co.uk/news/1024/cpsprodpb/E603/production/_97738885_p05fnl3q.jpg'},
  {'author': 'BBC News',
   'description': 'China has started "relevant research" but has not yet decided when the ban would come into force.',
   'publishedAt': '2017-09-10T10:25:24Z',
   'title': 'China looks at plans to ban petrol and diesel cars',
   'url': 'http://www.bbc.co.uk/news/business-41218243',
   'urlToImage': 'https://ichef.bbci.co.uk/news/1024/cpsprodpb/2E6B/production/_97738811_gettyimages-82199575.jpg'},
  {'author': 'BBC News',
   'description': 'A government spokesman said: "We have no policy to negotiate with terrorists."',
   'publishedAt': '2017-09-10T15:11:19Z',
   'title': 'Rohingya crisis: Myanmar rebuffs rebel ceasefire',
   'url': 'http://www.bbc.co.uk/news/world-asia-41220257',
   'urlToImage': 'https://ichef.bbci.co.uk/images/ic/1024x576/p05fp0wn.jpg'},
  {'author': 'BBC News',
   'description': 'The stateless former president of Georgia and governor in Ukraine had been stopped at the border.',
   'publishedAt': '2017-09-10T18:02:34Z',
   'title': 'Former governor Mikheil Saakashvili forces entry to Ukraine',
   'url': 'http://www.bbc.co.uk/news/world-europe-41219773',
   'urlToImage': 'https://ichef.bbci.co.uk/news/1024/cpsprodpb/4D70/production/_97742891_mediaitem97742886.jpg'},
  {'author': 'BBC News',
   'description': "At least 400 children are believed to be buried in a section of St Mary's Cemetery in Lanark.",
   'publishedAt': '2017-09-10T04:52:47Z',
   'title': "Bodies of 'hundreds' of children buried in mass grave",
   'url': 'http://www.bbc.co.uk/news/uk-41200949',
   'urlToImage': 'https://ichef.bbci.co.uk/news/1024/cpsprodpb/6893/production/_97717762_smyllum-editedforiplayer.png'},
  {'author': 'BBC News',
   'description': "The celebrity baker said he went to a 2003 party in a WW2 German uniform as an 'Allo 'Allo character.",
   'publishedAt': '2017-09-10T11:06:02Z',
   'title': 'Bake Off: Paul Hollywood in Nazi outfit apology',
   'url': 'http://www.bbc.co.uk/news/uk-41218525',
   'urlToImage': 'https://ichef.bbci.co.uk/news/1024/cpsprodpb/DD07/production/_97738565_hollywood_alloallo_pa_bbc.jpg'},
  {'author': 'BBC News',
   'description': 'Jens Stoltenberg tells the BBC that Nato and its allies want a peaceful solution to the nuclear threat.',
   'publishedAt': '2017-09-10T14:38:08Z',
   'title': 'North Korea: Nato chief says global response needed',
   'url': 'http://www.bbc.co.uk/news/world-asia-41219767',
   'urlToImage': 'https://ichef.bbci.co.uk/news/1024/cpsprodpb/63A0/production/_97740552_hi041584494.jpg'}],
 'sortBy': 'top',
 'source': 'bbc-news',
 'status': 'ok'}
In [5]:
data = a.get(source="bbc-news", sort_by='top')
In [6]:
# convert from json to dataframe
data = pd.DataFrame.from_dict(data)
data = pd.concat([data.drop(['articles'], axis=1), data['articles'].apply(pd.Series)], axis=1)
data.head()
Out[6]:
sortBy source status author description publishedAt title url urlToImage
0 top bbc-news ok BBC News Amateur footage shows Key Largo, where the sea... 2017-09-10T15:33:39Z Hurricane Irma: Storm drains bay in Florida Keys http://www.bbc.co.uk/news/av/world-us-canada-4... https://ichef.bbci.co.uk/news/1024/cpsprodpb/8...
1 top bbc-news ok BBC News BBC correspondents report from the British Vir... 2017-09-10T12:48:09Z Hurricane Irma: Troops go to work on devastate... http://www.bbc.co.uk/news/uk-41218817 https://ichef.bbci.co.uk/news/1024/cpsprodpb/8...
2 top bbc-news ok BBC News The operating system of unreleased iPhones is ... 2017-09-10T17:26:30Z Apple suffers 'major iPhone X leak' http://www.bbc.co.uk/news/technology-41220517 https://ichef.bbci.co.uk/news/1024/cpsprodpb/F...
3 top bbc-news ok BBC News Footage emerges from Oaxaca state after 90 peo... 2017-09-10T11:19:08Z Mexico earthquake: Drone reveals aftermath http://www.bbc.co.uk/news/av/world-latin-ameri... https://ichef.bbci.co.uk/news/1024/cpsprodpb/E...
4 top bbc-news ok BBC News China has started "relevant research" but has ... 2017-09-10T10:25:24Z China looks at plans to ban petrol and diesel ... http://www.bbc.co.uk/news/business-41218243 https://ichef.bbci.co.uk/news/1024/cpsprodpb/2...

For each article request, the api returns the following:

  • status - If the request was successful or not. Options: ok, error. In the case of error a code and message property will be populated.
  • source - The identifier of the source requested.
  • sortBY - Which type of article list is being returned. Options: top, latest, popular.
  • articles - An array that contains a list of headline metadata requested. We take this array and transform it into columns for easier access.
    • author - The author of the article
    • description - A description or preface for the article.
    • title - The headline or title of the article.
    • url - The direct URL to the content page of the article.
    • urlToImage - The URL to a relevant image for the article.
    • publishedAt - The best attempt at finding a date for the article, in UTC (+0).1

Sources

Provides a list of the news sources and blogs available on News API. You will need this to programmatically locate the identifier for the source you want articles from when querying the /articles endpoint (i.e., when we use the Articles object).

In [7]:
from newsapi.sources import Sources
s = Sources(API_KEY=apikey)
In [8]:
# return all available sources
sources = s.get()
# convert to dataframe and drop status column
sources = data = pd.DataFrame.from_dict(sources).drop('status', axis=1)
# take the array column 'sources' and spread it across multiple columns
sources = pd.concat([sources.drop(['sources'], axis=1), 
                     sources['sources'].apply(pd.Series)], axis=1).drop('urlsToLogos', axis=1)
sources.tail()
Out[8]:
category country description id language name sortBysAvailable url
65 general us Breaking news and analysis on politics, busine... the-washington-post en The Washington Post [top] https://www.washingtonpost.com
66 general us Breaking news and analysis from TIME.com. Poli... time en Time [top, latest] http://time.com
67 general us Get the latest national, international, and po... usa-today en USA Today [top, latest] http://www.usatoday.com/news
68 technology de Wired reports on how emerging technologies aff... wired-de de Wired.de [top, latest] https://www.wired.de
69 business de Das Online-Portal des führenden Wirtschaftsmag... wirtschafts-woche de Wirtschafts Woche [latest] http://www.wiwo.de

Now we have a dataframe with all the available sources, the code needed to access them with the api, what category the source is classified as, the country the source is located in, the language it is written in, what sorting methods we can use on it, and the url to the source page!

The .get() method for Sources takes three arguments: category (optional), language (optional), and country (optional). The default option for all three options is empty.

A description of these arguments is as follows:

  • category - The category you would like to get sources for. Possible options include business, entertainment, gaming, general, music, politics, science-and-nature, sport, & technology.
  • language - The 2-letter ISO-639-1 code of the language you would like to get sources for. Possible options include en, de, fr.
  • country - The 2-letter ISO 3166-1 code of the country you would like to get sources for. Possible options include au, de, gb, in, it, us.
In [9]:
# once you call the .information() method, you can extract any source info you want like so:
s.information()
s.all_base_information()
s.all_categories()
s.all_countries()
s.all_ids()
s.all_languages()
s.all_names()
s.all_urls()
Out[9]:
dict_values(['http://www.abc.net.au/news', 'http://www.aljazeera.com', 'http://arstechnica.com', 'https://apnews.com/', 'http://www.bbc.co.uk/news', 'http://www.bbc.co.uk/sport', 'http://www.bild.de', 'http://www.bloomberg.com', 'http://www.breitbart.com', 'http://www.businessinsider.com', 'http://uk.businessinsider.com', 'https://www.buzzfeed.com', 'http://www.cnbc.com', 'http://us.cnn.com', 'http://www.dailymail.co.uk/home/index.html', 'http://www.tagesspiegel.de', 'http://www.zeit.de/index', 'https://www.engadget.com', 'http://www.ew.com', 'http://espn.go.com', 'http://www.espncricinfo.com/', 'http://www.ft.com/home/uk', 'http://www.focus.de', 'http://www.football-italia.net', 'http://fortune.com', 'http://www.fourfourtwo.com/news', 'http://www.foxsports.com', 'https://news.google.com', 'http://www.gruenderszene.de', 'https://news.ycombinator.com', 'http://www.handelsblatt.com', 'http://www.ign.com', 'http://www.independent.co.uk', 'http://mashable.com', 'http://metro.co.uk', 'http://www.mirror.co.uk/', 'http://www.mtv.com/news', 'http://www.mtv.co.uk/news', 'http://news.nationalgeographic.com', 'https://www.newscientist.com/section/news', 'http://www.newsweek.com', 'http://nymag.com', 'http://www.nfl.com/news', 'http://www.polygon.com', 'http://www.recode.net', 'https://www.reddit.com/r/all', 'http://www.reuters.com', 'http://www.spiegel.de', 'http://t3n.de', 'http://talksport.com', 'https://techcrunch.com', 'http://www.techradar.com', 'http://www.economist.com', 'https://www.theguardian.com/au', 'https://www.theguardian.com/uk', 'http://www.thehindu.com', 'http://www.huffingtonpost.com', 'http://www.theladbible.com', 'http://www.nytimes.com', 'http://thenextweb.com', 'http://www.thesportbible.com', 'http://www.telegraph.co.uk', 'http://timesofindia.indiatimes.com', 'http://www.theverge.com', 'http://www.wsj.com', 'https://www.washingtonpost.com', 'http://time.com', 'http://www.usatoday.com/news', 'https://www.wired.de', 'http://www.wiwo.de'])

Getting a general idea of what kind of sources we have

In [10]:
sns.countplot(y = 'category', data = sources, palette='Set1')
plt.ylabel('News Category')
plt.xlabel('Number of News Sources')
plt.title('Breakdown of News Source Categories')
plt.show()
In [11]:
ax = sns.countplot(x = 'country', data = sources, palette='Set1')
plt.ylabel('Number of News Sources')
plt.xlabel('Publication Country')
ax.set_xticklabels(['Australia', 'USA', 'UK', 'Germany', 'Italy', 'India'])
plt.title('Breakdown of News Source Publication Countries')
plt.show()
In [12]:
ax = sns.countplot(x = 'language', data = sources, palette='Set1')
plt.ylabel('Number of News Sources')
plt.xlabel('Publication Language')
ax.set_xticklabels(['English', 'German'])
plt.title('Breakdown of News Source Languages')
plt.show()
In [13]:
# returning subsets of info
general = sources[sources['category'] == 'general']
general.head()
Out[13]:
category country description id language name sortBysAvailable url
0 general au Australia's most trusted source of local, nati... abc-news-au en ABC News (AU) [top] http://www.abc.net.au/news
1 general us News, analysis from the Middle East and worldw... al-jazeera-english en Al Jazeera English [top, latest] http://www.aljazeera.com
3 general us The AP delivers in-depth coverage on the inter... associated-press en Associated Press [top] https://apnews.com/
4 general gb Use BBC News for up-to-the-minute news, breaki... bbc-news en BBC News [top] http://www.bbc.co.uk/news
6 general de Die Seite 1 für aktuelle Nachrichten und Theme... bild de Bild [top, latest] http://www.bild.de
In [14]:
results = []
broken = []
#print(results)
for i in range(len(sources)):
    try: 
        results.append(a.get(sources['id'][i]))
    except: 
        print('The pull request for row ', str(i), 'is not working')
        broken.append(i)
results = pd.DataFrame.from_dict(results)
results = pd.concat([results.drop(['articles'], axis=1), 
                  results['articles'].apply(pd.Series)], axis=1)
#data.head()
results

The pull request for row  15 is not working
The pull request for row  16 is not working
The pull request for row  30 is not working
The pull request for row  59 is not working
The pull request for row  69 is not working
Out[14]:
sortBy source status 0 1 2 3 4 5 6 7 8 9
0 top abc-news-au ok {'author': None, 'title': 'More than 1 million... {'author': None, 'title': 'Live: Hurricane Irm... {'author': None, 'title': 'Cuba surveys topple... {'author': None, 'title': 'Turnbull urges Aust... {'author': None, 'title': 'North Korea's Kim J... {'author': 'http://www.abc.net.au/news/sophie-... {'author': None, 'title': 'Rohingya rebels dec... {'author': 'http://www.abc.net.au/news/7115474... {'author': 'http://www.abc.net.au/news/lily-ma... {'author': 'http://www.abc.net.au/news/david-s...
1 top al-jazeera-english ok {'author': 'Al Jazeera', 'title': 'Hurricane I... {'author': 'Al Jazeera', 'title': 'Saudi embas... {'author': 'Al Jazeera', 'title': 'Critics cir... {'author': 'Katie Arnold', 'title': 'Message t... {'author': 'Hugo Diaz', 'title': 'Open letter ... {'author': 'Jonathan Cook', 'title': 'Criminal... {'author': 'Loes Witschge', 'title': 'Chile: F... {'author': 'Ahmed H Adam', 'title': 'Sudan's r... {'author': '', 'title': 'Palestinians demand r... {'author': 'Inside Story', 'title': 'Is dialog...
2 top ars-technica ok {'author': 'Annalee Newitz', 'title': 'Experts... {'author': None, 'title': 'Look back at PAX We... {'author': 'Beth Mole', 'title': 'The dye whis... {'author': 'Eric Berger', 'title': 'A rare, de... {'author': 'David Kravets', 'title': 'Startup ... {'author': 'Megan Geuss', 'title': 'As Irma ap... {'author': 'Eric Berger', 'title': 'Key questi... {'author': 'Timothy B. Lee', 'title': 'Lyft’s ... {'author': 'Peter Bright', 'title': 'I’ve fall... {'author': 'Eric Berger', 'title': 'US forecas...
3 top associated-press ok {'author': '', 'title': 'The Latest: Utility s... {'author': 'JENNIFER KAY and FREIDA FRISARO', ... {'author': 'DESMOND BOYLAN and ANDREA RODRIGUE... {'author': 'CHRISTOPHER SHERMAN', 'title': 'De... {'author': 'NICOLE WINFIELD', 'title': 'Pope a... {'author': 'GARY FINEOUT', 'title': 'Irma reco... {'author': 'FRANCES D'EMILIO', 'title': 'Heavy... {'author': '', 'title': 'Myanmar accused of la... {'author': 'BRIAN MELLEY and SALLY HO', 'title... {'author': 'RALPH D. RUSSO', 'title': 'Sooner ...
4 top bbc-news ok {'author': 'BBC News', 'title': 'Hurricane Irm... {'author': 'BBC News', 'title': 'Hurricane Irm... {'author': 'BBC News', 'title': 'Apple suffers... {'author': 'BBC News', 'title': 'Mexico earthq... {'author': 'BBC News', 'title': 'China looks a... {'author': 'BBC News', 'title': 'Rohingya cris... {'author': 'BBC News', 'title': 'Former govern... {'author': 'BBC News', 'title': 'Bodies of 'hu... {'author': 'BBC News', 'title': 'Bake Off: Pau... {'author': 'BBC News', 'title': 'North Korea: ...
5 top bbc-sport ok {'author': 'BBC Sport', 'title': 'Froome compl... {'author': 'BBC Sport', 'title': 'Burnley 1-0 ... {'author': 'BBC Sport', 'title': 'Great North ... {'author': 'BBC Sport', 'title': 'Chris Froome... {'author': 'BBC Sport', 'title': 'European Mas... {'author': 'BBC Sport', 'title': 'Vote: August... {'author': 'BBC Sport', 'title': 'Britain's Be... {'author': 'BBC Sport', 'title': 'Great North ... {'author': 'BBC Sport', 'title': 'Swansea City... {'author': 'BBC Sport', 'title': 'James Anders...
6 top bild ok {'author': None, 'title': 'Bisswunde an der Hü... {'author': None, 'title': 'Täter schnell gefas... {'author': None, 'title': 'Damit ihr Baby lebe... {'author': None, 'title': 'BILD findet pervers... {'author': None, 'title': 'Alarm am Frankfurte... NaN NaN NaN NaN NaN
7 top bloomberg ok {'author': 'Brian K Sullivan', 'title': 'Hurri... {'author': 'Jennifer Epstein', 'title': 'Clint... {'author': 'Sahil Kapur', 'title': 'House Cons... {'author': 'Alex Morales', 'title': 'Russian W... {'author': 'Anousha Sakoui', 'title': 'Creepy ... {'author': 'David Tweed', 'title': 'China and ... {'author': 'Mark Gurman', 'title': 'Apple's To... {'author': 'John Tozzi', 'title': 'Stroke Is M... {'author': 'Jared S Hopkins', 'title': 'Lilly ... {'author': 'Matthew Bristow', 'title': 'Colomb...
8 top breitbart-news ok {'author': 'Alexander Marlow', 'title': 'Samue... {'author': 'Jeff Poor', 'title': 'Hurricane Ir... {'author': 'Charlie Spiering', 'title': 'Donal... {'author': 'Deborah Danan', 'title': 'Chabad E... {'author': 'Katherine Rodriguez', 'title': 'Al... {'author': 'Katherine Rodriguez', 'title': 'WA... {'author': 'Chriss W. Street', 'title': 'UC Su... {'author': 'Pam Key', 'title': 'GOP Rep Jordan... {'author': 'Dr. Susan Berry', 'title': 'Texas ... {'author': 'Warner Todd Huston', 'title': 'Flo...
9 top business-insider ok {'author': 'Kevin Loria and Dave Mosher', 'tit... {'author': 'Maxwell Tani', 'title': 'Hillary C... {'author': 'Maxwell Tani', 'title': ''I hope t... {'author': 'Rebecca Harrington', 'title': 'Her... {'author': 'Steve Kovach', 'title': 'Apple has... {'author': 'Lydia Ramsey', 'title': 'A biotech... {'author': 'Maxwell Tani', 'title': 'Recently ... {'author': 'Dennis Green', 'title': 'Amazon ha... {'author': 'Sonam Sheth', 'title': 'The US' bi... {'author': 'Ben Gilbert', 'title': 'In the new...
10 top business-insider-uk ok {'author': 'Kevin Loria and Dave Mosher', 'tit... {'author': 'Jim Edwards', 'title': 'The UK loo... {'author': 'Shona Ghosh', 'title': 'Uber rival... {'author': 'Shona Ghosh', 'title': 'How Florid... {'author': 'Alison Millington', 'title': 'What... {'author': 'Jim Edwards', 'title': 'iPhone 8 w... {'author': 'Sam Shead', 'title': 'The VC arm o... {'author': 'David Ibekwe', 'title': 'Watch as ... {'author': 'Dina Spector', 'title': 'Business ... {'author': 'Dave Mosher', 'title': 'Hurricane ...
11 top buzzfeed ok {'author': 'Sarah Aspler', 'title': '13 Cool A... {'author': 'Sheridan Watson', 'title': 'Leslie... {'author': 'Ryan Schocket', 'title': '25 Times... {'author': 'Lauren Strapagiel', 'title': 'This... {'author': 'Matt Stopera', 'title': '23 Pictur... {'author': 'Jasmin Freeman', 'title': 'Pick Se... {'author': 'Ryan Schocket', 'title': '21 Tweet... {'author': 'Sarah Aspler', 'title': '17 Border... {'author': 'Jake Kaplan', 'title': 'Do You Hat... {'author': 'Jasmin Nahar', 'title': '19 Tumblr...
12 top cnbc ok {'author': 'Thomas Franck', 'title': 'Hurrican... {'author': 'Robert Ferris, Ted Kemp', 'title':... {'author': 'CNBC', 'title': 'Florida utility: ... {'author': 'Leslie Josephs', 'title': 'Hurrica... {'author': 'Javier E. David', 'title': 'Hurric... {'author': 'Tae Kim', 'title': 'Tampa mayor: '... {'author': 'Christina Farr', 'title': 'Hurrica... {'author': 'Tae Kim', 'title': 'Florida sherif... {'author': 'Marty Steinberg', 'title': 'Georgi... {'author': 'Ester Bloom', 'title': 'Kellyanne ...
13 top cnn ok {'author': 'Holly Yan, Faith Karimi and Susann... {'author': 'Meg Wagner and Amanda Wills, CNN',... {'author': 'Dakin Andone, CNN', 'title': 'This... {'author': None, 'title': 'Tornado hits near F... {'author': None, 'title': 'LIVE: Jacksonville ... {'author': None, 'title': 'LIVE: Ft. Lauderdal... {'author': 'Dakin Andone, CNN', 'title': 'Tamp... {'author': None, 'title': 'Tracking the path o... {'author': 'Sophie Tatum, CNN', 'title': 'Clin... {'author': 'Eli Watkins, CNN', 'title': 'John ...
14 top daily-mail ok {'author': 'By Sarah Oliver For The Mail On Su... {'author': 'By Joe Sheppard For Mailonline', '... {'author': 'By Katie French For Mailonline', '... {'author': 'By Emily Crane, Jennifer Smith, Re... {'author': 'By Harvey Day For Mailonline', 'ti... {'author': 'By Katie French For Mailonline', '... {'author': 'By Scott Campbell and Joe Sheppard... {'author': 'By Simon Walters and Glen Owen for... {'author': 'By Katie French For Mailonline', '... {'author': 'By Alistair Jackson For The Mail O...
15 top engadget ok {'author': 'Roberto Baldwin', 'title': 'A 114-... {'author': 'Nathan Ingraham', 'title': 'What t... {'author': 'Devindra Hardawar', 'title': 'Dear... {'author': 'Daniel Cooper', 'title': 'What’s t... {'author': 'Rob LeFebvre', 'title': '‘Futurama... NaN NaN NaN NaN NaN
16 top entertainment-weekly ok {'author': 'Lynette Rice', 'title': '‘Outlande... {'author': 'Oliver Gettell', 'title': '‘Great ... {'author': 'Christopher Rosen', 'title': 'Lord... {'author': 'Oliver Gettell', 'title': '‘It’ sm... {'author': 'Christopher Rosen', 'title': 'Hele... {'author': 'Oliver Gettell', 'title': 'Louis C... {'author': 'Christopher Rosen', 'title': 'Hill... {'author': 'Nick Romano', 'title': 'Tig Notaro... {'author': 'Christopher Rosen', 'title': 'Hill... {'author': None, 'title': 'Summer 2017', 'desc...
17 top espn ok {'author': None, 'title': 'Eagles vs. Redskins... {'author': None, 'title': 'Biggest NFL injurie... {'author': None, 'title': 'W2W4 at US Open: Ma... {'author': 'Adam RittenbergESPN Staff Writer '... {'author': None, 'title': 'The game-changing p... {'author': 'Heather DinichESPN Senior Writer '... {'author': 'Edward AschoffESPN Staff Writer ',... {'author': None, 'title': 'Week 1 fantasy foot... {'author': None, 'title': 'Play the No. 1 Fant... {'author': 'Buster OlneyESPN Senior Writer ', ...
18 top espn-cric-info ok {'author': None, 'title': 'ESPNcricinfo - Cric... NaN NaN NaN NaN NaN NaN NaN NaN NaN
19 top financial-times ok {'author': None, 'title': 'Irma unleashes its ... {'author': None, 'title': 'Tears and terror as... {'author': None, 'title': 'Hurricane Irma set ... {'author': None, 'title': 'Investors watch for... {'author': None, 'title': 'Investors plan safe... {'author': None, 'title': 'Apple tests custome... {'author': None, 'title': 'Goldman Sachs to ta... {'author': None, 'title': 'France drives EU ta... {'author': None, 'title': 'Merkel primed for v... {'author': None, 'title': 'The global upswing ...
20 top focus ok {'author': 'Von FOCUS-Online-Autorin Julia Hub... {'author': 'FOCUS Online', 'title': 'Streit mi... {'author': 'FOCUS Online', 'title': '+++ Hurri... {'author': 'FOCUS Online', 'title': 'Es ist no... {'author': 'Von FOCUS-Online-Gastautor Urs Mei... {'author': 'FOCUS Online', 'title': 'Streit mi... {'author': 'FOCUS Online', 'title': 'Felix Bas... {'author': 'Lisa Mayerhofer', 'title': 'Er ver... {'author': 'FOCUS Online', 'title': 'Schon wie... {'author': 'Von FOCUS-Online-Autor Sebastian L...
21 top football-italia ok {'author': 'Football Italia Staff', 'title': '... {'author': 'Football Italia Staff', 'title': '... {'author': 'Football Italia Staff', 'title': '... {'author': 'Football Italia Staff', 'title': '... {'author': 'Football Italia Staff', 'title': '... {'author': 'Football Italia Staff', 'title': '... {'author': 'Football Italia Staff', 'title': '... {'author': 'Football Italia Staff', 'title': '... {'author': 'Football Italia Staff', 'title': '... {'author': 'Football Italia Staff', 'title': '...
22 top fortune ok {'author': 'Sy Mukherjee', 'title': 'This Is t... {'author': 'David Z. Morris', 'title': 'iOS Le... {'author': 'David Z. Morris', 'title': 'How Te... {'author': 'Reuters', 'title': 'Utility Crews ... {'author': None, 'title': 'Drones May Be Used ... {'author': 'David Z. Morris', 'title': 'Apple ... {'author': 'Laura Entis', 'title': 'The Top 10... {'author': 'Stacy Perman', 'title': 'How to Ri... {'author': 'Ellen McGirt', 'title': 'Include U... NaN
23 top four-four-two ok {'author': 'FourFourTwo', 'title': 'Montella u... {'author': 'FourFourTwo', 'title': 'Milan fail... {'author': 'FourFourTwo', 'title': 'Swansea Ci... {'author': 'FourFourTwo', 'title': 'Joao Mario... {'author': 'FourFourTwo', 'title': 'Lazio 4 AC... {'author': 'FourFourTwo', 'title': 'De Boer on... {'author': 'FourFourTwo', 'title': 'Heaton suf... {'author': 'FourFourTwo', 'title': 'Dortmund c... {'author': 'FourFourTwo', 'title': 'Palace wil... {'author': 'FourFourTwo', 'title': 'Seri speak...
24 top fox-sports ok {'author': None, 'title': 'Jay Glazer has the ... {'author': None, 'title': 'Houston hero J.J. W... {'author': None, 'title': 'Rob Riggle's 'I Wan... {'author': None, 'title': 'Baker Mayfield plan... {'author': None, 'title': 'Jimmy Johnson's mes... {'author': None, 'title': 'Watch Josh Norman t... {'author': None, 'title': 'Peter Schrager's su... {'author': None, 'title': 'Mike Vick: Matt Rya... {'author': None, 'title': 'Tony Gonzalez expec... {'author': None, 'title': 'Watch Louisiana Tec...
25 top google-news ok {'author': 'https://www.facebook.com/markberma... {'author': 'IAN KULLGREN', 'title': 'McCain: '... {'author': None, 'title': 'Hillary Clinton say... {'author': None, 'title': 'Two Florida officer... {'author': 'Mehr News Agency', 'title': 'Iran ... {'author': 'Sean Rossman', 'title': 'Dozens of... {'author': 'Carla Herreria', 'title': 'Hurrica... {'author': None, 'title': 'Pope arrival in Car... {'author': None, 'title': 'Sensing disaster, T... {'author': 'Sheryl Gay Stolberg', 'title': 'Wh...
26 top gruenderszene ok {'author': 'Frank Schmiechen', 'title': 'Noch ... {'author': 'Dr. Sabina Krispenz', 'title': 'Wo... {'author': 'Sponsored Post', 'title': 'Diese S... {'author': 'Kathrin Spoerr', 'title': 'Die ung... {'author': 'Sponsored Post', 'title': 'Warum s... {'author': 'Michel Penke', 'title': 'Kai Diekm... {'author': 'Anne-Katrin Schade', 'title': 'Lit... {'author': 'Jana Kugoth', 'title': 'Mit dieser... {'author': 'Marie Gracher', 'title': 'Jobs für... {'author': 'Timo Brücken', 'title': 'Die About...
27 top hacker-news ok {'author': 'Leo Kelion', 'title': 'Apple suffe... {'author': None, 'title': 'Massive genetic stu... {'author': 'Alex Kotch', 'title': 'Equifax Lob... {'author': None, 'title': 'Remembering Lotfi Z... {'author': None, 'title': 'Why Sub-prime Lende... {'author': None, 'title': 'How to conduct a go... {'author': None, 'title': 'DuckDuckHack', 'des... {'author': 'Badennova', 'title': 'Smarter spee... {'author': None, 'title': 'Show HN: Hurricane ... {'author': 'Science X staff', 'title': 'Human ...
28 top ign ok {'author': 'Jesse Schedeen', 'title': '8 Direc... {'author': 'Alex Osborn', 'title': 'My Hero Ac... {'author': 'Jim Vejvoda', 'title': 'Stephen Ki... {'author': 'Terri Schwartz', 'title': 'Why the... {'author': None, 'title': 'Disney Plans to Pul... {'author': None, 'title': 'Public Events - Des... {'author': 'Arnold T. Blumberg', 'title': 'Pen... {'author': 'Eric Song', 'title': 'Daily Deals:... {'author': 'William Bibbiani', 'title': '9 Mov... {'author': 'josh norem', 'title': 'The Best Dr...
29 top independent ok {'author': 'Samuel Osborne, Andrew Buncombe, C... {'author': 'Andrew Buncombe', 'title': 'Three ... {'author': 'Rachael Revesz', 'title': 'Florida... {'author': 'Rachael Revesz', 'title': 'Hurrica... {'author': 'Sally Hayden', 'title': 'Astronaut... {'author': 'Jack Shepherd', 'title': 'Paul Hol... {'author': 'Rachael Revesz', 'title': 'Britain... {'author': 'Harriet Agerholm', 'title': 'Bodie... {'author': 'May Bulman', 'title': 'Victoria tu... {'author': 'Rob Merrick', 'title': 'Tony Blair...
... ... ... ... ... ... ... ... ... ... ... ... ... ...
35 top national-geographic ok {'author': 'Christina Nunez', 'title': 'Get a ... {'author': 'Liz Langley', 'title': 'Go, Baby! ... {'author': 'Sarah Gibbens', 'title': '42 of Wo... {'author': 'Simon Worrall', 'title': 'Dogs Hav... {'author': 'Sarah Gibbens', 'title': 'See Resc... {'author': 'Sarah Gibbens', 'title': 'Can This... {'author': 'National Geographic Staff', 'title... {'author': None, 'title': 'Equinoxes', 'descri... {'author': None, 'title': 'Epic Trails Across ... {'author': 'Francis Rivera', 'title': 'Best of...
36 top new-scientist ok {'author': '', 'title': 'Meet the vampire ant ... {'author': '', 'title': 'Ankle fossil suggests... {'author': '', 'title': 'Low-oxygen dwarf gala... {'author': '', 'title': 'Gel-like ice is the l... {'author': '', 'title': 'Hurricane Irma tears ... {'author': '', 'title': 'Equifax data breach h... {'author': '', 'title': 'We seem to be getting... {'author': '', 'title': 'Mexico on tsunami ale... {'author': '', 'title': 'Box jellyfish will de... {'author': '', 'title': 'Bats crash into windo...
37 top newsweek ok {'author': 'Tom Porter', 'title': 'Hurricane I... {'author': 'Tom Porter', 'title': 'Don't fire ... {'author': None, 'title': 'The Best of EQUIS A... {'author': 'Jessica Firger', 'title': 'Sexual ... {'author': 'Tim Marcin', 'title': 'Hurricane I... {'author': None, 'title': 'The Best of EQUIS A... {'author': 'Graham Lanktree', 'title': 'How ca... {'author': 'Janissa Delzo', 'title': 'Nudist r... {'author': 'Walter Olson', 'title': 'The Gerry... {'author': 'https://www.facebook.com/NinaBurle...
38 top new-york-magazine ok {'author': 'Chas Danner', 'title': 'Irma Makes... {'author': 'Jonathan Chait@jonathanchait', 'ti... {'author': 'Jen Chaney@chaneyj', 'title': 'Out... {'author': 'Max Read@max_read', 'title': 'The ... {'author': 'Emily Yoshida', 'title': 'The Scar... {'author': 'Andrew Sullivan', 'title': 'Andrew... {'author': 'Jesse Singal', 'title': 'Is the U.... {'author': 'Sasha Geffen', 'title': 'Introduci... {'author': 'Nabile Quenum', 'title': 'See the ... {'author': 'E. Alex Jung@e_alexjung', 'title':...
39 top nfl-news ok {'author': 'Conor Orr', 'title': 'Eagles CB Ro... {'author': 'Around The NFL staff', 'title': 'I... {'author': 'Nick Shook', 'title': 'J.J. Watt, ... {'author': 'Around The NFL staff', 'title': 'I... {'author': 'Kevin Patra', 'title': 'Colts QB A... {'author': 'Kevin Patra', 'title': 'Steve Smit... {'author': 'Kevin Patra', 'title': 'Marcell Da... {'author': 'Kevin Patra', 'title': 'Steve Smit... {'author': 'Lakisha Jackson', 'title': 'Raider... {'author': 'Jeremy Bergman', 'title': 'Bears s...
40 top polygon ok {'author': 'Owen S. Good', 'title': 'Gold farm... {'author': 'Owen S. Good', 'title': 'Destiny 2... {'author': 'Samit Sarkar', 'title': 'Destiny 2... {'author': 'Chloe Reznikov', 'title': 'Destiny... {'author': 'Allegra Frank', 'title': 'The Poké... {'author': 'Chelsea Stark', 'title': 'Come be ... {'author': 'Allegra Frank', 'title': 'Your big... {'author': 'Allegra Frank', 'title': 'Pokémon ... {'author': 'Michael McWhertor', 'title': 'Bung... {'author': 'Charlie Hall', 'title': 'A new gam...
41 top recode ok {'author': 'Johana Bhuiyan', 'title': 'Twitter... {'author': 'Johana Bhuiyan', 'title': 'Florida... {'author': 'Rani Molla', 'title': 'Amazon’s ne... {'author': 'Rani Molla', 'title': 'Florida sto... {'author': 'Rani Molla', 'title': 'Satellite i... {'author': 'Tom Warren', 'title': 'Leaked iPho... {'author': 'Rani Molla', 'title': 'Millennials... {'author': 'Recode Staff', 'title': 'Full tran... {'author': 'Kurt Wagner', 'title': 'Snapchat w... {'author': 'Dan Frommer', 'title': 'Watch Mary...
42 top reddit-r-all ok {'author': 'GFYCAT', 'title': 'Untitled 3 - Cr... {'author': 'Imgur', 'title': 'Firefighters tak... {'author': 'Imgur', 'title': 'Namaste', 'descr... {'author': 'Imgur', 'title': 'Right... so, tha... {'author': None, 'title': 'Streamable - free v... {'author': 'LetsDIY', 'title': 'The best photo... {'author': None, 'title': 'Streamable - free v... {'author': 'Imgur', 'title': 'MRW My mom asks ... {'author': 'Imgur', 'title': 'Leaked scene fro... {'author': 'Imgur', 'title': 'Goodyear Illumin...
43 top reuters ok {'author': 'Robin Respaut and Andy Sullivan', ... {'author': 'Robin Respaut and Brian Thevenot',... {'author': 'Reuters Editorial', 'title': 'Flor... {'author': 'Nelson Acosta', 'title': 'Irma's 3... {'author': 'Doina Chiacu and Roberta Rampton',... {'author': 'Reuters Editorial', 'title': 'U.S.... {'author': 'Raya Jalabi and Ulf Laessing', 'ti... {'author': 'Reuters Editorial', 'title': 'U.S.... {'author': 'John Davison', 'title': 'Syria arm... {'author': 'Reuters Editorial', 'title': 'Nort...
44 top spiegel-online ok {'author': 'Spiegel Online', 'title': 'Wahlkam... {'author': 'Spiegel Online', 'title': 'Struktu... {'author': 'Spiegel Online', 'title': 'Wirbels... {'author': 'Spiegel Online', 'title': 'Flüchtl... {'author': 'Spiegel Online', 'title': 'Wie ans... {'author': 'Spiegel Online', 'title': '"Hells ... {'author': 'Spiegel Online', 'title': 'Türkisc... {'author': 'Spiegel Online', 'title': 'Bundesl... {'author': 'Spiegel Online', 'title': 'Opfer v... {'author': 'Spiegel Online', 'title': 'Dieses ...
45 top t3n ok {'author': 'Jessy Kösterke', 'title': '25 CSS-... {'author': 'Andreas Floemer', 'title': 'iPhone... {'author': 'Jochen G. Fuchs', 'title': '5 Grün... {'author': 'Daniel Hüfner', 'title': '6 großar... {'author': 'Cornelia Dlugos', 'title': 'Neues ... {'author': 'Jessy Kösterke', 'title': 'Kein De... {'author': 'Helge Husemann', 'title': 'Fake Ne... {'author': 'Jörn Brien', 'title': 'Risiko: War... {'author': 'Jörn Brien', 'title': 'China will ... {'author': 'Melanie Dahrendorf', 'title': 'Bit...
46 top talksport ok {'author': 'talkSPORT', 'title': 'Swansea 0-1 ... {'author': 'talkSPORT', 'title': 'Manchester U... {'author': 'Billy Hawkins', 'title': 'Burnley ... {'author': 'talkSPORT', 'title': 'Carlo Ancelo... {'author': 'Billy Hawkins', 'title': 'Manchest... {'author': 'talkSPORT', 'title': 'Top 10 highe... {'author': 'talkSPORT', 'title': 'Manchester C... {'author': 'talkSPORT', 'title': '10 British p... NaN NaN
47 top techcrunch ok {'author': 'Brian Heater', 'title': 'Samsung’s... {'author': 'Brian Heater', 'title': 'What to e... {'author': 'Josh Constine', 'title': 'We’re al... {'author': 'Mike Butcher', 'title': 'Zello top... {'author': 'Greg Kumparak', 'title': 'Tesla fl... NaN NaN NaN NaN NaN
48 top techradar ok {'author': 'James Peckham', 'title': '9 things... {'author': 'Mark Knapp', 'title': 'Switch carr... {'author': 'Emma Boyle', 'title': 'Apple 4K TV... {'author': 'Joe Osborne', 'title': 'The Best L... {'author': 'Matt Swider', 'title': 'Samsung Ga... {'author': 'Matt Hanson', 'title': 'What happe... NaN NaN NaN NaN
49 top the-economist ok {'author': 'The Economist', 'title': 'Argument... {'author': 'The Economist', 'title': 'Emmanuel... {'author': 'The Economist', 'title': 'Advances... {'author': 'The Economist', 'title': 'To deter... {'author': 'The Economist', 'title': 'As storm... {'author': 'The Economist', 'title': 'A craft-... {'author': 'The Economist', 'title': 'Is the A... {'author': 'The Economist', 'title': 'The Depa... {'author': 'The Economist', 'title': '“Call Me... {'author': 'The Economist', 'title': 'A new pl...
50 top the-guardian-au ok {'author': 'Alan Yuhas, Martin Pengelly, Jessi... {'author': 'Ben Doherty', 'title': 'Australia ... {'author': 'Katharine Murphy', 'title': 'Energ... {'author': 'Katharine Murphy, Christopher Knau... {'author': 'Katharine Murphy', 'title': 'Fryde... {'author': 'Eleanor Ainge Roy', 'title': 'New ... {'author': 'Sabrina Siddiqui', 'title': 'Hilla... {'author': 'Julian Borger', 'title': 'Merkel o... {'author': 'Heather Stewart', 'title': 'Tony B... {'author': 'Benjamin Haas', 'title': 'Whiff of...
51 top the-guardian-uk ok {'author': 'Alan Yuhas, Martin Pengelly, Jessi... {'author': 'Sarah Marsh, Sally Weale, Richard ... {'author': 'Harriet Sherwood', 'title': 'Grenf... {'author': 'Robert Booth', 'title': 'EU flags ... {'author': 'Heather Stewart', 'title': 'Barack... {'author': 'Rajeev Syal', 'title': 'McDonnell ... {'author': 'Jon Henley', 'title': 'Norway goes... {'author': 'Julian Borger', 'title': 'Merkel o... {'author': 'Zoe Wood', 'title': 'Noel Edmonds ... {'author': 'Robert Booth', 'title': 'Paul Holl...
52 top the-hindu ok {'author': 'Haroon Habib', 'title': 'In Bangla... {'author': 'Peerzada Ashiq', 'title': 'Will tu... {'author': 'Dennis S. Jesudasan', 'title': 'DM... {'author': 'Prakash Kamat', 'title': 'Indian N... {'author': 'Staff Reporter', 'title': 'One kil... {'author': 'PTI', 'title': 'Patel quota stir a... {'author': '​Staff Reporter', 'title': 'Delhi ... {'author': 'PTI', 'title': 'Gurugram school mu... {'author': 'Ruchir Joshi', 'title': 'The curse... {'author': 'Keerthik Sasidharan', 'title': 'Wh...
53 top the-huffington-post ok {'author': 'Paige Lavender', 'title': 'Read Li... {'author': 'Reuters', 'title': 'John McCain Sa... {'author': 'Marina Fang', 'title': 'Hillary Cl... {'author': 'Reuters', 'title': 'Death Toll Fro... {'author': 'Nina Golgowski', 'title': 'Dramati... {'author': 'Reuters', 'title': 'Trump Speaks T... {'author': 'Reuters', 'title': 'Over 500 Dead ... {'author': 'Marina Fang', 'title': 'Hillary Cl... {'author': 'Nina Golgowski', 'title': 'Gator P... {'author': 'HuffPost Video', 'title': 'Spelman...
54 top the-lad-bible ok {'author': 'Stewart Perrie', 'title': 'Natalie... {'author': 'Michael Minay', 'title': 'BBC News... {'author': 'Jack Kenmare', 'title': 'Goran Pan... {'author': 'Michael Minay', 'title': 'YouTuber... {'author': 'Stewart Perrie', 'title': 'LAD Ren... {'author': 'Claire Reid', 'title': 'Vodka And ... {'author': 'Stewart Perrie', 'title': 'People ... {'author': 'Mark McGowan', 'title': 'A Group O... {'author': 'Will Fitzpatrick', 'title': 'The '... {'author': 'Michael Minay', 'title': 'Nestle I...
55 top the-new-york-times ok {'author': 'The New York Times', 'title': 'Sto... {'author': 'Sarah Almukhtar, Jeremy Ashkenas, ... {'author': 'The New York Times', 'title': '‘Le... {'author': 'The New York Times', 'title': 'Hur... {'author': 'Emily Badger, Quoctrung Bui and Cl... {'author': 'Peter Baker', 'title': 'Bound to N... {'author': 'Ron Lieber', 'title': 'Equifax’s I... {'author': 'Rachel Sherman', 'title': 'Opinion... {'author': 'Marc Santora, Henry Fountain and V... {'author': 'Jonah Engel Bromwich', 'title': 'H...
56 top the-sport-bible ok {'author': 'Jack Kenmare', 'title': 'Mancheste... {'author': 'Jack Kenmare', 'title': 'A Selecti... {'author': 'Jack Kenmare', 'title': 'The First... {'author': 'Nasir Jabbar', 'title': 'WATCH: Th... {'author': 'Jack Kenmare', 'title': 'The Shock... {'author': 'Nasir Jabbar', 'title': 'A Manches... {'author': 'Jack Kenmare', 'title': 'Goran Pan... {'author': 'Nasir Jabbar', 'title': 'Sadio Man... {'author': 'Jack Kenmare', 'title': 'WATCH: Th... {'author': 'SPORTbible', 'title': 'Chelsea Rel...
57 top the-telegraph ok {'author': None, 'title': 'Couple trapped in r... {'author': None, 'title': 'Hurricane Irma: Flo... {'author': None, 'title': 'Serbia has chosen t... {'author': None, 'title': 'Goodwood Revival 20... {'author': None, 'title': 'A sharp Last Night ... {'author': None, 'title': 'Gotta sing, gotta d... {'author': None, 'title': 'Worshippers stabbed... {'author': None, 'title': 'Marco Pierre-White ... {'author': None, 'title': 'Paul Hollywood 'dev... {'author': None, 'title': 'Promoting Joe Root ...
58 top the-times-of-india ok {'author': None, 'title': 'CAG report flagging... {'author': None, 'title': ''Secret tunnels, ir... {'author': None, 'title': 'PM Modi, Shinzo Abe... {'author': None, 'title': 'Return was the most... {'author': None, 'title': 'Navy's women team s... {'author': 'Binay Singh', 'title': 'Army chief... {'author': None, 'title': 'Gurugram school mur... NaN NaN NaN
59 top the-verge ok {'author': 'Chaim Gartenberg', 'title': 'The t... {'author': 'Angela Chen', 'title': 'Franklin F... {'author': 'Tasha Robinson', 'title': 'I Kill ... {'author': 'Loren Grush', 'title': 'The Sun ha... {'author': 'Tom Warren', 'title': 'Apple iPhon... {'author': 'Loren Grush', 'title': 'How NASA i... {'author': 'Russell Brandom', 'title': 'Our en... {'author': 'Jacob Kastrenakes', 'title': 'New ... {'author': 'Andrew Liptak', 'title': 'Virginia... {'author': 'James Vincent', 'title': 'AI learn...
60 top the-wall-street-journal ok {'author': 'Arian Campo-Flores', 'title': 'Hur... {'author': 'WSJ Graphics', 'title': 'Tracking ... {'author': 'WSJ.com News Graphics', 'title': '... {'author': 'Robbie Whelan', 'title': 'Looting ... {'author': 'Susan Carey', 'title': 'Airlines M... {'author': 'Leslie Scism', 'title': 'What Home... {'author': 'Cameron McWhirter', 'title': 'Mich... {'author': 'Wall Street Journal', 'title': 'Hu... {'author': 'Erin Ailworth', 'title': 'Irma Kno... {'author': 'Dudley Althaus', 'title': 'Irma Le...
61 top the-washington-post ok {'author': 'Perry Stein, Mark Berman, Wesley L... {'author': 'Jason Samenow, Greg Porter, Jason ... {'author': None, 'title': 'Everything you need... {'author': 'WashPostPR, WashPostPR', 'title': ... {'author': 'Cleve R. Wootson Jr., Cleve R. Woo... {'author': None, 'title': 'Tracking Hurricane ... {'author': None, 'title': 'Video shows dangero... {'author': None, 'title': 'Irma’s wrath: The m... {'author': 'Angela Fritz, Angela Fritz', 'titl... {'author': None, 'title': 'Escaping North Kore...
62 top time ok {'author': 'Haley Sweetland Edwards', 'title':... {'author': 'Alana Abramson', 'title': ''I Am D... {'author': 'Andrew Taylor / AP', 'title': 'Tru... {'author': 'Stephanie Zacharek', 'title': 'Rev... {'author': 'Christopher Sherman / AP', 'title'... {'author': 'Michael Weissenstein / AP', 'title... {'author': 'Nariman El-Mofty / AP', 'title': '... {'author': 'Ken Sweet / AP', 'title': 'Equifax... {'author': 'Matthew Daly / AP', 'title': 'Repu... {'author': 'Julie Watson / AP', 'title': ''Lar...
63 top usa-today ok {'author': None, 'title': 'Hurricane Irma slam... {'author': None, 'title': 'Mammoth Hurricane I... {'author': None, 'title': 'Hurricane Irma: Wh... {'author': None, 'title': 'Amway Coaches Poll:... {'author': None, 'title': 'Dozens of dogs aban... {'author': None, 'title': 'Florida sheriff's o... {'author': None, 'title': 'High-rise crane col... {'author': None, 'title': 'Evacuees got out of... {'author': None, 'title': 'Stephen King's cree... {'author': None, 'title': 'Insider: Why this N...
64 top wired-de ok {'author': 'Juliane Görsch', 'title': 'Cyborg-... {'author': 'WIRED Staff', 'title': 'Modder bri... {'author': 'Cindy Michel', 'title': 'So ließe ... {'author': 'David Pierce', 'title': 'Wie ein S... {'author': 'Steve Haak', 'title': 'Deshalb wil... NaN NaN NaN NaN NaN

65 rows × 13 columns

In [15]:
results.head()
Out[15]:
sortBy source status 0 1 2 3 4 5 6 7 8 9
0 top abc-news-au ok {'author': None, 'title': 'More than 1 million... {'author': None, 'title': 'Live: Hurricane Irm... {'author': None, 'title': 'Cuba surveys topple... {'author': None, 'title': 'Turnbull urges Aust... {'author': None, 'title': 'North Korea's Kim J... {'author': 'http://www.abc.net.au/news/sophie-... {'author': None, 'title': 'Rohingya rebels dec... {'author': 'http://www.abc.net.au/news/7115474... {'author': 'http://www.abc.net.au/news/lily-ma... {'author': 'http://www.abc.net.au/news/david-s...
1 top al-jazeera-english ok {'author': 'Al Jazeera', 'title': 'Hurricane I... {'author': 'Al Jazeera', 'title': 'Saudi embas... {'author': 'Al Jazeera', 'title': 'Critics cir... {'author': 'Katie Arnold', 'title': 'Message t... {'author': 'Hugo Diaz', 'title': 'Open letter ... {'author': 'Jonathan Cook', 'title': 'Criminal... {'author': 'Loes Witschge', 'title': 'Chile: F... {'author': 'Ahmed H Adam', 'title': 'Sudan's r... {'author': '', 'title': 'Palestinians demand r... {'author': 'Inside Story', 'title': 'Is dialog...
2 top ars-technica ok {'author': 'Annalee Newitz', 'title': 'Experts... {'author': None, 'title': 'Look back at PAX We... {'author': 'Beth Mole', 'title': 'The dye whis... {'author': 'Eric Berger', 'title': 'A rare, de... {'author': 'David Kravets', 'title': 'Startup ... {'author': 'Megan Geuss', 'title': 'As Irma ap... {'author': 'Eric Berger', 'title': 'Key questi... {'author': 'Timothy B. Lee', 'title': 'Lyft’s ... {'author': 'Peter Bright', 'title': 'I’ve fall... {'author': 'Eric Berger', 'title': 'US forecas...
3 top associated-press ok {'author': '', 'title': 'The Latest: Utility s... {'author': 'JENNIFER KAY and FREIDA FRISARO', ... {'author': 'DESMOND BOYLAN and ANDREA RODRIGUE... {'author': 'CHRISTOPHER SHERMAN', 'title': 'De... {'author': 'NICOLE WINFIELD', 'title': 'Pope a... {'author': 'GARY FINEOUT', 'title': 'Irma reco... {'author': 'FRANCES D'EMILIO', 'title': 'Heavy... {'author': '', 'title': 'Myanmar accused of la... {'author': 'BRIAN MELLEY and SALLY HO', 'title... {'author': 'RALPH D. RUSSO', 'title': 'Sooner ...
4 top bbc-news ok {'author': 'BBC News', 'title': 'Hurricane Irm... {'author': 'BBC News', 'title': 'Hurricane Irm... {'author': 'BBC News', 'title': 'Apple suffers... {'author': 'BBC News', 'title': 'Mexico earthq... {'author': 'BBC News', 'title': 'China looks a... {'author': 'BBC News', 'title': 'Rohingya cris... {'author': 'BBC News', 'title': 'Former govern... {'author': 'BBC News', 'title': 'Bodies of 'hu... {'author': 'BBC News', 'title': 'Bake Off: Pau... {'author': 'BBC News', 'title': 'North Korea: ...
In [16]:
results = pd.melt(frame = results, id_vars=['source','status','sortBy'], 
        var_name='new').drop(['status','sortBy','new'],axis=1)
In [ ]:
In [17]:
results.tail()
Out[17]:
source value
645 the-wall-street-journal {'author': 'Dudley Althaus', 'title': 'Irma Le...
646 the-washington-post {'author': None, 'title': 'Escaping North Kore...
647 time {'author': 'Julie Watson / AP', 'title': ''Lar...
648 usa-today {'author': None, 'title': 'Insider: Why this N...
649 wired-de NaN
In [18]:
#pd.concat([results['value'].apply(pd.Series)])
results = pd.concat([results.drop(['value'], axis=1), 
                     results['value'].apply(pd.Series)], axis=1)

results.tail()

C:\Users\Elizabeth Rychlinski\Anaconda3\lib\site-packages\pandas\core\indexes\api.py:43: RuntimeWarning: '>' not supported between instances of 'int' and 'str', sort order is undefined for incomparable objects
  union = _union_indexes(indexes)
C:\Users\Elizabeth Rychlinski\Anaconda3\lib\site-packages\pandas\core\indexes\api.py:77: RuntimeWarning: '>' not supported between instances of 'int' and 'str', sort order is undefined for incomparable objects
  result = result.union(other)
Out[18]:
source author description publishedAt title url urlToImage 0
645 the-wall-street-journal Dudley Althaus Hurricane Irma left widespread human and econo... 2017-09-10T15:20:00Z Irma Leaves Battered Caribbean in Its Wake https://www.wsj.com/articles/irma-leaves-batte... https://si.wsj.net/public/resources/images/BN-... NaN
646 the-washington-post None Ordinary North Koreans are risking their lives... None Escaping North Korea: 'We had already decided ... http://www.washingtonpost.com/sf/world/2017/09... http://www.washingtonpost.com/sf/world/wp-cont... NaN
647 time Julie Watson / AP 'Larger than life.' 2017-09-09T13:08:08Z 'Larger Than Life.' 65-Foot Photo of Toddler P... http://time.com/4934744/giant-photo-toddler-pe... https://timedotcom.files.wordpress.com/2017/09... NaN
648 usa-today None Don't ask Brian Kelly about it though. 2017-09-10T14:39:15Z Insider: Why this Notre Dame loss should feel ... http://www.indystar.com/story/sports/college/n... https://www.gannett-cdn.com/-mm-/4b0124d75a9fa... NaN
649 wired-de NaN NaN NaN NaN NaN NaN NaN
In [19]:
results = results.drop([0], axis=1)
In [20]:
results = results.dropna()
In [21]:
results.head()
#if description empty use title
Out[21]:
source author description publishedAt title url urlToImage
1 al-jazeera-english Al Jazeera Category 4 storm slams into US state with wind... 2017-09-10T11:53:00Z Hurricane Irma pounds southern Florida http://www.aljazeera.com/news/2017/09/hurrican... http://www.aljazeera.com/mritems/Images/2017/9...
2 ars-technica Annalee Newitz Librarians would have "rebutted it in a heartb... 2017-09-10T18:35:14Z Experts are extremely dubious about the Voynic... https://arstechnica.com/science/2017/09/expert... https://cdn.arstechnica.net/wp-content/uploads...
3 associated-press ST. PETERSBURG, Fla. (AP) — The Latest on Hurr... 2017-09-10T17:45:28Z The Latest: Utility says it will take weeks to... https://apnews.com/fe1652ad9c4941dcaa12b7a48bf... https://storage.googleapis.com/afs-prod/media/...
4 bbc-news BBC News Amateur footage shows Key Largo, where the sea... 2017-09-10T15:33:39Z Hurricane Irma: Storm drains bay in Florida Keys http://www.bbc.co.uk/news/av/world-us-canada-4... https://ichef.bbci.co.uk/news/1024/cpsprodpb/8...
5 bbc-sport BBC Sport Live text coverage and the best of social medi... 2017-09-10T18:40:48Z Froome completes historic Vuelta win - reaction http://www.bbc.co.uk/sport/live/cycling/40881041 http://newsimg.bbc.co.uk/media/images/67165000...
In [22]:
results = results.merge(right=sources, how='left', left_on='source', right_on='id')
In [23]:
results.head()
results = results.drop(['description_y', 'name', 'sortBysAvailable', 'url_y', 'id', 'urlToImage'], axis=1)
In [24]:
results.head()
Out[24]:
source author description_x publishedAt title url_x category country language
0 al-jazeera-english Al Jazeera Category 4 storm slams into US state with wind... 2017-09-10T11:53:00Z Hurricane Irma pounds southern Florida http://www.aljazeera.com/news/2017/09/hurrican... general us en
1 ars-technica Annalee Newitz Librarians would have "rebutted it in a heartb... 2017-09-10T18:35:14Z Experts are extremely dubious about the Voynic... https://arstechnica.com/science/2017/09/expert... technology us en
2 associated-press ST. PETERSBURG, Fla. (AP) — The Latest on Hurr... 2017-09-10T17:45:28Z The Latest: Utility says it will take weeks to... https://apnews.com/fe1652ad9c4941dcaa12b7a48bf... general us en
3 bbc-news BBC News Amateur footage shows Key Largo, where the sea... 2017-09-10T15:33:39Z Hurricane Irma: Storm drains bay in Florida Keys http://www.bbc.co.uk/news/av/world-us-canada-4... general gb en
4 bbc-sport BBC Sport Live text coverage and the best of social medi... 2017-09-10T18:40:48Z Froome completes historic Vuelta win - reaction http://www.bbc.co.uk/sport/live/cycling/40881041 sport gb en

Translation API

In [25]:
from google.cloud import translate
path = 'C:/Users/Elizabeth Rychlinski/Documents/GW/DATS-6450/Project/googleAPI.json'
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = path
# define client
translate_client = translate.Client()
In [26]:
def translated_test(text, target='en'):
    result = translate_client.translate(text, target_language=target)
    
    print('Translation: ', result['translatedText'])
    print('Source language: ', result['detectedSourceLanguage'])
In [27]:
text_test = 'Hola amigo'
translated_test(text_test)    

Translation:  Hello Friend
Source language:  es
In [28]:
german = results[results.language == 'de']
german = german.reset_index().drop(['index'], axis=1)
In [29]:
def translate(to_trans):    
    for i in range(len(to_trans)):
        translation = translate_client.translate(to_trans[i], target_language='en')
        to_trans[i] = translation['translatedText']
In [30]:
translate(german.description_x)
In [31]:
english = results[results.language == 'en']
In [49]:
to_combine = [english, german]
translated = pd.concat(to_combine, ignore_index=True)
In [35]:
translated.head()
Out[35]:
source author description_x publishedAt title url_x category country language
0 al-jazeera-english Al Jazeera Category 4 storm slams into US state with wind... 2017-09-10T11:53:00Z Hurricane Irma pounds southern Florida http://www.aljazeera.com/news/2017/09/hurrican... general us en
1 ars-technica Annalee Newitz Librarians would have "rebutted it in a heartb... 2017-09-10T18:35:14Z Experts are extremely dubious about the Voynic... https://arstechnica.com/science/2017/09/expert... technology us en
2 associated-press ST. PETERSBURG, Fla. (AP) — The Latest on Hurr... 2017-09-10T17:45:28Z The Latest: Utility says it will take weeks to... https://apnews.com/fe1652ad9c4941dcaa12b7a48bf... general us en
3 bbc-news BBC News Amateur footage shows Key Largo, where the sea... 2017-09-10T15:33:39Z Hurricane Irma: Storm drains bay in Florida Keys http://www.bbc.co.uk/news/av/world-us-canada-4... general gb en
4 bbc-sport BBC Sport Live text coverage and the best of social medi... 2017-09-10T18:40:48Z Froome completes historic Vuelta win - reaction http://www.bbc.co.uk/sport/live/cycling/40881041 sport gb en

Get Positive or Negative Articles with the News API and Google Cloud API

In [37]:
#Import libraries after installing the Google Language API: https://cloud.google.com/natural-language/docs/reference/libraries

from google.cloud import language

from google.cloud.language import enums

from google.cloud.language import types

# define client variable
client = language.LanguageServiceClient()
In [51]:
def sentiment_list():
    
    #local list, will be saved for later
    sent_list = []
    
    #iterates through the all articles capture by the News API. The datatframe is called "results"
    for i, row in translated.iterrows():
        
        #creates a dictionary to store the data capture in the loop
        sent_dict={}
        
        #uses the types.Document() function to perform analysis on the text and store it 
        document = types.Document(
            content=translated['description_x'][i],
            type=enums.Document.Type.PLAIN_TEXT)
           
        # Detects the sentiment of the text stored above
        sentiment = client.analyze_sentiment(document=document).document_sentiment
        
        # Assigns dictionary values for the URL, Title, Description, Sentiment, Magnitude, and Category
        sent_dict['URL'] = translated.url_x[i]
        sent_dict['Title'] = translated.title[i]
        sent_dict['Sentiment'] = sentiment.score
        sent_dict['Magnitude'] = sentiment.magnitude
        sent_dict['Description'] = translated.description_x[i]
        sent_dict['Category'] = translated.category[i]
        
        #Appending the values in our dictionary to a list
        sent_list.append(sent_dict)
        
    #We want to use this list later, so we will return it    
    return sent_list
In [52]:
sent_list = sentiment_list()
In [54]:
sent_list[5]
Out[54]:
{'Category': 'business',
 'Description': 'Hurricane Irma smashed into Florida as a powerful Category 4 storm, driving a wall of water and violent winds ashore and marking the first time since 1964 the U.S. was hit by back-to-back major hurricanes.',
 'Magnitude': 0.10000000149011612,
 'Sentiment': -0.10000000149011612,
 'Title': 'Hurricane Irma Makes Florida Landfall',
 'URL': 'https://www.bloomberg.com/news/articles/2017-09-10/hurricane-irma-smashes-ashore-in-florida-as-category-4-storm'}
In [55]:
def choose_news(sent_list):
    
    #User selects type of articles
    choice = input("Are you looking for positive or negative articles?  \n")
    
    #sorts list from highest to lowest (reverse=True) for the positive articles
    if choice == 'positive':
        sent_list = sorted(sent_list, key=lambda k: k['Sentiment'], reverse=True)
    
    #sorts list from lowest to highest for the negative articles
    if choice == 'negative':
        sent_list = sorted(sent_list, key=lambda k: k['Sentiment'])

    i=0
    for score in sent_list:
        if choice == 'positive':
            print('')
            print('Here is a generally POSITIVE sentiment article:')
            print('')
            if (score['Sentiment'] > .25 and  score['Magnitude'] > .5):
                print (u"Title: {}\nURL: {}\nDescription: {}\nSentiment: {}".format(score['Title'],score['URL'],score['Description'],score['Sentiment']))
            i=i+1   
            if i == 10:
                break
       
        if choice == 'negative':
            print('')
            print('Here is a generally NEGATIVE sentiment article:')
            print('')
            #sorted_list = sorted(sent_list, key=lambda k: k['Sentiment'])
            if (score['Sentiment'] <= -.25 and  score['Magnitude'] > .5):
                print (u"Title: {}\nURL: {}\nDescription: {}\nSentiment: {}".format(score['Title'],score['URL'],score['Description'],score['Sentiment']))
            i=i+1   
            if i == 10:
                break
In [56]:
choose_news(sent_list)

Are you looking for positive or negative articles?  
negative

Here is a generally NEGATIVE sentiment article:

Title: A Selection Of The Worst Hairstyles In Football History
URL: http://www.sportbible.com/football/news-reactions-take-a-bow-a-selection-of-the-worst-hairstyles-in-football-history-20170910
Description: After Goran Pandev's penis hairline, here are some of the worst we've ever seen...
Sentiment: -0.8999999761581421

Here is a generally NEGATIVE sentiment article:

Title: De Boer on the brink as Palace suffer worst start in Premier League history
URL: https://www.fourfourtwo.com/news/de-boer-brink-palace-suffer-worst-start-premier-league-history
Description: A 1-0 loss to Burnley on Sunday means Crystal Palace have suffered the worst ever start to a Premier League season.
Sentiment: -0.8999999761581421

Here is a generally NEGATIVE sentiment article:

Title: Noel Edmonds says litigation funder will bankroll his £300m battle with Lloyds
URL: https://www.theguardian.com/business/2017/sep/10/noel-edmonds-says-litigation-funder-will-bankroll-his-300m-battle-with-lloyds
Description: TV presenter seeks compensation over scam at group’s HBOS Reading arm, for which two ex-employees were convicted of fraud
Sentiment: -0.8999999761581421

Here is a generally NEGATIVE sentiment article:

Title: Wahlkampf: Die Nichtregierungskoalition
URL: http://www.spiegel.de/politik/deutschland/bundestagswahl-die-nichtregierungskoalition-von-union-spd-fdp-gruenen-und-linken-a-1166979.html
Description: SPD chancellor candidate Schulz calls a few coalition conditions, the FDP and Greens do not want to go to Jamaica, the union does not want with the Greens: the election campaign goes into the exclusion phase.
Sentiment: -0.8999999761581421

Here is a generally NEGATIVE sentiment article:

Title: Mihajlovic: 'Benevento unlucky'
URL: http://www.football-italia.net/109688/mihajlovic-benevento-unlucky
Description: Sinisa Mihajlovic admits “Benevento absolutely did not deserve to lose, but last season Torino lost this sort of game.”
Sentiment: -0.800000011920929

Here is a generally NEGATIVE sentiment article:

Title: Creepy Clown Lures Filmgoers, Making ‘It’ a Record-Setting Hit
URL: https://www.bloomberg.com/news/articles/2017-09-10/creepy-clown-lures-filmgoers-back-to-theaters-making-it-a-hit
Description: “It,” based on the Stephen King horror story, topped the box office in a record-breaking debut for a September release, a sign of hope for an industry reeling from its worst summer in more than a decade.
Sentiment: -0.800000011920929

Here is a generally NEGATIVE sentiment article:

Title: Equifax data breach hits nearly half of US – and isn’t over yet
URL: https://www.newscientist.com/article/2146758-equifax-data-breach-hits-nearly-half-of-us-and-isnt-over-yet/
Description: “Horrendous” hack of Equifax, a US firm responsible for millions of people’s credit ratings, is not the biggest in US history, but the data is the most sensitive
Sentiment: -0.800000011920929

Here is a generally NEGATIVE sentiment article:

Title: Andrew Sullivan: The Religious Right’s Suicidal Gay Obsession
URL: http://nymag.com/daily/intelligencer/2017/09/the-religious-rights-suicidal-gay-obsession.html
Description: Evangelical leaders are destroying their own movement by denying the humanity of LGBT people.
Sentiment: -0.800000011920929

Here is a generally NEGATIVE sentiment article:

Title: New Zealand rugby must change attitudes towards women, review finds
URL: https://www.theguardian.com/sport/2017/sep/11/new-zealand-rugby-must-change-attitudes-towards-women-review-finds
Description: Report reveals 36 incidents investigated in the last four years, including violence, sexual comments, homophobia, and drug and alcohol breaches
Sentiment: -0.800000011920929

Here is a generally NEGATIVE sentiment article:

Title: 21 Tweets From This Week That Will Have You Screaming
URL: https://www.buzzfeed.com/ryanschocket2/twenty-one-tweets-from-this-week-that-will-have-you
Description: Ever wonder if the bank looks at your account and thinks 'Wtf is this person doing?'
Sentiment: -0.800000011920929

Mining the results

In [43]:
def mine_results(sent_list):
    import re
    from nltk.tokenize import sent_tokenize, word_tokenize
    from nltk.corpus import stopwords
    from collections import Counter
    import warnings
    warnings.filterwarnings('ignore')
    # english stop words from nltk package
    stopWords = set(stopwords.words('english'))
    # send to dataframe
    sent_list = pd.DataFrame.from_dict(sent_list)
    # manipulate the string data
    for i in range(len(sent_list)):
        # replace any non-letter, space, or digit character in the headlines.
        sent_list.Description[i] = re.sub(r'[^\w\s\d]','',sent_list.Description[i].lower())
        # replace sequences of whitespace with a space character.
        sent_list.Description[i] = re.sub("\s+", " ", sent_list.Description[i])
    
    headlines = pd.DataFrame(sent_list.Description.astype(str))
    headlines = headlines.Description.str.cat()
    headlines = word_tokenize(headlines)
    
    remove_stopwords = []
    for i in range(len(headlines)):
        if headlines[i] not in stopWords:
            remove_stopwords.append(headlines[i])
    
    top_words = pd.Series(remove_stopwords).value_counts()[:10]
    top_words = top_words.sort_index()
    top_words.plot(kind = 'bar')
    plt.xticks(rotation = 45, fontsize = 13)
    plt.xlabel('Words')
    plt.ylabel('Frequency')
    plt.title('Top Words in the Current Trending Articles')
    plt.show()
In [44]:
mine_results(sent_list)
In [ ]:
In [ ]:
In [ ]:
Go Top