Comparing Parts of Speech with NLTK

In an earlier post, I showed you how to work with NLTK (the Natural Language Toolkit) in Python, using the texts that are optionally included with NLTK. There, we just sort of poked around trying various things. In this post, I will explain an actual statistical investigation of differences in the use of parts of speech using the same tools.

A bit of background: I recently ran a study comparing some parts of speech in blog posts by people who disclose an autism diagnosis to those parts of speech in blog posts by people without such a disclosure (presumed to not have autism). This is obviously messy public data not generated in a lab, but I found the signal I expected! You can view all the scripts used in my project at the GitHub repo for the study, if you’re interested.

In this post, we’re going to reproduce just the language analysis bit I used, not the web scraping. We’ll do the language analysis of parts of speech in State of the Union (SOTU) addresses. My (completely made up) hypothesis is that republicans and democrats differ in their use of comparative and superlative adjectives and adverbs (“best”, “most”, “angrier”, “grander”). I propose that one party has more extreme ways of describing the state of the nation in their State of the Union addresses.

Want to download this code instead of typing it yourself or copy/pasting? You can download:

• A Jupyter notebook of the code (recommended) or
• A python script of the code

Get Started

We’ll start with the obvious:

import nltk

Then we’ll import some texts to work with. If you haven’t already (or, if it’s been awhile), do the following, and (in a new window), choose to download “all”. It might take a while, but it’s worth it to have a lot of texts and tools to play with:

nltk.download()

Once that is accomplished (close the pop-up window), we can access some texts:

from nltk.corpus import state_union

Let’s take a peek at what’s inside!

state_union.fileids()

	['1945-Truman.txt',
	 '1946-Truman.txt',
	 '1947-Truman.txt',
	 '1948-Truman.txt',
	 '1949-Truman.txt',
	 '1950-Truman.txt',
	 '1951-Truman.txt',
	 '1953-Eisenhower.txt',
	 '1954-Eisenhower.txt',
	 '1955-Eisenhower.txt',
	 '1956-Eisenhower.txt',
	 '1957-Eisenhower.txt',
	 '1958-Eisenhower.txt',
	 '1959-Eisenhower.txt',
	 '1960-Eisenhower.txt',
	 '1961-Kennedy.txt',
	 '1962-Kennedy.txt',
	 '1963-Johnson.txt',
	 '1963-Kennedy.txt',
	 '1964-Johnson.txt',
	 '1965-Johnson-1.txt',
	 '1965-Johnson-2.txt',
	 '1966-Johnson.txt',
	 '1967-Johnson.txt',
	 '1968-Johnson.txt',
	 '1969-Johnson.txt',
	 '1970-Nixon.txt',
	 '1971-Nixon.txt',
	 '1972-Nixon.txt',
	 '1973-Nixon.txt',
	 '1974-Nixon.txt',
	 '1975-Ford.txt',
	 '1976-Ford.txt',
	 '1977-Ford.txt',
	 '1978-Carter.txt',
	 '1979-Carter.txt',
	 '1980-Carter.txt',
	 '1981-Reagan.txt',
	 '1982-Reagan.txt',
	 '1983-Reagan.txt',
	 '1984-Reagan.txt',
	 '1985-Reagan.txt',
	 '1986-Reagan.txt',
	 '1987-Reagan.txt',
	 '1988-Reagan.txt',
	 '1989-Bush.txt',
	 '1990-Bush.txt',
	 '1991-Bush-1.txt',
	 '1991-Bush-2.txt',
	 '1992-Bush.txt',
	 '1993-Clinton.txt',
	 '1994-Clinton.txt',
	 '1995-Clinton.txt',
	 '1996-Clinton.txt',
	 '1997-Clinton.txt',
	 '1998-Clinton.txt',
	 '1999-Clinton.txt',
	 '2000-Clinton.txt',
	 '2001-GWBush-1.txt',
	 '2001-GWBush-2.txt',
	 '2002-GWBush.txt',
	 '2003-GWBush.txt',
	 '2004-GWBush.txt',
	 '2005-GWBush.txt',
	 '2006-GWBush.txt']

Well, our dataset is limited, but it’s good enough for us to get started!

We can peer inside one more carefully (here, I’m only showing a bit of the output):

print(state_union.raw('2006-GWBush.txt'))

	PRESIDENT GEORGE W. BUSH'S ADDRESS BEFORE A JOINT SESSION OF THE 
	CONGRESS ON THE STATE OF THE UNION
	 
	January 31, 2006

	THE PRESIDENT: Thank you all. Mr. Speaker, Vice President Cheney, 
	members of Congress, members of the Supreme Court and diplomatic corps, 
	distinguished guests, and fellow citizens: Today our nation lost a 
	beloved, graceful, courageous woman who called America to its founding 
	ideals and carried on a noble dream. Tonight we are comforted by the 
	hope of a glad reunion with the husband who was taken so long ago, and 
	we are grateful for the good life of Coretta Scott King. (Applause.)...

Proof of Concept

First, we’ll create a function that will take a single text, and analyze it for parts of speech. First we’ll import a couple of needed modules:

from collections import Counter
from nltk.text import Text

And then we’ll create a function that turns the text to lower case, breaks it into words (tokenizing it), then assigns a part of speech for each token and returns the count for each part of speech in a dictionary:

def analyzeTextForPOS (text):
    # Tokenize the text
    tokens = nltk.word_tokenize(text.lower())
    text = nltk.Text(tokens)
    # get parts of speech for each token
    tags = nltk.pos_tag(text)
    # count how many times each pos is used
    counts = Counter(tag for word,tag in tags)
    # note that the POS abbreviations can be understood here:
    # https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html
    # return the counts as a dictionary
    return(dict(counts))

Let’s try the function to see if we get the kind of output we expect:

analyzeTextForPOS(state_union.raw('2006-GWBush.txt'))

	{'NN': 1080,
	 'VBD': 44,
	 'POS': 13,
	 'IN': 725,
	 'DT': 525,
	 'JJ': 515,
	 'CD': 71,
	 ',': 319,
	 ':': 81,
	 'PRP': 218,
	 '.': 336,
	 'NNS': 422,
	 'CC': 300,
	 'PRP$': 156,
	 'VBN': 110,
	 'WP': 19,
	 'TO': 175,
	 'VBP': 196,
	 'RB': 177,
	 '(': 68,
	 ')': 68,
	 'VB': 328,
	 'RP': 17,
	 'RBS': 4,
	 'VBZ': 130,
	 'EX': 9,
	 'MD': 122,
	 'JJR': 44,
	 'WDT': 29,
	 'VBG': 121,
	 'PDT': 5,
	 'WRB': 9,
	 'RBR': 8,
	 '``': 1,
	 "''": 1,
	 'JJS': 7,
	 '$': 4}

The output is a bit confusing. We can go to the Penn Treebank homepage to find out what these labels mean. Or, we could do the following, which shows examples (output truncated for this post):

nltk.help.upenn_tagset()

	$: dollar
	    $ -$ --$ A$ C$ HK$ M$ NZ$ S$ U.S.$ US$
	'': closing quotation mark
	    ' ''
	(: opening parenthesis
	    ( [ {
	): closing parenthesis
	    ) ] }
	,: comma
	    ,
	--: dash
	    --
	.: sentence terminator
	    . ! ?
	:: colon or ellipsis
	    : ; ...
	CC: conjunction, coordinating
	    & 'n and both but either et for less minus neither nor or plus so
	    therefore times v. versus vs. whether yet
	CD: numeral, cardinal
	    mid-1890 nine-thirty forty-two one-tenth ten million 0.5 one forty-
	    seven 1987 twenty '79 zero two 78-degrees eighty-four IX '60s .025
	    fifteen 271,124 dozen quintillion DM2,000 ...
	DT: determiner
	    all an another any both del each either every half la many much nary
	    neither no some such that the them these this those
	EX: existential there
	    there
	FW: foreign word
	    gemeinschaft hund ich jeux habeas Haementeria Herr K'ang-si vous
	    lutihaw alai je jour objets salutaris fille quibusdam pas trop Monte
	    terram fiche oui corporis ...

Looking there, we discover, for example, that PRP is the label for personal pronoun (like me, you, they) and PRP$ is the label for possessive pronoun (like my, your, their).

Great, but how can we do this for every text, combining where needed (e.g. combining all of a single president’s SOTU addresses into one measurement) and comparing where appropriate (Republican vs Democrat, 1960s vs 1980s)? One approach is to build a table of counts such that we have all the counts for all the texts, and then we can remix as needed for our study.

Note that when we get into analysis like this, there are usually several ways to proceed – so if you would approach the problem differently, that’s okay!

Textual Analysis

We’re going to create a function that sets a few things up, then iterates over every file in the state of the union corpus and collects three sets of data:

• The “basic” data of the president’s name, the year of the address, and a sequence number, if applicable
• The “part of speech” data that counts each part of speech in that state of the union address
• The total parts of speech that were tagged

We’ll then combine these three sets into one single data frame, which we return as the result.

First, we need to load a couple of libraries:

import pandas as pd
import re

And now we’ll create the function that does all the good stuff!

def analyzeTexts():

    # Create a pattern for the regular expression (regex)
    # we're going to do further on.
    
    pattern = "(\d+)\-([A-Za-z]+)\-?(\d?)\.txt"
    # This pattern captures the date:     (\d+)
    # Then looks for a dash:              \-
    # Then captures a name:               ([A-Za-z]+)
    # Then looks for an optional dash:    \-?
    # Then captures an optional number:   (\d?)
    # Then looks for the .txt ending:     \.txt

    all_basic_info = []  # list of basic info dicts
    all_pos_info = []    # list of part of speech dicts
    all_totals = []      # list of total POS counted dicts
    
    # Note: we create three lists so that when we assemble
    # a data frame, these three types of information can be
    # placed in order, with basic info followed by pos counts
    # followed by total.  Since dicts are inherently unordered,
    # if we did a single list, we'd end up with columns in 
    # alphabetical order, which is not what we want.

    
    # iterate over files and extract and organize data
    
    for filename in state_union.fileids():
        address_details = {}    # empty dict
        pos = {}                # empty dict
        totals = {}             # empty dict
        
        # Below, we're going to use  regular expressions to 
        # extract meaningful info from the file name.
        
        m = re.match(pattern, filename)
        if m:
            address_details['address_date'] = m.group(1)
            address_details['president_name'] = m.group(2)
            address_details['optional_sequence'] = m.group(3)
            all_basic_info.append(address_details)  # add details of this file
            
            pos = analyzeTextForPOS(state_union.raw(filename))
            all_pos_info.append(pos)                # add part of speech for this file
            
            totals['Total'] = sum(pos.values())
            all_totals.append(totals)               # add total POS for this file

        else:
            print("Wonky problem with filename " + filename)
        
    return(pd.DataFrame(all_basic_info).join(pd.DataFrame(all_pos_info).join(pd.DataFrame(all_totals))))
    # return the results in a data frame -- leftmost columns are basic info,
    # then the part of speech info, then the totals in the rightmost column.

Now, let’s execute that function!

sotu_data = analyzeTexts()

Take a peek at what we’ve built:

sotu_data

Keep in mind that the numbers in the part of speech columns are just counts! We’ll have to figure out percentages, since we want to compare the relative frequency of parts of speech.

Also, we want to combine the part of speech counts and the total parts of speech counted for each president – so, all of Nixon’s SOTU speeches combined into one count, etc. The reason for this is that the number of speeches for each president are unbalanced. If we don’t combine them, we risk over-weighting the influence of one president compared to another in our statistical analysis.

We also have too many parts of speech to work with. We really want to work with just a handful.

So, we have to slim down and consolidate our large data frame. We can do this by indexing and by using the pandas groupby function.

We’ll do this in steps. First, we’ll get all rows and just the columns starting with president_name forward.

sotu_by_president = sotu_data.loc[:,'president_name':]

Then we’ll aggregate (combine rows) by president name:

sotu_by_president = sotu_by_president.groupby('president_name').sum() 

Let’s take a peek so far:

sotu_by_president

And we’ll limit to just comparative and superlative adjectives and adverbs. But we’ll also hang on to ‘Total’, because that allows us to determine the percentage of speech that each part of speech takes up.

cols_i_want = ['JJR', 'JJS','RBR','RBS', 'Total']
sotu_by_president = sotu_by_president.loc[:,cols_i_want]

We’ll separate out republicans from democrats:

republicans = ['Bush','Eisenhower','Ford','Nixon','Reagan','GWBush']
democrats = ['Carter','Clinton','Johnson','Kennedy','Truman']

republican_speech = sotu_by_president.loc[sotu_by_president.index.isin(republicans)]
democrat_speech = sotu_by_president.loc[sotu_by_president.index.isin(democrats)]

Let’s take a quick peek:

republican_speech

democrat_speech

Now we have some nice, compact data frames! We still have to figure out percentages, however. We’ll divide everything by the ‘Total’ column:

democrat_speech = democrat_speech.div(democrat_speech['Total'], axis=0)
republican_speech = republican_speech.div(republican_speech['Total'], axis=0)

Statistical Analysis

Do Republicans and Democrats differ in the way they use comparative and superlative adjectives and adverbs in their State of the Union addresses? Let’s take each type of part of speech separately and compare them, then sum all four parts of speech and make an overall comparison as well. We’ll visualize differences using a box plot, and also conduct a two-sample T test.

We’ll need some additional python libraries, and need to set up some plot parameters to display our visualizations in our Jupyter Notebook:

from scipy.stats import ttest_ind
from statistics import mean, stdev
from math import sqrt
import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams.update({'font.size': 18})

And now we’ll plot the comparative adjective use of both groups.

It doesn’t look very promising – these distributions overlap rather a lot! Let’s do a T test to see if the difference is likely due to random variation:

ttest_ind(republican_speech["JJR"],democrat_speech["JJR"])

	Ttest_indResult(statistic=0.27734514630142165, pvalue=0.7877802746796846)

That’s a very high p value. 78.8% probability that the differences we see are due to random chance, not a true group difference.

Let’s move on to superlative adjectives!

fig = plt.figure(figsize=(10,10))
fig.suptitle('Superlative Adjective Usage', fontsize=30, fontweight='bold')

ax = fig.add_subplot(111)
ax.boxplot([democrat_speech["JJS"],republican_speech["JJS"]], labels=["Democrats", "Republicans"])

ax.set_xlabel('Party')
ax.set_ylabel('Percent of Language')

plt.show()

ttest_ind(republican_speech["JJS"],democrat_speech["JJS"])

	Ttest_indResult(statistic=-0.7498826364182734, pvalue=0.47247171059893134)

Once again, both visually and statistically, there’s no significant difference. Let’s do the same for both comparative and superlative adjectives!

fig = plt.figure(figsize=(10,10))
fig.suptitle('Comparative Adverb Usage', fontsize=30, fontweight='bold')

ax = fig.add_subplot(111)
ax.boxplot([democrat_speech["RBR"],republican_speech["RBR"]], labels=["Democrats", "Republicans"])

ax.set_xlabel('Party')
ax.set_ylabel('Percent of Language')

plt.show()

ttest_ind(republican_speech["JJS"],democrat_speech["JJS"])

	Ttest_indResult(statistic=1.0781202441947808, pvalue=0.30902662581074575)

fig = plt.figure(figsize=(10,10))
fig.suptitle('Superlative Adverb Usage', fontsize=30, fontweight='bold')

ax = fig.add_subplot(111)
ax.boxplot([democrat_speech["RBS"],republican_speech["RBS"]], labels=["Democrats", "Republicans"])

ax.set_xlabel('Party')
ax.set_ylabel('Percent of Language')

plt.show()

ttest_ind(republican_speech["RBS"],democrat_speech["RBS"])

	Ttest_indResult(statistic=0.12929601859348358, pvalue=0.8999668706431508)

Well, none of the individual parts of speech were notably different, but what if we sum them?

republican_all = republican_speech["JJR"] + republican_speech["JJS"] + republican_speech["RBR"] + republican_speech["RBS"]
democrat_all = democrat_speech["JJR"] + democrat_speech["JJS"] + democrat_speech["RBR"] + democrat_speech["RBS"]

ttest_ind(republican_all,democrat_all)

	Ttest_indResult(statistic=0.2468003972480926, pvalue=0.8106000875259137)

Nope, no finding here. It looks like we can reject my hypothesis that presidential speakers of different parties express different concentrations of comparative and superlative adjectives and adverbs!

How could this investigation have been done better? Well, we’d ideally like more text from more presidents – maybe get all of the SOTU addresses instead of a small chunk of years, or add in other texts. We also want to be careful to remove annotations like “(Applause)” and photo credits, which aren’t really part of the presidential address. We didn’t do that here. There are lots of ways to remix this. Let me know what you come up with and if you use this technique in your research!