USA Today once was a mediocre newspaper that had one virtue: it was convenient for travelers, and sadly more useful for following non-locale news development than all but a handful of city publications. Now it isn’t a newspaper at all, but some hybrid monstrosity that is laid out like a website, has articles too short to be complete or helpful, and a product pandering to those with small vocabularies and attention spans that have been destroyed by the internet. But it’s often free, so on my latest (horrible, miserable, disaster-filled) seminar tour around Virginia, I had the pleasure of opening an edition and seeing what immediately struck me as the kind of feature no respectable journalistic enterprise would tolerate.
USA Today political writer Paul Singer thought newsworthy a ridiculous exercise that could only have sprung from a toxic mix of bias and silliness. It’s objective: let’s either prove that Republicans and conservatives are dumber than their Democrat, liberal counterparts, or prove that an accepted way of measuring intelligence is inaccurate for the purpose, because it doesn’t prove that Republicans are morons, and we all know they are. The feature was called “Democrats crush Republicans in grammar; Chafee on top.”
This is yet another self-rebutting exercise, as proven by the headline. Lincoln Chafee is a well-established boob, as they will tell you, if you ask, in his home base of Rhode Island. The man announced his Presidential run citing his primary cause as getting the U.S. to adopt the metric system. This immediately places him in the long and amusing line of wacko candidates, including…
Homer Aubrey Tomlinson, who was a New York City preacher that ran for the presidency under the banner of the Theocratic Party in five elections, from 1952 until his death in 1968. He wanted to replace taxation with tithing and promised to create a new cabinet post: Secretary of Righteousness. Later, Tomlinson declared himself King of the World and staged coronation ceremonies in 101 different countries, in which he appeared wearing a gold-plated crown, an inflatable globe and a folding chair as his throne. And…
California congressman John G. Schmitz, who was the American Independent Party candidate for president in 1972. He was expelled from the John Birch Society for “extremism,” which sort of says it all. Schmitz also endorsed the return of segregated schools, and later announced that he was rooting for a military coup. Mary Kay Letourneau is his daughter. Then there is…
HRM Caesar St. Augustine de Buonaparte, who is running now as The Absolute Dictator Party’s candidate. He says that all the major politicians are “niggers” and so is everyone else “because we all die on our death bed and watch our offspring fight over our money.” He pledges to replace any government employee who does not have an IQ of at least 150.
So if Chafee has the followers with the most facility with the language, what does it tell us about the usefulness of that factor in assessing, well, anything? It tells me that this was an inquiry designed to embarrass Republicans that failed, but USA Today decided to publish it anyway with big color graphics using up about half a page in a paper that typically has only a couple of pages as substance.
The stunt was the brainchild of some Marketing flack at Grammarly, a writing app that thought it might increase the number of people who ever heard of it from five to nineteen. According to a Grammarly release, using the app on the websites of presidential candidates’ Facebook pages showed that Democratic commenters made an average of 4.2 mistakes per 100 words compared to 8.7 mistakes for supporters of Republican candidates. The Democratic supporters also showed a larger vocabulary, using on average 300 unique words per 1,000 words, while Republicans used only 245. Here was the methodology:
We began by taking a large sample of Facebook comments containing at least fifteen words from each candidate’s official page between April, 2015 and August, 2015. Next, we created a set of guidelines to help limit (as much as possible) the subjectivity of categorizing the comments as positive or negative. Since the point of the study was to analyze the writing of each candidate’s supporters, we considered only obviously positive or neutral comments. Obviously negative or critical comments, as well as ambiguous or borderline negative comments, were disqualified.
We then randomly selected at least 180 of these positive and neutral comments (~6,000 words) to analyze for each candidate. Using Grammarly, we identified the errors in the comments, which were then verified and tallied by a team of live proofreaders. For the purposes of this study, we counted only black-and-white mistakes such as misspellings, wrong and missing punctuation, misused or missing words, and subject-verb disagreement. We ignored stylistic variations such as the use of common slang words, serial comma usage, and the use of numerals instead of spelled-out numbers.
Finally, we calculated the average number of mistakes per one hundred words by dividing the total word count of the comments by the total number of mistakes for each candidate.
There are many problems with this, of course, the primary one being “Who cares?,” followed by “How do you know that the same commenters aren’t writing on the walls of multiple candidates?” “Isn’t this another classist, pro-coastal, elitist exercise?” “Since when is Facebook spelling and grammar an accepted measure of anything?” “How about finding out how many supporters of each candidate read USA Today, or worse, trust it?”
Now there’s an intelligence test.
Why would people waste their time writing on campaign Facebook pages, when almost none of the candidates actually look at them? How do we know the smartest Democratic supporters waste their time on Facebook, while only the dumbest Republican supporters use is? But never mind all the problems with the methodology: Grammarly is a lousy app and doesn’t work.
The website Grammarist used to advertise Grammerly but took down the ads in 2012 after testing the software and finding it seriously lacking. (USA Today apparently didn’t do any due diligence to vet its source—it liked the results of the “study” too much.) Grammarist began its inquiry by noting…
Grammarly claims to (1) correct grammar, (2) give useful explanations for grammar mistakes, (3) check spelling, (4) identify and help correct plagiarism, and (5) check word usage. We’re skeptical on every count, but number five strikes us as particularly implausible. “Usage” refers to the way words are actually used by speakers of a language, and we have trouble believing that a piece of software could evaluate usage in a way that takes the language’s flux into account or intelligently interprets all the connotations of words.
When we think about it, the idea of one-size-fits-all software to correct writing raises all sorts of questions. What about different varieties of English? How does Grammarly evaluate words spelled differently in American and British English? What about words that can be pluralized in multiple ways? What about new words, colloquialisms, idioms, and words that are changing meaning in real-world usage? What about redundancies and wordiness? And what are Grammarly’s style policies when it comes to, say, the Oxford comma, capitalization of titles, and the hyphenation of phrasal adjectives? If Grammarly has policies on these matters, how do they accommodate differing views?
No doubt Grammarly will come up short in some ways, but we’ll keep an open mind. Not everyone is as obsessed with English as we are, and maybe the software will prove serviceable for everyday writing of papers, letters, blog posts, and so on.
To test Grammarly, Grammarist signed up for a Grammarly membership and composed a series of sentences that it considered incorrect or questionable. It ran them through Grammarly and recorded what the software’s verdict was. It also tested correctly written sentences “that some stodgy grammarians might consider incorrect.” These were the results in the areas relevant to the study of presidential supporters. The bolded comments in brackets are mine.
Obvious spelling errors
I definately love cats.
Grammarly makes no corrections.
Kittens are the cuetest.
Grammarly correctly catches the spelling error, but incorrectly suggests changing are to is.
I especially like the mischievious ones.
Grammarly instructs us to review this sentence for sentence fragments, but we have no idea why, [and] No correction of “mischievious.”
Less obvious spelling errors
I got a letter form my girlfriend.
She said she wont ever stop loving me.
She makes me loose my mind.
Grammarly correctly suggests lose in place of loose.
Grammar and punctuation mistakes
The term paper had no typo’s.
No corrections. [ Me: !!!!!]
I don’t understand it’s point.
Grammarly tells us to avoid contractions. This is good advice in some types of formal writing, but it’s by no means a blanket rule. [ Me: Especially on Facebook!]
Grammarly correctly suggests its in place of it’s.
So the student and me had a talk after class.
Grammarly incorrectly suggests inserting a comma after “So the student.”
Grammarly does not suggest student and I in place of student and me.
I said it was well written, however the argument was confusing.
Grammarly correctly catches this run-on sentence and suggests a semicolon in place of the comma.
Grammarly urges us to review this sentence for the passive voice. Making passive sentences active sometimes improves them, but the passive voice is often useful and shouldn’t be prohibited.
Reading it, the ideas just never came together.
No correction of the dangling modifier.
Not only was the paper confusing but also late.
No correction of the misplaced correlative conjunctions.
Grammarist’s conclusion: “Grammarly doesn’t work. As the above results show, Grammarly did not catch several of our intentional grammar and spelling errors, it had nothing to say about any of our intentionally misused words, and it makes recommendations based on 19th-century grammar superstitions. This last point is especially interesting considering that Grammarly is rather laissez-faire with new words such as winningest. Elsewhere, Grammarly is inconsistent in applying its rules. And there is no way to tell Grammarly that we are American writers, so we could spell everything in the British manner and Grammarly would not question it. There’s also no way to indicate to Grammarly the level of formality of the text being checked.
We might be tempted to suggest Grammarly for students or learners of English, but the fact that Grammarly has nothing to say about so many of our intentionally incorrect sentences leads us to the sad conclusion that Grammarly is useless for everyone.”
The Ethics Alarms verdict: USA Today and Singer were unethical, biased, careless and dishonest to cite the apps’ self-promoting study in multiple ways. Unless Singer could independently verify that the software did what it claimed to do, its results were unreliable, even if the theory behind the whole exercise wasn’t so shot through with hidden biases and flaws, which it was. Is it possible that the app has improved since 2012? I’d say it’s likely, but USA Today had an obligation to check and so inform readers, which it did not. Singer’s article was junk science at best, and an embarrassing example of confirmation bias at work on the author’s part and on the part of USA Today editors.
Here’s the published chart: