Facebook Grammar, Lincoln Chafee, USA Today, and Confirmation Bias

The smartest supporters of all!

He has the smartest supporters of all!

USA Today once was a mediocre newspaper that had one virtue: it was convenient for travelers, and sadly more useful for following non-locale news development than all but a handful of city publications. Now it isn’t a newspaper at all, but some hybrid monstrosity that is laid out like a website, has articles too short to be complete or helpful, and a product pandering to those with small vocabularies and attention spans that have been destroyed by the internet. But it’s often free, so on my latest (horrible, miserable, disaster-filled) seminar tour around Virginia, I had the pleasure of opening an edition and seeing what immediately struck me as the kind of feature no respectable journalistic enterprise would tolerate.

USA Today political writer Paul Singer thought newsworthy a ridiculous exercise that could only have sprung from a toxic mix of bias and silliness. It’s objective: let’s either prove that Republicans and conservatives are dumber than their Democrat, liberal counterparts, or prove that an accepted way of measuring intelligence is inaccurate for the purpose, because it doesn’t prove that Republicans are morons, and we all know they are. The feature was called “Democrats crush Republicans in grammar; Chafee on top.”

This is yet another self-rebutting exercise, as proven by the headline. Lincoln Chafee is a well-established boob, as they will tell you, if you ask, in his home base of Rhode Island. The man announced his Presidential run citing his primary cause as getting the U.S. to adopt the metric system. This immediately places him in the long and amusing line of wacko candidates, including…

Homer Aubrey Tomlinson, who was a New York City preacher that ran for the presidency under the banner of the Theocratic Party in five elections, from 1952 until his death in 1968. He wanted to replace taxation with tithing and promised to create a new cabinet post: Secretary of Righteousness. Later, Tomlinson declared himself King of the World and staged coronation ceremonies in 101 different countries, in which he appeared wearing a gold-plated crown, an inflatable globe and a folding chair as his throne. And…

California congressman John G. Schmitz, who was the American Independent Party candidate for president in 1972. He was expelled from the John Birch Society for “extremism,” which sort of says it all. Schmitz also endorsed the return of segregated schools, and later announced that he was rooting for a military coup. Mary Kay Letourneau is his daughter. Then there is…

HRM Caesar St. Augustine de Buonaparte, who is running now as The Absolute Dictator Party’s candidate. He says that all the major politicians are “niggers” and so is everyone else “because we all die on our death bed and watch our offspring fight over our money.” He pledges to replace any government employee who does not have an IQ of at least 150.

So if Chafee has the followers with the most facility with the language, what does it tell us about the usefulness of that factor in assessing, well, anything? It tells me that this was an inquiry designed to embarrass Republicans that failed, but USA Today decided to publish it anyway with big color graphics using up about half a page in a paper that typically has only a couple of pages as substance.

The stunt was the brainchild of some Marketing flack at Grammarly, a writing app that thought it might increase the number of people who ever heard of it from five to nineteen. According to a Grammarly release, using the app on the websites of presidential candidates’ Facebook pages showed that Democratic commenters made an average of 4.2 mistakes per 100 words compared to 8.7 mistakes for supporters of Republican candidates. The Democratic supporters also showed a larger vocabulary, using on average 300 unique words per 1,000 words, while Republicans used only 245. Here was the methodology:

We began by taking a large sample of Facebook comments containing at least fifteen words from each candidate’s official page between April, 2015 and August, 2015. Next, we created a set of guidelines to help limit (as much as possible) the subjectivity of categorizing the comments as positive or negative. Since the point of the study was to analyze the writing of each candidate’s supporters, we considered only obviously positive or neutral comments. Obviously negative or critical comments, as well as ambiguous or borderline negative comments, were disqualified.

We then randomly selected at least 180 of these positive and neutral comments (~6,000 words) to analyze for each candidate. Using Grammarly, we identified the errors in the comments, which were then verified and tallied by a team of live proofreaders. For the purposes of this study, we counted only black-and-white mistakes such as misspellings, wrong and missing punctuation, misused or missing words, and subject-verb disagreement. We ignored stylistic variations such as the use of common slang words, serial comma usage, and the use of numerals instead of spelled-out numbers.

Finally, we calculated the average number of mistakes per one hundred words by dividing the total word count of the comments by the total number of mistakes for each candidate.

There are many problems with this, of course, the primary one being “Who cares?,” followed by “How do you know that the same commenters aren’t writing on the walls of multiple candidates?” “Isn’t this another classist, pro-coastal, elitist exercise?” “Since when is Facebook spelling and grammar an accepted measure of anything?” “How about finding out how many supporters of each candidate read USA Today, or worse, trust it?”

Now there’s an intelligence test.

Why would people waste their time writing on campaign Facebook pages, when almost none of the candidates actually look at them? How do we know the smartest Democratic supporters waste their time on Facebook, while only the dumbest Republican supporters use is? But never mind all the problems with the methodology: Grammarly is a lousy app and doesn’t work.

The website Grammarist used to advertise Grammerly but took down the ads in 2012 after testing the software and finding it seriously lacking. (USA Today apparently didn’t do any due diligence to vet its source—it liked the results of the “study” too much.) Grammarist  began its inquiry by noting…

Grammarly claims to (1) correct grammar, (2) give useful explanations for grammar mistakes, (3) check spelling, (4) identify and help correct plagiarism, and (5) check word usage. We’re skeptical on every count, but number five strikes us as particularly implausible. “Usage” refers to the way words are actually used by speakers of a language, and we have trouble believing that a piece of software could evaluate usage in a way that takes the language’s flux into account or intelligently interprets all the connotations of words.

When we think about it, the idea of one-size-fits-all software to correct writing raises all sorts of questions. What about different varieties of English? How does Grammarly evaluate words spelled differently in American and British English? What about words that can be pluralized in multiple ways? What about new words, colloquialisms, idioms, and words that are changing meaning in real-world usage? What about redundancies and wordiness? And what are Grammarly’s style policies when it comes to, say, the Oxford comma, capitalization of titles, and the hyphenation of phrasal adjectives? If Grammarly has policies on these matters, how do they accommodate differing views?

No doubt Grammarly will come up short in some ways, but we’ll keep an open mind. Not everyone is as obsessed with English as we are, and maybe the software will prove serviceable for everyday writing of papers, letters, blog posts, and so on.

To test Grammarly, Grammarist signed up for a Grammarly membership and composed a series of sentences that it considered incorrect or questionable. It ran them through Grammarly and recorded what the software’s verdict was. It also tested correctly written  sentences “that some stodgy grammarians might consider incorrect.” These were the results in the areas relevant to the study of presidential supporters. The bolded comments in brackets are mine.

Obvious spelling errors

I definately love cats.

 Grammarly makes no corrections.

Kittens are the cuetest.

Grammarly correctly catches the spelling error, but  incorrectly suggests changing are to is.

I especially like the mischievious ones.

Grammarly instructs us to review this sentence for sentence fragments, but we have no idea why, [and] No correction of “mischievious.”

Less obvious spelling errors

I got a letter form my girlfriend.

No corrections.

She said she wont ever stop loving me.

No corrections.

She makes me loose my mind.

Grammarly correctly suggests lose in place of loose.

Grammar and punctuation mistakes

The term paper had no typo’s.

No corrections. [ Me: !!!!!]

I don’t understand it’s point.

Grammarly tells us to avoid contractions. This is good advice in some types of formal writing, but it’s by no means a blanket rule. [ Me: Especially on Facebook!]

Grammarly correctly suggests its in place of it’s.

So the student and me had a talk after class.

Grammarly incorrectly suggests inserting a comma after “So the student.”

Grammarly does not suggest student and I in place of student and me.

I said it was well written, however the argument was confusing.

Grammarly correctly catches this run-on sentence and suggests a semicolon in place of the comma.

Grammarly urges us to review this sentence for the passive voice. Making passive sentences active sometimes improves them, but the passive voice is often useful and shouldn’t be prohibited.

Reading it, the ideas just never came together.

No correction of the dangling modifier.

Not only was the paper confusing but also late.

No correction of the misplaced correlative conjunctions.

Grammarist’s conclusion: “Grammarly doesn’t work. As the above results show, Grammarly did not catch several of our intentional grammar and spelling errors, it had nothing to say about any of our intentionally misused words, and it makes recommendations based on 19th-century grammar superstitions. This last point is especially interesting considering that Grammarly is rather laissez-faire with new words such as winningest. Elsewhere, Grammarly is inconsistent in applying its rules. And there is no way to tell Grammarly that we are American writers, so we could spell everything in the British manner and Grammarly would not question it. There’s also no way to indicate to Grammarly the level of formality of the text being checked.

We might be tempted to suggest Grammarly for students or learners of English, but the fact that Grammarly has nothing to say about so many of our intentionally incorrect sentences leads us to the sad conclusion that Grammarly is useless for everyone.”

The Ethics Alarms verdict: USA Today and Singer were unethical, biased, careless and dishonest to cite the apps’ self-promoting study in multiple ways. Unless Singer could independently verify that the software did what it claimed to do, its results were unreliable, even if the theory behind the whole exercise wasn’t so shot through with hidden biases and flaws, which it was. Is it possible that the app has improved since 2012? I’d say it’s likely, but USA Today had an obligation to check and so inform readers, which it did not. Singer’s article was junk science at best, and an embarrassing example of confirmation bias at work on the author’s part and on the part of USA Today editors.

Here’s the published chart:

Presidential grammar

______________________

Pointer: Newsbusters

Sources: USA Today, Grammarist

47 thoughts on “Facebook Grammar, Lincoln Chafee, USA Today, and Confirmation Bias

    • Wyo, I was neither. I use Word 2000, which has both a spell checker and a supposed grammar checker. The spell-checker does not check for inappropriate word usage, not does the grammar checker, apparently. There is no substitute for accurate proof-reading, and even then, typo’s escape notice. Can’t speak to the latest version of Office, but I’d certainly trust Microsoft more than some off-brand software.

  1. I find the “unique words per thousand words” comparison particularly galling. Elegant variation is the worst idea ever foisted upon good expository writing technique. If it’s accurate, maybe the results mean Republicans express their ideas more simply so they are more readily understood.

    • Elegant variation is something that French prose stylists take to even greater extremes, which sometimes leads to contortions that can even reach absurdity. For instance, essays discussing France are often written to use synonyms or metaphors for France, e.g. “the hexagon” (because of the approximate shape of France on a map) is fairly often used in the expression “the four corners of the hexagon”.

  2. The methodology does state: “which were then verified and tallied by a team of live proofreaders.” It doesn’t mean the proofreaders have any idea of course, but the whole thing is still gutter press anyway you look at it.

    My question would be, why look at a rag like that anyway? Yes, a good place to find inspiration for a blog, if you only read worthwhile publications you won’t have much to read, and you’re on a mission to improve ethical standards globally.

    The latter won’t really work unless you send a copy of your blog to the paper as a letter to the editor and they actually publish it. Good luck with that!

    Did you like my Oxford comma?

    I trust everyone has seen Weird Al Yakovic’s ‘Word Crimes’?

  3. Well, we should be on the metric system, but it is amazing to me that this is the thrust of Chafee’s platform.

    I could not read the rest of your post because grammar errors make my blood boil. I know I am guilty of some here, but that is because I am not typing carefully. Work is another story.

      • Because it is so easy to use. Here in New Zealand we converted to the metric system in 1976. My father was a carpenter and he told me that the change from feet, inches, and fractions of an inch to metres and millimetres made measurements much easier to use.

        • That’s actually subtly wrong. Like decimal coinage, the metric system isn’t easier to use in general; it depends on the application area, with some areas getting harder if the metric system works with units that are less convenient for human beings (which is why beer is served in pints or half pints but wine usually comes in non-standard multiples of litres). However, decimal coinage and the metric system are easier to learn – which was a non-problem for most of the old systems’ users after they had learned those at school (pounds, shillings and pence actually helped a great deal with evenly splitting amounts among several people as the ratios had many factors, which was important in the days when money was valuable enough that any rounded amounts were significant).

        • Easy to use, not so easy to convert to when so much of the culture–that word again—is so entrenched in another system. The baseball field isn’t going to stop being 90 ft between bases (and the game won’t work right if is any longer or shorter.) Football fans understand what a 60 yard rush is. 6 feet tall and 200 pounds means something: Abe was 6’4″…Taft weighed 300 pounds. Pound for pound, give them an inch and they’ll take a mile, the whole 9 yards, and my oil is a quart low. American pace of life is too fast to stop and learn this stuff, so it’s an annoyance and not a perceived benefit. I remember the attempted conversion and everyone’s reaction to it as Big Brotherism. When Americans are ordered to change, they tend to dig in their heels.

          Thank God.

  4. It has long been my suspicion that political Conservative ranks have a higher concentration of both very smart people and very dumb people, while Liberals have a higher concentration of people in the middle (mental “adolescents”) and fewer extremes on both ends of the spectrum.

    One reason I suspect this is that I find the best conservative minds and arguments far more logical than the best Liberal ones (on the whole) but there seem to be more reasonably intelligent people siding with the Left.

    People of middling intelligence can write and spell well. And this USA Today exercise (I don’t deign to call it a study) corroborates my little theory. Assuming the exercise is not just plain flawed…it could only be proving that conservatives candidates attract more of the rabble. Or that the Conservative rabble is more likely to at least care about politics than the Liberal rabble (also very likely.)

    • So you think the best measure of intelligence is correct spelling? Amazingly naive. I suggest you check out Wikipedia and see how intelligence tests are constructed. Spelling is a measure of achievement not intelligence.

      • I don’t think that at all. I do assume there is some correlation between intelligence and correct use of language in groups of people. Just as there is some correlation between intelligence and achievement. The USA Today hacks know that people know this, and are implying that Democrats are smarter by presenting evidence (however flawed) that Democrats use language more correctly. Jack has pointed out several reasons why this is a bad line of reasoning, including the flawed method of data collection. I just added a couple more reasons why, even assuming that their data is useful and reflects the relative intelligence of the Facebook posters, it still probably doesn’t lead to the conclusion they want it to.

      • On further review, I think my offending sentence was “People of middling intelligence can write and spell well.” My bad. I meant generally speaking, in the context of large groups. All sociology deals in generalizations. I didn’t mean that individuals can be judged that way. That was confusing, sorry.

        • That’s not a bad theory Isaac. There is some support for that. Anecdotally, I had to quit watching This Week with David Brinkley when they brought on George Stephanopoulos. George Will and George S were supposed to be equals, I suppose, but it was like following Mozart with Milli Vanilli. I’m not sure they can come up with a counterpart to George Will. Limbaugh has ruled the air waves for years with virtually no left opposition (and whatever you think of his theatrics, anyone who invents an industry, and then dominates it for decades, is intelligent).

          Further support for your theory may be that uber-smart conservatives are not using Facebook.

  5. Since I just can’t seem to get sleepy…here are some more possible conclusions different from the one USA Today wants you to draw…

    -Democrats are more experienced at online banter, because they spend less time doing productive things.
    -All but the least intelligent Republicans aren’t sheeplike enough to follow a politician’s Facebook page.
    -Republicans for whom English is a second language are more active in politics than Democrats for whom English is a second language.
    -Smart Democrats use more social media than smart Republicans, because Democrats, of all linguistic skill levels, are more egotistical and derive more self-worth from having a wide online audience.
    -Democratic candidates are better at mobilizing slacktivists.
    -Democrats are more likely to hire online reputation management companies and create sock-puppet accounts.
    -Democratic social-media managers are more likely to hide or delete unflattering posts by their own allies, while Republicans prefer to value freedom of expression.
    -All the smart Republicans are waiting until the herd thins on their side to even start caring.

        • Amusingly enough, a good rule of thumb is this:

          When Leftwingers start beating the drum about some evil the Right-wing is supposedly pushing, there’s a solid chance the Left IS actually engaging in it. I didn’t realize it until recently, but Hillary screaming about a “vast Right wing conspiracy” was all diversion in the 90s as the final ties were being solidified between Big Education-Big Government-Big Media-Big Business-Big Legal etc… you know…an actual Vast Leftwing Conspiracy.

          When the Left cries about evil Republicans wanting government and business to be in bed with each other, actual research will show that large corporations and big business is by an large in bed with the Democrats.

          The Left accuses the Right of perpetuating racism and seeking de jure segregation all while the policies pushed by the Left have INCREASED racism and a de facto segregation.

          It’s all very Orwellian.

    • My last response to the assertion that spelling achievement is strongly correlated with IQ: At the low end of the bell curve (IQ below 70) there is a fairly robust correlation although I have known “kids” who are excellent spellers who fall below an IQ of 70, I.e. the idiot savant phenomena. However, at the higher end, there is a low correlation due to the presence of learning disabilities. Einstein, for example was a notoriously poor speller. A link follows addressing standardized testing: http://schoolpsychologistfiles.com/testscores/

  6. The whole thing is ridiculous; an obvious leftist attempt to prove…something… negative about Republicans. Not worth caring about, except for the size of the readership of USA Today, many of whom might think it important. (On the other hand, what do they know about grammar, anyway?)

    I might point out — as a student of literature — that many of the best, most elegant, most creative writers use incomplete sentences, odd word combinations, and “ungrammatical” prose all the time — for effect, nuance, and impact. Creativity with the English language — at least among professional writers — is no measure of intelligence. Quite the contrary.

    USA Today should run a similar article about grammar “mistakes” in debates, wherein fast talking is required, and respondents don’t have the time to think through their answers — either for grammar, or sometimes (as important) the metaphorical analyses (later) of their answers. (Unless, for example, one is Hillary Clinton, who has 10 stock answers to everything, and has repeated them hundreds of times so they just roll of the (ugh) tongue…)

  7. Chiming in late as usual, but there’s another answer to how this Grammarly “research” got into USA Today: they paid for the space. It’s basically just an ad (with a bonus anti-Republican slant).

    It’s pretty much routine these days, even in “bigtime” venues. The institution I work for has placed many paid/solicited articles (skillfully written by one of my coworkers) in major papers like the Seattle Times and (authored by yours truly) in humbler venues like regional business journals and local papers. We recently entered into a partnership in which a Seattle-area TV station has agreed to use our in-house experts as its go-to sources for authoritative opinions whenever possible (I think there’s some quota, like 25% of newscasts will feature at least a positive mention of our people/research/resources/expertise in some way).

    Fortunately the org I work for is politically neutral (at least theoretically) and quite benign, but it gave me a new perspective on the public information/news racket. It really is a racket.

      • You better believe it. They’re all over the place.

        I think the long-term partnerships are worse, though. The top brass at my workplace said they had good assurances from their TV news partner that these partnerships were routine and were managed so as to respect the news media’s public integrity. I’m not convinced.

        How can an agreement like this *not* slant the news? And my workplace is actually late to the game. Even if every partnership is well-intentioned (and I strongly believe my workplace supports the public good), it’s still a constant force, a current pushing everything ever so slightly in a particular direction and providing hidden incentives to ignore or downplay or play up all sorts of seemingly insignificant little things… They add up until they’re like the Mississippi River straining against the levees.

        I guess it wouldn’t bother me so much if the journalism industry hadn’t gone to such lengths to cloak itself in false objectivity. They’ve been trading on Edward R. Murrow’s integrity for far too long.

        • I should be clear: it’s not commercial airtime, it’s NEWS airtime. Bought and paid for. Not only do they put our experts’ analysis and opinions and research on air, we feed them stories. The exact circumstances are always in flux, and they control the timing, but even in the worst possible news cycle we’ll be featured in a positive light in their newscasts somehow, no matter what.

          Isn’t that great? Not skeevy at all. (I’ve registered my opinion, but I’m pretty low on the totem pole…and hey, lighten up, you libertarian nut, we’re the good guys!)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.