One regrettable feature of the presidential campaign season is the inevitable ranking of candidate speeches by reading level. And so, after the first Republican debate, Politico’s Jack Shafer announced with glee that Donald Trump “talks like a third-grader,” while Ted Cruz’s language sounds more ninth grade. However, anyone who spends time around schoolchildren will tell you neither claim is true.
Shafer calculates candidates’ grade level by running transcripts of their debate responses through a formula known as the Flesch-Kincaid reading scale. Whether or not you agree that Trump is simple and Cruz, high-brow—among Republican candidates, simple has proven to be a vote-getter and high-brow is a dirty word—the fact is that formulas like Flesch-Kincaid, designed to measure the understandability of writing, fail miserably at that task, and they’re completely useless for rating speech.
Some 200 different readability formulas have been created since the 1920s. The number alone should tell you that the search for a magic formula to turn prose into gold isn’t working. The best-known of these was developed by Rudolph Flesch in 1948. Flesch, long a fan of “plain English,” a term whose meaning is far from plain, decided that short sentences with words of few syllables were easiest to understand, so he came up with a complicated equation that would allow him to rank documents designed for general adult consumption—self-help books, government publications, magazine articles—on a 100-point scale, with a high score for the easiest to read, and a low score for difficult texts. In the 1970s, J. P. Kincaid revised the Flesch formula to rank prose according to U.S. grade level.
To give you an idea of how reading formulas works, or rather, how they don’t work, the paragraph above scores a 62 on the Flesch Reading Ease Scale, pretty much a perfect score from Flesch’s point of view (he believed the average adult American in 1948 read at seventh-grade level, and he felt eighth-grade prose was something worth aiming for), and it’s ranked at grade 8.6 by the Flesch-Kincaid equation, slightly above the target level. However, another popular scale, the SMOG Index, rates my paragraph as tenth-grade material, and the Gunning Fog scale puts it at fourteenth-grade, or very difficult indeed. So the formulas disagree, which suggests that reading formulas are rubbish.
But wait, there’s more. Encoding that same paragraph with a common cipher that replaces A with Z, B with Y, and so on, yields a Flesch score of 87.8, very, very easy to read, even though ciphering doesn't change word and sentence length. The other formulas agree that the coded text is easier to read than the unencoded original. Both Flesch-Kincaid and the SMOG Index give the cipher a ranking of fifth grade, and even Gunning Fog finds it dumbed down from fourteenth to tenth grade. To risk stating the obvious, the formulas treat content as irrelevant, although any reader would have to decipher the text before they could understand it, not just a fifth-grader, but anyone at all.
Here’s the cipher version that none of the formulas recognize as not-English, yet they all call it easy to read, so you can judge for yourself how readable it is (HT, Mark Liberman, on ciphers and reading formulas; and on the meaningless formula numbers):
Hlnv 200 wruuvivmg ivzwzyrorgb ulinfozh szev yvvm xivzgvw hrmxv gsv 1920h. Gsv mfnyvi zolmv hslfow yv zm rmwrxzgrlm gszg dv’ev bvg gl urmw z ulinfoz, zmw kilyzyob gsv hvzixs uli lmv rh ylfmw gl uzro. Gsv yvhg-pmldm ulinfoz dzh wvevolkvw yb Ifwloks Uovhxs rm 1948. Uovhxs, olmt z uzm lu “kozrm Vmtorhs,” rghvou z gvin dslhv nvzmrmt rh uzi uiln kozrm, wvxrwvw gszg hslig hvmgvmxvh drgs dliwh lu uvd hboozyovh dviv vzhrvhg gl fmwvihgzmw, hl sv xznv fk drgs z xlnkorxzgvw vjfzgrlm gszg dlfow zoold srn gl izmp wlxfnvmgh wvhrtmvw uli tvmvizo zwfog xlmhfnkgrlm—hvou-svok yllph, tlevimnvmg kfyorxzgrlmh, nztzarmv zigrxovh—lm z 100-klrmg hxzov, drgs z srts hxliv uli gsv vzhrvhg gl ivzw, zmw z old hxliv uli wruurxfog gvcgh. Rm gsv 1970h, Q. K. Prmxzrw iverhvw gsv Uovhxs ulinfoz gl izmp kilhv zxxliwrmt gl F.H. tizwv ovevo, uiln vovnvmgzib hxsllo gsilfts xloovtv.
The cipher is far from plain Engish, but even for unmasked English prose, shorter sentences and words of fewer syllables don't necessarily make for clarity. Such forced oversimplification may leave out crucial information, and the text may fail to convey complex ideas effectively. Flesch may have tacitly recognized this, because he also paired his reading-level formula with another that purported to determine the interest quotient of a text (Vieth 1988). Even though no formula can predict a reader’s interest level, at least Flesch realized that motivation is a vital component of literacy: wanting to read and write really does help. I suppose a spy motivated to crack my cipher would be happy to discover it was just a simple replacement code, easy to read after all, not a secret worthy of the Enigma machine.
Even though readability can’t be reduced to a formula, many organizations rely on formulas like Flesch-Kincaid, including the U.S. government and the military. The guidelines for the Centers for Medicare and Medicaid Services warn government writers about the unreliability of Flesch and friends:
- Readability formulas ignore most factors that contribute to ease of reading and comprehension.
- Grade level scores tend to be unreliable.
- Grade level scores are less precise than they sound and prone to misinterpretation.
- Imposing a grade level requirement has the potential to do harm.
Then, ignoring these warnings like smokers ignore the Surgeon General, the guidelines present detailed information on how to use the formulas they have just debunked.
Here’s another example where government writing guidelines embrace the notion of readability level while seeming to reject one-size-fits-all formulas: the Federal Plain Language Guidelines advise pitching the difficulty of a text to the audience’s expectation in terms of grade levels:
Don’t write for an 8th grade class if your audience is composed of PhD candidates, small business owners, working parents, or immigrants. Only write for 8th graders if your audience is, in fact, an 8th grade class.
The National Institutes of Health states flat out that writers should run their prose through a readability formula, leaving to the writer the choice of which dumb-down machine to use:
Keep within a range of about a 6th to 7th grade reading level. See reading level calculator information below [links to Flesch, Fry, SMOG, MS Word, and other readability apps]. This is especially important in the first few lines of text. If the reader has difficulty at the beginning, they may stop reading. [An interesting use of singular they.]
Unfortunately, no formula can predict whether an eighth grader or a doctoral student will understand a text, or even guarantee that the intended audience will want to read it.
No one really expects government writing advice to be useful or even consistent, as we see in this example where military writers are told with no sense of irony to “use short, simple words” in the Defense Department’s “Writing Style Guide for DoD Issuances.”
The style guide was issuanced (if your document title has the noun issuances in it, you know the DoD won’t be satisfied with a short, simple verb like issued) in response to the Plain Writing Act of 2010, which mandates that all government agencies “shall use plain writing in every covered document of the agency that the agency issues or substantially revises.” Like DoD issuances, the Plain Writing Act hardly follows its own mandate “to improve the effectiveness and accountability of Federal agencies to the public by promoting clear Government communication that the public can understand and use.” Instead, the law is written in the complex, technical legalese required of all statutes. Take this example, from Section 6 of the statute:
(a) JUDICIAL—There shall be no judicial review of compliance or noncompliance with any provision of this Act.
(b) ENFORCEABILITY.—No provision of this Act shall be construed to create any right or benefit, substantive or procedural, enforceable by any administrative or judicial action.
What this means, in plain English, is that government agencies can ignore the law because it has no teeth. According to the enforceability section of the law, the Plain Writing Act, like the Surgeon General's Warning, is unenforceable.
To be fair, the reading formulas all crunch the plain language law as “difficult to read” or “college level.” As for the grade level of Donald Trump’s language: I ran a short passage from Trump’s book, The Art of the Deal (2004), through Flesch-Kincaid and it came out at eighth grade, not third grade. But not only is the Flesch-Kincaid scale not designed for speech and invalid for writing, Trump’s book itself may not be representative of his writing, since he hired a ghost to help him write it. Plus, whatever Trump may or may not have written has no correlation to his debate performance, where his casual, off-the-cuff remarks may or may not have been carefully scripted and vigorously rehearsed, like the words of the other debate participants.
Perhaps the worst part of the presidential campaign season has nothing to do with applying readability formulas to the candidates' speeches, or their offhand comments, and everything to do with the fact that the election itself is over a year away, so commentators have to seize on anything they can to keep the news cycle going. During the run-up to the 2008 election, researchers applied gender stereotypes to the speech of Bill and Hillary Clinton and concluded that then-Senator Clinton “talks like a girl.” Other commentators actually found Barack Obama’s speech too professorial and not black enough to attract African American votes. And of course George W. Bush’s frequent language gaffes made him the butt of jokes for his critics, though the same studied lack of eloquence made him sound just like one of the boys to his supporters.
In the end, my objections to readability formulas won’t stop journalists from skewering the debate performance of politicians, whose language is always on display, always being poked and prodded for clues about their intelligence, their personality, their hidden agendas. It may even offer a clue as to how they’ll perform if elected. As the old radio ad for a vocabulary improvement course used to intone so ominously, “People judge you by the words you use.” There's just no formula you can apply or course you can buy to make them judge you the way you want to be judged.
