Comparing the First and Last Books of 20 Authors

By Julian Yanover

How do the first books of acclaimed writers, published in their youth, compare to their last works released 20, 40, or even 60 years later?

We wanted to find out if an author’s language saw changes in the lexicon they used over time. This meant we had to find a way to measure the quantity and variety of vocabulary in relation to the full content of the book.

Does vocabulary become enriched due to experience? Or does it decrease?

With these initial questions, we began our research using artificial intelligence and natural language processing software, new tools that make this previously unthinkable task possible.

This is what we found.

What We Analyzed

• 20 authors.
• 40 books.

We selected 20 authors, taking one of their earliest works and one of their last published works.

Thus, for instance, we examined Doris Lessing‘s “The Grass Is Singing” (1950) and “Ben, in the World” (2000), Ray Bradbury‘s “Fahrenheit 451” (1953) and “Farewell Summer” (2006), and Virginia Woolf‘s “The Voyage Out” (1915) and “Between the Acts” (1941).

It’s important to note that we always sought the longest time gap between books that we could find.

How We Processed It

• Database with 3,509,555 words.
• Natural language processing (NLP) software and artificial intelligence to identify and unify similar words through lemmatization.
• Our own scoring system to standardize criteria and evaluate lexical diversity.

The books were digitally stored, word by word, in a database.

We performed lemmatization of each word, which involves reducing all the variants that a word might have to obtain its base lemma. For instance, buy, buys and bought are grouped under the lemma buy. This technique is a tool of natural language processing that ensures the final result is more accurate by only counting lemmas and not all the variations of a word.

A glimpse at the database with over 3 million stored and processed words

The first problem we faced was that, in longer books, there generally were more unique words, but a smaller percentage of these compared to the total. It’s natural: in a book of 100,000 words, we’ll encounter more terms, but at the same time, there’ll be more repetition than in a 5,000 words book.

We found the way to solve this in collaboration with artificial intelligence.

The AI presented us with some possible methodologies, like dividing the number of unique lemmas by the square root of the total words in the book or similarly dividing it by its logarithm. However, although the calculation was less influenced by the book’s length, it was still affected by the total number of words in each publication.

We also considered comparing only samples of the same size from each text, but this left in some cases 90% of the book out of the analysis.

Finally, we took the sliding window approach. We developed a script that analyzed each book’s language diversity in segments of 1,000 words and took the average variety of lemmas as the result score for each work. This way, the entire book was included in the analysis, and the factors that could skew the final figures were minimized.

In this manner, we obtained a lexical diversity score for each book, on a scale of 1 to 100, where a higher score indicates a broader use of language.

The Case of Agatha Christie

The case of Agatha Christie deserves special mention.

This detective novel author made headlines in 2009 when a group of researchers suggested, after analyzing several of her works and noting how her vocabulary narrowed in her books as she aged, that she must have suffered from Alzheimer’s.

This news was the inspiration for this article.

It also provided an opportunity to run the code we created on two of her books and see for ourselves whether we found any noteworthy results, and if they aligned with the findings of those researchers.

We analyzed “The Mysterious Affair at Styles“, her first Hercules Poirot novel published in 1920 when she was 30 years old, and “Elephants Can Remember” from 1972, also featuring detective Poirot, when she was 82.

While “The Mysterious Affair at Styles” achieved a lexical diversity score of 84.9, “Elephants Can Remember“, which was published 52 years later, only managed a score of 67.4, showing an extremely wide difference of 17.5 points between them.

As you will see in the conclusions below, within our study of 20 authors, the next largest difference recorded between a single writer’s works were 13 points, and it was an increase in Margaret Drabble’s language variety over the years, not a decrease like in Christie’s case.

Increased Lexical Diversity Over Time

Margaret Drabble stands out with the most significant increase in lexical diversity. “The Millstone“, published when she was just 26, had a score of 78.7. Years later, “The Sea Lady“, written at the age of 67, exhibited a substantial leap to 92.4, suggesting a significant broadening of her linguistic canvas over time.

Beryl Bainbridge also showed an upward trajectory in lexical diversity. Her early work “The Dressmaker“, published when she was 41 years old, had a lexical diversity score of 87.3. In contrast, “The Girl in the Polka Dot Dress“, published posthumously, registered a notable increase to 96.7, demonstrating a more varied use of language in her later years.

Similarly, Mary Shelley, the famed author of “Frankenstein“, showcased a remarkable growth in her lexical repertoire. Her debut novel, written at the young age of 21, had a lexical diversity score of 90.3. This score rose to an impressive 93.7 in her later novel “Falkner“, reflecting a more complex use of language as she matured.

Decreased Lexical Diversity Over Time

On the flip side, some authors saw a decline in lexical diversity as they aged.

Doris Lessing‘s “The Grass Is Singing“, written at 31, had a lexical diversity of 84.2, which fell to 73 when “Ben, in the World” was published as she was turning 81. This suggests a narrowing in the variety of her word choices in her latter years.

Kazuo Ishiguro also experienced a decline in lexical diversity over his career. His early work “A Pale View of Hills” scored 76.9 at age 28, but by “Never Let Me Go,” published when he was 51, his score had dipped to 72.9, hinting at a more restrained use of language in his more mature work.

Alice Munro, a master of the short story, exhibited a decline in lexical diversity between her early and later works. Her collection “Dance of the Happy Shades“, published when she was 37, had a lexical diversity score of 83.3. However, by the time “Dear Life” was published in 2012, when Munro was 81, the score had decreased to 79.4. While this decline is relatively modest, it could suggest a honing of her narrative voice and a possible shift towards a more concise and distilled mode of storytelling in her later years.

Up next, we display the full graph with all the authors researched.

Comparative Chart

Full table

Final thoughts

This analysis is not to suggest that a decrease in lexical diversity diminishes the quality of work—on the contrary, it may indicate a more focused and refined literary voice. Similarly, an increase does not necessarily equate to improved writing, but rather a diversification in language use. The shifts in lexical diversity are but one aspect of an author's evolving craft, reflecting changes in creative priorities, experiences, and possibly the influence of the times in which they write.

About

Julian Yanover has been a web developer for over 20 years and is the director of MyPoeticSide.com

MyPoeticSide.com covers all topics related to poetry and literature and also hosts a community of poets who share their work on the platform every day.

With more than 15 years of work, MyPoeticSide.com continues to expand its content to provide its users with objective and quality information. Contact us.