We compared 25 works on World Book Day and English Language Day

By Julian Yanover

On April 23, commemorating the death of William Shakespeare, both World Book Day and English Language Day are celebrated. We combined these two celebrations in this analysis, which reviews the vocabulary of 25 books to identify the most complex and accessible texts for readers.

With this goal in mind, we used artificial intelligence and natural language processing software to dissect each work and obtain answers.

This is what we found.

What We Analyzed

We’ve curated a list of 25 of the most acclaimed English-language books, striving for a mix that showcases diversity in terms of time periods and stylistic approaches.

How We Processed It

• Database with 4,493,150 words.
• Natural language processing (NLP) software and artificial intelligence to identify and unify similar words through lemmatization.
• Our own scoring system to standardize criteria and evaluate lexical diversity.

The books were digitally stored, word by word, in a database.

We performed lemmatization of each word, which involves reducing all the variants that a word might have to obtain its base lemma. For instance, buy, buys and bought are grouped under the lemma buy. This technique is a tool of natural language processing that ensures the final result is more accurate by only counting lemmas and not all the variations of a word.

A glimpse at the database with over 4 million stored and processed words

The first problem we faced was that, in longer books, there generally were more unique words, but a smaller percentage of these compared to the total. It’s natural: in a book of 100,000 words, we’ll encounter more terms, but at the same time, there’ll be more repetition than in a 5,000 words book.

We found the way to solve this in collaboration with artificial intelligence.

The AI presented us with some possible methodologies, like dividing the number of unique lemmas by the square root of the total words in the book or similarly dividing it by its logarithm. However, although the calculation was less influenced by the book’s length, it was still affected by the total number of words in each publication.

We also considered comparing only samples of the same size from each text, but this left in some cases 90% of the book out of the analysis.

Finally, we took the sliding window approach. We developed a script that analyzed each book’s language diversity in segments of 1,000 words and took the average variety of lemmas as the result score for each work. This way, the entire book was included in the analysis, and the factors that could skew the final figures were minimized.

In this manner, we obtained a lexical diversity score for each book, on a scale of 1 to 100, where a higher score indicates a broader use of language.

Up next, we display the full results.

Ranking

Key takeaways

The top three scorers in lexical diversity are all works by female authors: Beryl Bainbridge, Edith Wharton, and Emily Brontë. This might suggest a particularly rich use of language among female authors in different periods, highlighting their contribution to the literary landscape with a dense and varied vocabulary.

Also, despite the varying lengths of their narratives, from Bainbridge's relatively short "The Girl in the Polka Dot Dress" to Wharton's lengthier "The Valley of Decision," these authors manage to maintain a high level of lexical diversity. This showcases their skill in crafting detailed and complex narratives without diluting the richness of their language.

Moreover, the diversity in publication dates among the top-scoring novels suggests that lexical richness is not confined to a specific era. For instance, "Wuthering Heights" was published in the 19th century, while "The Girl in the Polka Dot Dress" is a more contemporary work. This cross-era representation underscores the timeless value of a rich vocabulary in storytelling.

For a beginner reader, it might be a good option to start with the books at the bottom of the ranking table and gradually move up, slowly making the reading more complex.

Interactive table

Click on any column header to arrange the data in the order you choose.

About

Julian Yanover has been a web developer for over 20 years and is the CEO of MyPoeticSide.com

MyPoeticSide.com covers all topics related to poetry and literature and also hosts a community of poets who share their work on the platform every day.

With more than 15 years of work, MyPoeticSide.com continues to expand its content to provide its users with objective and quality information. Contact us.