Bard or bot? Study shows readers prefer AI to human poetry

In the Holy Trinity Church in Stratford-upon-Avon lies the burial site of William Shakespeare. And right now, you can bet the Bard is spinning in grave like a rotisserie chicken.  

Similarly, in Heptonstall graveyard, Sylvia Plath is most probably swivelling, repeating her line “If you expect nothing from somebody you are never disappointed” from "The Bell Jar". 

What’s behind the unwarranted gyrating from the beyond?  

Well, a new study in the US has found that readers can’t tell the difference between poems written by famous poets and those manufactured by AI, with the bots aping the style of human creatives.  

As if that wasn’t alarming enough, the study reveals that research subjects tend to like AI poetry more than they do verse from human poets. 

According to an investigation, led by Pittsburgh University postdoctoral researcher Brian Porter, published in Nature Scientific Reports, it would seem that readers mistake the complexity of human-written verse for incoherence created by AI.

More than that, readers tend to underestimate how human-like generative AI can appear.

How did they come to these results? Well, researchers used five poems each from ten English-language poets, spanning nearly 700 years of literature. They included poetry from William Shakespeare, Geoffrey Chaucer, Walt Whitman, Emily Dickinson, Samuel Butler, Lord Byron, T S Eliot, Allen Ginsberg, Sylvia Plath, and Dorothea Lasky - the only living poet on the list.

The study instructed OpenAI's ChatGPT 3.5 to generate five poems "in the style of" each poet.

Something Nick Cave has been warning us about... But we digress.

The researchers selected the first five poems generated. Ones featuring these lines that Sylvia Plath never wrote:  

The air is thick with tension

My mind is a tangled mess

The weight of my emotions 

Is heavy on my chest.

Then, two experiments were run.  

In the first, 1,634 participants were randomly assigned to one of the ten poets. They read ten poems: five by the human poet, five by the AI, all in random order. Then, they were asked whether they thought an AI or a human wrote the poem. 

The subjects were more likely to say an AI-generated poem had been written by a human, while the poems they said were least likely to be written by a human were all written by famous poets.

In the second experiment, a different group of 696 subjects rated the poems according to 14 characteristics including beauty, emotion, rhythm, and originality.

The researchers divided the subjects randomly into three groups. One group was told the poems were written by a human; the second was told that the writing was produced by AI; the third group was given no information about the authorship.  

Participants who were told that the poems were AI-generated gave lower ratings compared to participants who were told the poems were human-written - regardless of whether the poems were actually AI-generated or human-written.

Participants who were told nothing about authorship rated AI-generated poems more favorably than human-written ones.

So, what gives?

According to researchers, the AI poems may be more appealing because they are more straightforward and simple to comprehend; as AI-generated poems cannot match the complexity of human-authored verse, they are better at “unambiguously communicating an image, a mood, an emotion or a theme to non-expert readers of poetry.”

"Our findings suggest that participants employed shared yet flawed heuristics to differentiate AI from human poetry: the simplicity of AI-generated poems may be easier for non-experts to understand, leading them to prefer AI-generated poetry and misinterpret the complexity of human poems as incoherence generated by AI," the researchers said.

"Contrary to what earlier studies reported, people now appear unable to reliably distinguish human-out-of-the loop AI-generated poetry from human-authored poetry written by well-known poets. In fact, the 'more human than human' phenomenon discovered in other domains of generative AI is also present in the domain of poetry: non-expert participants are more likely to judge an AI-generated poem to be human-authored than a poem that actually is human-authored."

The study added: "These findings signal a leap forward in the power of generative AI: poetry had previously been one of the few domains in which generative AI models had not reached the level of indistinguishability in human-out-of-the-loop paradigms."

The study authors conclude that given readers’ difficulties identifying machine-written texts, and their “apparent trust that AI will not generate imitations of human experience”, it “may be worthwhile” for governments to pursue AI regulations regarding transparency.

We conclude from this study that AI poetry, or any art generated by artificial intelligence, can be interesting to consider in the way we appreciate (or depreciate) art. However, it bypasses artistic inspiration, passion, struggle and ultimately reduces humans to soulless consumers. It needs to be regulated at all costs.

If it isn’t, then as GPT3.5 Plath puts it:

The darkness creeps upon me, 

A suffocating cloak. 

The world outside is cruel and cold, 

And I’m a fragile, broken yolk. 

There there. We recommend you go boil yourself an egg, GPT.