.

Human Poetry Versus ChatGPT’s AI Poetry

Can you tell the difference between artificial intelligence
poetry and human poetry?

by Evan Mantyk and Mike Bryant

Artificial intelligence, most notably ChatGPT, is gaining a lot of attention for creating works of writing that seem effective and even human-like to a scary degree. But can ChaptGPT write poetry? Some people claim it can. Google’s AI chatbot is even called Bard. Is there a difference? Take the poetry quiz below and decide for yourself.

The quiz features poems that have appeared here at the Society of Classical Poets, a non-profit organization based in New York that promotes traditional poetry with meter and often rhyme. This creates a perfect comparison because these are the kinds of poems that ChatGPT normally writes. (Perhaps this is because free verse poetry is so formless that it is arguably not poetry at all but eccentric prose, but that’s a different story…)

For each question, write down the letter of the poem that you think is human-written. There is only one human-written poem in each question. You can check your answers at the very end and share your thoughts in the comments below.

.

QUESTION 1

.

Villanelle A

The sun sets slowly in the western sky,
Its golden light fades into shades of red,
A peaceful calm descends as day goes by.

The trees sway gently as the breeze drifts nigh,
Birds fly home to their cozy feathered bed,
The sun sets slowly in the western sky.

The world around us seems to slowly die,
As darkness creeps in and the stars are shed,
A peaceful calm descends as day goes by.

The moon arises, casting shadows sly,
And the night creatures emerge from their shed,
The sun sets slowly in the western sky.

The universe seems to heave a deep sigh,
As the earth prepares to rest its weary head,
A peaceful calm descends as day goes by.

So let us savor this moment nigh,
For tomorrow is another day ahead,
The sun sets slowly in the western sky,
A peaceful calm descends as day goes by.

.

Villanelle B

Our paths vanish; our footsteps fade
From forest glades, from wave-damp sands
From sunlight’s dazzle and twilight’s shade.

Babes and toddlers, not yet afraid,
We reach and roll and often stumble
From sunlight’s dazzle to twilight’s shade.

In youth we lie ‘neath moonlight’s blade,
Cut by love; like kisses stolen,
Our paths vanish; our footsteps fade.

Mothers, fathers, we make the trade—
Our days, our tears, our blood for children—
From sunlight’s dazzle to twilight’s shade.

At waves’ last reach, a foamy braid,
Elders converse, words a murmur—
Our paths vanish; our footsteps fade

Fingers stiffen; our hands grow cold;
Voices soften from steel to gold.
Our paths vanish; our footsteps fade
From sunlight’s dazzle to twilight’s shade.

.

.

QUESTION 2

.

Limerick A

An acrobat named Larry Loops
was best among all circus troupes,
But he quit one day
and was heard to say,
“My boss made me jump all those hoops!”

.

Limerick B

There once was a fellow named Clyde,
Who attempted to surf on the tide.
He fell off his board,
But would not be ignored,
So he swam with the waves, undenied.

.

Limerick C

There once was a cat from Peru
Whose antics were quite a to-do
He’d dance on his paws
And break all the laws
While singing a jolly “meow-dee-doo!”

.

.

QUESTION 3

.

Haiku A

Whispering wind speaks,
Nature’s wisdom flows through leaves,
Silent songs of peace.

.

Haiku B

Whispering breeze sighs,
Cherry blossoms gently fall,
Spring’s sweet lullaby.

.

Haiku C

Footsteps approaching;
calling birds fall silent: sh-h!
The willows whisper.

.

.

QUESTION 4

.

Sonnet A

My true love laughs, and angel choirs sing;
The mountains echo back his tuneful mirth
While Nature dances, heav’n in step with earth.
‘Tis music to mine ear; mine heart takes wing!
Into my love’s embrace I fly secure,
And tenderly he holds me in his arms
As if to shelter me from sin’s alarms.
Like rooted rock his faithfulness is sure.
If I could sing, my song would rise on high
As counterpoint to laughter’s melody.
My thanks and praise emblazoned in the sky
To God, who gave mine own true love to me.
And so, with heart and mind and strength and soul
I sing of thee, whose love has made me whole.

.

Sonnet B

Upon the verdant fields of waking spring,
Where flowers bloom and gentle zephyrs sing,
I find the solace of a quiet morn,
When thoughts of love and dreams are newly born.
The golden sun in sky doth proudly rise,
Its warm embrace a blessing undisguised,
Each ray of light a painter’s subtle brush,
That brings to life the colors nature hushed.
Yet in this tapestry of life and light,
A beauty far more radiant takes flight,
The glowing ember of your gentle smile,
A beacon in the twilight of my trials.
So let us journey hand in hand, my love,
Beneath the boundless canopy above.

.

.

.

Source of ChatGPT poems can be found here.

Answers

View answers here.

Post your score and thoughts in the comments section below.


NOTE TO READERS: If you enjoyed this poem or other content, please consider making a donation to the Society of Classical Poets.

The Society of Classical Poets does not endorse any views expressed in individual poems or commentary.


CODEC Stories:

28 Responses

  1. jd

    I agree. Was able to choose only one correctly and that was B for the first. I felt it sounded more human. After that, I gave up.

    Reply
  2. ABB

    I have tried to get Chat GPT to write poetry and I find the outcome stilted and old-fashioned, with simple diction that gives the poem a hackneyed feel.

    Villanelle B features more unusual diction than ‘A’ does, like “wave-damp.” In Villanelle A, th line “So let us savor this moment nigh,” is antiquated and awkward.

    In Limerick A, ‘troupes’ is a slightly unusual word, and it doesn’t start with ‘There once was…’ which is kind of cliché.

    In Haiku C, ‘sh-h’ is, again, unusual diction.

    Sonnet B is less old-fashioned sounding than Sonnet A. The line from ‘A,’ Like rooted rock his faithfulness is sure,” need a definite article before ‘rooted’ and is, again, an awkward construction.

    Based on these hunches, my guess is that Villanelle B, Limerick A, Haiku C, and Sonnet B are human.

    Of course, this assumes that the human writers were not either mediocre poets or intentionally trying to seem like a simplistic AI program.

    I have not looked at the answer key.

    Reply
    • ABB

      Well, got 3/4 right. Not only did I get the last one wrong, I inadvertently insulted James Tweedie in the process!

      Reply
      • Mike Bryant

        I don’t think you insulted James at all. You assumed that the more old fashioned the poem, the more likely it would be produced by AI. You couldn’t have known that James was purposely going for that old fashioned feel. Excellent reasoning and good guesses.

      • James A. Tweedie

        No insult taken, ABB. If you go to the original posting you will see that my sonnet was one of a set of three that generated some marvelous comments and suggestions. Not to mention that the AI sonnet appears to be a worthy opponent.

  3. Mary Gardner

    My goodness, I got all three wrong. I don’t know whether to praise ChatGPT or fear it.

    Reply
  4. Monika Cooper

    I got most right but one of the limericks fooled me. Nice try AI.

    The AI poems don’t carry the simultaneous continuity and complexity of thought that the real poems do. When it came to the haiku, one had a crispness to it that was lacking in the others. And that’s how I guessed it was the real one.

    Reply
  5. ABB

    Had two other people take it. They guessed right on the villanelle and haiku, but thought Limerick ‘C’ and (like me) Sonnet ‘B’ were human. All of us thought the sonnets were hardest to distinguish.

    For the next test, lets see how AI does writing an epic poem!

    Reply
  6. Joshua C. Frank

    I guessed all of them right. ChatGPT is a glorified Speak and Spell and writes poetry accordingly.

    Snakeskin published a poem of mine in the April 2023 issue along with a ChatGPT sonnet: https://snakeskinpoetry.wordpress.com/2023/03/31/thoughts-of-a-robot/

    Also, it turns out that ChapGPT is staunchly leftist: https://www.onelowerlight.com/writing/my-first-attempt-to-write-a-short-story-with-chat-gpt/

    (Not that leftist humans are much better at disobeying the culture’s programming…)

    Reply
  7. Robert Zimmerman

    I guessed right on the Limerick, Villanelle, and Sonnet. I missed the Haiku. There were slight usages that drew my attention and swayed my choices. I had no clue with the Haiku. To me, AI is binary but the world is analog. Good poetry is analog.

    Reply
  8. Mark Stellinga

    Verifying the pieces we share were people-generated is not difficult when they appear in published books from years past, thank God. Will AI realize, or even care, if and when it plagiarizes? I doubt it. Got 2, missed 2…

    Reply
  9. Paul Freeman

    3/4 – the haiku I got wrong, but then I don’t write many haikus (he says having written 3 a few hours ago) and the other forms were meatier. With the limericks, the last words were the clinchers to which were AI for me.

    The villanelle and sonnet that were AI were lacking in some areas – vocabulary-wise, grammar-wise and feeling-wise.

    I recently won a small prose competition, writing a 100-word short story on the topic ‘out the airlock’. Afterwards, as an experiment, I typed the parameters into ChatGPT and a decent story came out the other end (a bit generic, though) which I believe could have been up there vying for the first prize if it had been submitted.

    Reply
  10. Yael

    Very cool poetry challenge! This is scary creepy indeed.
    Going into this I figured that a human poem would have more depth of thought, a more distinctly human perspective on life, and overall better and more coherent presentation of imagery than AI. Based on this I got the first 3 correct, but failed number 4, which was a really tough call for me. God is mentioned in Sonnet A, and I had my doubts that AI would mention God, so at first I thought that A must be human. But after I read each Sonnet again, I thought that in comparison A lacks in pace and overall organization of thoughts, and it doesn’t flow as well as B. I thought the imagery in Sonnet B was superior, it has a more pleasing pace to me and looks better organised, although line 5 is highly problematic. So it was kind of a toss-up for me, and I picked B because I just liked it better than A. This might re-enforce the notion that first impressions are often the most accurate.

    Reply
  11. James A. Tweedie

    I found the villanelles to be a tossup and, in the end, chose incorrectly.

    The human villanelle fails to rhyme the “b” line throughout, ending the line sequence with “sands” “stumble” “stolen” “children” and “murmur” while the final quatrain scans as two consecutive couplets instead of the expected abaa (which wouldn’t matter, of course, since there is no “b” rhyme pattern to be mirrored anyway).

    On the other hand, the AI villanelle follows the normative form perfectly which is why I chose it.

    The rest I guessed correctly even though the AI sonnet flows relatively well in both thought and form, despite having imperfect rhymes in three of its seven couplets.

    Neither sonnet is Shakespearean, however, which adds to the confusion, mine being abba cddc effe gg and the AI being aabb ccdd eeff gg with, I think, my unusual pattern would be less likely to be produced by an AI generator.

    Also, I’m not exactly sure this generator necessarily qualifies as AI since similar constructs have been around for a while. I even have a bound flip-card “Shakespeare Insult Generator” where different combinations of Shakespearean verbs, nouns and adjectives can be made to generate phrases like, “Mammering, rug-headed fleshmonger” or “Lisping, dull-brained foot-licker.”

    This poem generator is far less sophisticated than the AIs that gather data and reshape it into a cogent essay or compelling short story and far less so than the programs play high-level chess where they have to process a response to a human move that allows for thousands of possible responses where only a mere handful avoid disaster and even fewer maintain or improve its position on the board.

    In any case, the world’s-a-changing faster than we can process the change. The world my grandchildren will experience with be very different than anything we have experienced in the past. During the Apollo 11 moon expedition I stood looking at the moon alongside an elderly mother of an older friend and asked how she felt about the fact that there were men walking around on the moon’s surface at that very moment. She, who had been born before the Wright brother’s first flight, said she didn’t believe it, that it was not possible.

    I think we are like that. How AI will impact the world in the future is beyond our imagining. I have no doubt, however, that it will be for good, for ill, for both, and for worse.

    Reply
  12. The Society

    Dear SCP Readers,

    Thank you for taking this quiz. In hindsight, it is highly favored toward ChatGPT’s poetry in the first place since I started with its AI poems and then I went and tried to match them in form and theme. If we worked in the other direction, starting first with an extraordinary human-written poem that we, living human beings that we are, chose and said “AI, try to beat that!” then I have no doubt that it would be easy to tell the AI from the human. Just try taking Susan’s Earth Day pantoum “Green… Greener… Greenest…” and put it against ChatGPT’s Earth Day pantoum (this is Susan’s pantoum https://classicalpoets.org/2023/04/22/a-poem-for-earth-day-green-greener-greenest-by-susan-jarvis-bryant/).

    Thank you all for your input!

    -Evan, SCP Editor

    Reply
  13. Roy Eugene Peterson

    3 of 4. I missed on the Limerick. I can just see the critiques now in the Comments: “This poem reads like it was artificially written.”

    Reply
  14. Damian Robin

    Thanks the fun with a serious edge, Evan and Mike. You dignified the ChatGBT by letting it lead. But traditionalists need to be dignified and uphold courtesy.

    I got three real person ones. The sonnet I flunked as I found ‘God’ to be a red herring (sorry God). I thought the AI, coming from the left field, would filter out ‘God’ and religious language. Then it occurred to me that if it used the whole back catalogue of English poetry, where divine things are abundant, spiritual-sounding expressions would slip in.

    But I still went away from the archaic language without looking at the rhyme scheme that may have pulled me back to the first if I’d been as rigourous as Jim Tweedie. He says, above, that rhyme form on the first sonnet, his, is less likely to be produced by an AI generator. Making sense while following standard rhyme schemes may be too complex for a bot.

    Is this and the anomalies in the villanelle, that Mr Tweedie diligently points out, a cautionary tale to us, as standard bearers of tradition, to tighten up. Maybe we should, like I have seen in an academic essay defining the sonnet, eject from under the umbrella of sonnets all poems that use couplets all the way through. Evan and Mike saw it as legitimate sonnet but maybe we could be less forgiving in the future. Fine poems are made with seven rhyming couples, but we could give them another name than an unqualified ‘sonnet’.

    Reply
  15. James Sale

    I am certainly not going to take the quiz because it would take me hours to try to fathom it; the thing is, As Evan points out, it’s matching like with like. If we think about the four forms used, what do we find? First, the haiku is an extremely simplistic form, which is why primary school teachers are so fond of it. It should not surprise us, then, if AI can make a passing go of it. Second, and opposite, the limerick is so tightly packed as a form, the same thing applies – given an opening line … de-dah from Devizes and something about ‘sizes’ and we’re all home. The villanelle is a wonderful form which is capable of great poetry, as we see on the SCP and in Dylan Thomas. But here’s the thing: it’s an ideal form for AI to imitate because of the repeating lines. AI specialises in repeat code. This leaves only the fourth form – the sonnet – as really a challenge for AI. But here of course it can draw on the millions that have been written in its database; plus too, it is a closed form – ideal again for a machine.

    So what we need is a real challenge for ChatGPT. And I have given it one:

    “Create a narrative poem about Shakespeare’s Romeo and Juliet in the terza rima form of Dante’s The Divine Comedy”

    In fair Verona, where we lay our scene,
    Two households both alike in dignity,
    An ancient grudge, new mutiny between.

    But love did bloom, amidst this enmity,
    A son of Montague, Romeo by name,
    And Capulet’s daughter, Juliet, lovely.

    Their love was true, but fate was not the same,
    For feuding families brought them to despair,
    And tragedy struck, a deadly game.

    Young Romeo, in sorrow and despair,
    Believed his love was lost forevermore,
    And drank a potion, in his heart to bear.

    But when his Juliet awoke once more,
    And found her love had perished by her side,
    She took her own life, her heart to pour.

    Thus ended their love, so pure and wide,
    A tale of woe, that still doth resonate,
    In Dante’s terza rima, it doth abide.

    Now you can see clearly – at least I can – how good (or more accurately, bad) ChatGPT really is. Enjoy.

    Reply
    • The Society

      Thank you, James. The problem here is whether some 12-year-old tries to pass this off as his own work. In that case, the teacher might really be in a pickle, but they will quickly learn how to avoid it. Now, we need someone to write a real good one with the same parameters to shame this one into obsolescence.

      Reply
      • James Sale

        Ha ha ha – Evan – I’d like to but at the moment I am working on my terza rima version of Paradiso, so can’t stop! But I understand an extract from Canto 2 will be appearing on these pages on the 6th May, so for those wishing to taste terza rima Paradise, head that way!!!

  16. Monika Cooper

    A terza rima extract from Canto 2! That will be lovely.

    Reply
  17. Peirithus

    That was scary, I did only the first quiz, and I failed! Smash the machines.

    Reply
  18. Peirithus

    I did the second quiz and got it right.
    The first poem was much more refined and beautiful.
    Smash the machines.

    It makes me feel that the AI is some kind of living, evil, alien mind.

    Reply
  19. Maxim

    I noticed straight away that only 1 poem ever had abbreviations and/or a proliferation of hyphenations. The Limericks were the toughest ones to discern to me. I quickly ruled out be as it sounded far too quaint to be a modern work, and spent some time on A and C, but eventually ruling out C as too traditional of a limerick.

    Reply

Leave a Reply to Mark Stellinga Cancel Reply

Your email address will not be published.

Captcha loading...

This site uses Akismet to reduce spam. Learn how your comment data is processed.