toki poems

2021-03-17

Computer-generated poems in the toki pona language.

Tags: creative data


I wrote a cute little program to generate poetry in toki pona, the 125-word language that was published by Sonja Lang in 2001.

Here's one of the sick rhymes generated by the program:

jan sama pi jan,
ike li pona ala,
tawa mi la tan.

tenpo ni e kala.

And an attempted translation by u/jedgrei from Reddit. I say attempted because this particular poem doesn't fully make sense.

Because I dislike the enemy of the sibling, now [fish?].

Before I explain how it all works, I should share some relevant facts about toki pona:

Now for an EXPLANATION. The first thing I needed was a bunch of toki pona text (a "corpus"), which I gathered by downloading comments from r/tokipona using a script that I happened to have lying around. Then I created a big ol' Markov chain based on that corpus, using another script that I happened to have lying around.

A Markov chain / graph where the nodes are toki pona words and the edges between the nodes show the probability of going from one word to another. There are 4 nodes: start, toki, pona and awen. An arrow from start to toki has the label 11; start to pona has 2; toki to pona has 5; toki to awen has 1; and pona to awen has 2.
A Markov chain with toki pona words!

So to generate text with this Markov chain graph thing, you randomly traverse the graph, beginning at the node called start, which represents the start of a sentence. The probability of going from one node to another is weighted by the number of times the words appear next to each other in the corpus. The weight is stored in the edge between the nodes. In this example, the word "toki" appeared at the start of 11 sentences, which means there's a probability of 11/(11+2) that you go from start to toki. Similarly, there's a probability of 2/(11+2) that you go from start to pona. Don't panic!, the probabilities add up to 1.

Here's a path we might follow through the Markov chain, resulting in the sentence "toki pona awen". It depends on which random numbers we pull from our hat, though. We could also end up with "toki awen" or "pona awen".

A path through the Markov chain from before.
A possible path through the Markov chain.

That's fine for normal text generation, but when you're generating poems, there are extra constraints on the output, such as the number of syllables per line. Or maybe the final word on a line should rhyme with a word from another line.

Here's how the algorithm works with the addition of constraints. Let's say we're generating a line in a poem using the Markov chain from before. We've followed the path starttokiawen. But we need the last word to rhyme with "sona", which "awen" doesn't.

A path in the Markov chain from start, to toki, to awen. Which violates the constraints, so we have to backtrack.
A path that violates the poem constraints.

We retrace our steps back to toki.

A path in the Markov chain from start to toki. We've backtracked from awen, because it doesn't rhyme.
The path post-backtracking.

Then we try picking pona as the next node, giving: starttokipona.

A path in the Markov chain from start, to toki, to pona.
Complete path that meets the poem constraints.

We can stop here, because "pona" happens to rhyme with "sona". We'd also have to backtrack if we exceeded the number of syllables allowed on the line.

That's pretty much how the poem generation works. A concise description would be: weighted random search through a graph, with backtracking.

Maybe, rather than throwing this poem generator on the trash heap of all the useless programs I've written, it'll be the basis for a Twitter or Mastodon bot. Stay tuned. In the meantime, here's the code, and below are a few more poems I've generated.

A limerick (rhyming scheme aabba).

a a mama mije ala pali wan,
wan wan seli e pilin pona pi jan.

lon li kalama en,
nasin ni li len,
noka nasa mute li pona e pan.

A Shakespearean sonnet (abab cdcd efef gg).

pona taso soweli sina ken,
ante la sijelo pi sitelen,
pona mute li nimi wan wan en,
nimi wan wan wan wan taso mi ken.

mani lili tan mi li kon li sin,
lon e kulupu tomo suli kin,
la sina lon pi sona pona kin,
la ona li pona e ma lukin.

luka en pilin telo seli sin,
e oko lili pi luka wan wan,
la nanpa lili ni la ona kin,
la lipu mute la mi ali jan.

pona ala tan ni li kepeken,
nasin sewi li suli li awen.

And a haiku (abc).

pilin ala ken,
ala mute o lukin,
e kon mun en tan.

Appendix A: all the rhymes

Here are all groups of rhyming words, according to my definition of a rhyme. My definition of a rhyme, by the way, is where the last syllables of two words are the same, regardless of whether they're stressed syllables.

There are 61 rhyming words, split between 21 groups. That leaves 64 words without a rhyme buddy :(

Appendix B: data spelunking

It would be a waste not to do anything with the text data from r/tokipona, so here are some quick plots.

Here's a word cloud. The bigger a word is, the more common it is on r/tokipona.

A word cloud where the biggest words are 'a', 'toki', 'pona', 'jan'.

Here's Zipf's law in action, in the form of a cumulative frequency distribution. It basically shows that the top 10 words make up about 50% of word occurrences, while the top 50 words make up almost 90%. This pattern shows up in all human languages, I think.

Cumulative frequency distribution for words on r/tokipona. It shows an inverse power relationship (grows fast at first but slows down exponentially).

Aaaand here's a table of all the words, ordered by frequency. BUT WAIT. Something just occurred to me. A lot of people write in English on r/tokipona, so the letter "a" probably appears more popular here than it is in actual toki pona.

wordcount% wordcount% wordcount%
a233110.59ale1230.56kule390.18
pona14196.45jo1200.55pan390.18
li13706.22lipu1180.54seli360.16
toki11545.24moku1170.53ko360.16
e10554.79lawa1070.49mije360.16
mi9814.46sama1020.46mani340.15
ni7003.18ilo940.43palisa320.15
jan6212.82tomo920.42unpa320.15
pi6162.80suli910.41olin310.14
la5632.56nasa890.40awen310.14
ala4952.25pakala840.38akesi290.13
tawa4552.07pana840.38open280.13
sina4011.82nanpa840.38esun280.13
lon4001.82suno840.38kute270.12
mute3881.76kulupu840.38alasa260.12
tenpo3031.38luka830.38pimeja250.11
ona2931.33pini820.37uta240.11
sona2621.19kalama820.37suwi230.10
sitelen2531.15sewi800.36jaki230.10
ken2461.12soweli760.35poki210.10
wan2351.07sike760.35loje210.10
o2231.01sin750.34len210.10
wile2211.00anu710.32selo200.09
ike2020.92ali650.30lete200.09
nimi1990.90moli610.28kipisi200.09
kama1990.90poka600.27jelo200.09
pilin1940.88wawa600.27oko180.08
ma1880.85linja580.26waso180.08
tan1750.80kiwen550.25sinpin170.08
nasin1710.78insa550.25monsi160.07
tu1680.76kin540.25laso160.07
lili1630.74telo540.25anpa150.07
kepeken1570.71mama510.23nena140.06
taso1450.66sijelo500.23pipi140.06
musi1430.65kili490.22noka120.05
ante1410.64mu480.22lape110.05
ijo1350.61meli470.21kala110.05
pali1330.60utala470.21walo110.05
en1330.60mun450.20namako70.03
pu1310.60weka440.20supa40.02
seme1300.59kon430.20lupa30.01
lukin1250.57kasi400.18

I'd be happy to hear from you at galligankevinp@gmail.com.