My model uses Markov chains for letter-to-letter transition probabilities, hierarchically on word length.

What that means is: when I generate a random word, I first choose a word length. To do that, I’ve consulted a word list for the particular language. If 21% of words in the list have 5 letters, then I give myself a 21% chance of generating a 5-letter word, and so on for other word lengths. Having selected the word length, the second part is to pick the letters in the word. I’ve consulted the word list to find how often 5-letter words start with a given letter — for 5-letter words, h is the first letter 2.5% of the time; suppose I pick h to be the first letter. Then I consult my data to see what letters follow h as the second letter of five-letter words, and how often, maybe picking a. Consulting my data to see what letters follow a as the third letter of a five-letter word, maybe I pick b: this gets me up to hab _ _. And so on, to the last letter of the word.

What makes this a Markov-chain model is that I only look at transition probabilities from one letter to the next — I don’t look at correlations between the frequency of, say, the 1st and 4th letters. What makes it hierarchical is that I keep a separate Markov chain for each word length. The five-letter words don’t know what the four-letter words are doing.

Plusses:

• This is a simple model, easy to implement in software.

• The randomly generated words often look like they do belong to the correct language. The above samples, of 20 words each, were made by acquiring statistics from word lists for English, French, Spanish, and German, respectively. In particular, it’s reassuring to see that actual words (such as egg, sans, tío, and alte) do sometimes appear. (Certainly, when you ask a native speaker of a language to make up a nonsense word, the method their mind uses is nothing like the one presented here. Birds flap their wings and airplanes do not; when we build machinery which imitates nature, we do not imitate exactly — if we can build it, and if it flies, we use it.)

  By contrast, here are a few 5-letter words where each letter was drawn out of a (metaphorical) hat, with each of the 26 letters a-z being equally likely, and no relationships between which letter is more or less likely to follow any other:

  xyeot ivgcv bvxvq aqobd nneat rllvb ownrn qxovq ulsfa agerq mhhnj ryciv fioyp snsbl iwijo jdkyd ezwul ytedg nhqbj kzzid

  The hierarchical Markov-chain method certainly does better than this.

Minuses:

The word lists were all found via web searches, and are cached here. (Set your browser’s text encoding to Unicode after clicking through the French, Spanish, or German links, so that characters such as é will appear correctly.)

How long do English words tend to be? It depends! One of the central issues in statistics is the sampling problem: when we want to collect information about English, which English utterances do we look at? Everything written since 1900 (too much data!), an abridged dictionary, an unabridged dictionary, the front page of today’s New York Times? What is a representative sample? If I picked not the front page of the Times but the sports page, would I still be getting a representative sample of English words? How many pages of the newspaper would be enough? The same issue confronts pollsters: any sample which is smaller than the entire population has its own randomness, introduced by the very selection of the sample.

So, the answers you get depend on the sample you pick. Here are word-length frequencies for the English 2000-word general-service list and the Scrabble dictionary, respectively:

      *General-service list*        *Scrabble dictionary*
    Word length Count Percent     Word length  Count Percent
    ----------- ----- -------     -----------  ----- -------
              1     2   0.08%               1      0   0.00%
              2    18   0.78%               2    102   0.05%
              3   170   7.44%               3   1015   0.56%
              4   493  21.58%               4   4030   2.25%
              5   472  20.66%               5   8938   5.00%
              6   390  17.07%               6  15788   8.83%
              7   293  12.82%               7  24029  13.44%
              8   178   7.79%               8  29766  16.65%
              9   128   5.60%               9  29150  16.31%
             10    87   3.80%              10  22326  12.49%
             11    28   1.22%              11  16165   9.04%
             12    13   0.56%              12  11417   6.38%
             13     7   0.30%              13   7750   4.33%
             14     4   0.17%              14   5059   2.83%
             15     1   0.04%              15   3157   1.76%
    -----------  ----             ----------- ------
    Total:       2284             Total:      178692

Also note that the English GSL-2000 words tend to be shorter than the 178,692 words of the Scrabble dictionary — as we would expect for more common words.

Here’s some more output:

English GSL-2000 used as input:

English Scrabble dictionary used as input:

If we use three letters’ worth of memory instead of two, the results are better. (In each of the examples below, I used the longest word list available for the given language.) We no longer get un-English-looking bits such as oux or tss:

Here’s some multilingual output using the three-letter hierarchical model. Hopefully, it’s obvious by now which language is which!