How text becomes the pieces a language model reads. Start with single letters. Repeatedly glue together the most common neighbouring pair. Drag the slider and watch words appear.
Each coloured block is one token. A leading space belongs to the word after it, shown as a dot. The outlined blocks are the ones created by the most recent merge. The text is a short passage I wrote about this very process, so you can watch its own words learn themselves.
Latest merges:
Nothing leaves your browser. Try code, another language, or a repeated phrase. Rare words stay in pieces; common ones become single tokens.