While investigating some compression schemes I extracted some statistics from a (random) subset of our data so far.
The statistics are how many times a word occurs in each document (in documents that contain the word). The subset may be slightly biased and incomplete representation of the complete web, but the differences are intriguing nonetheless. The counts includes all occurrences of the word (both in title, keywords, description, body text, headings, …). The counts are without synonyms and variations, so exact-match.
Why is “potato” repeated 5 times in each document while “lingonberries” barely has one occurrence?
I’m not sure if one can draw any conclusions from that, but I have the hunch that documents with “lego” in them will refer to that word only once and after that just use the word “piece”. The weirdness of “linux” I have no theories about.
City names turned out to have some intriguing differences
Are well-known city names repeated more? No. Are names of big cities repeated more? No. Are northern city names repeated more than southern city names? No. I can find no pattern in these numbers. Canada, please tell us what is going on 🙂