top left Wikipedia Suggest top right
Suggest: English
Suggest: Deutsch
Suggest: Français
Suggest: Polski
Suggest: 日本語
Suggest: Nederlands
Suggest: Italiano
Suggest: Svenska
Suggest: Português
Suggest: Español

Download
Project history
FAQ
Web statistics

Personal Web Site

Valid XHTML 1.1

Download


Grabbing

At the begining, I wrote some programs and scripts to grab the whole content of wikipedia articles. But I realized that it is not the good way to obtain the content of Wikipedia. It is faster and better to use the wikipedia dumps available at download.wikipedia.org. All languages are available : english, Deutsch, French, ...

Analysis

Program that convert XML articles from download.wikipedia.org in a trie and after a program to perform queries on this trie. For the moment, I wrote two versions of the query program :
  1. The first one (DiskQuery) is a mono-thread implementation using very few RAM (only 16M of RAM for the whole automaton of english articles) but perform the query in a random access disk file. This solution is ideal if you have a server and very low RAM available for this service (this is the solution I use on my web site)
  2. The second version (MemoryQuery)is a multi-thread capable implementation allowing a lot of queries per second on a standard server (about 6000 queries per second on a pendium D940). This version use about 200Mb of RAM for the whole automaton of english articles
By the way, it is also possible to implement a distributed queries program that allow us to have a very huge automaton in memory.
All these programs are available under GPL licence in this [archive], you can read the README file inside the tarball for a fast help.

Compiled resources

If you do not want to compile files from download.wikipedia.org or you do not have time to download them, you can download the compiled resources here :