A short history of farming

Here is a very nice email we received some time ago:

re: Donating some compute power to the zim farm

Hi,
I have some spare compute power for at least a month, if that’s useful.
If so, here’s a virtual machine that can be used for the Kiwix project:

Storage: 4TB
Memory: 32GB
Network: ~950Mbps symmetrical, IPv4 & IPv6
CPU: 8 cores of i7-6900K

You can do anything on it as long as it’s legal, secured against hacking, and for the Kiwix project.

Thanks for Kiwix!

Well, that is nice. But then you ask: “What is this zim farm? What does it grow? Is zim a type of vegetable?”

Kiwix is an open-source project and we feel that transparency is not only about publishing the farm’s code but also explaining how things work. Therefore, kids, today we will learn about the Ancient Art of Farming. Or more specifically: ZIM file farming.

In the Olden Days, whenever we wanted a new version of a zim file (e.g., the September version of Wikipedia because the last available copy was from June), someone would have to type a command and manually start a process of retrieving the target website’s data, copy it, compress it, download and upload the resulting zim file. Yes, it was that tedious.

Booh.

So we set out to automate all of this because, well, we could (and it made sense, particularly with more than 2,000 ZIM files to update every month). And that’s how the Zimfarm came about – you can check it out for yourself on farm.openzim.org:

Once the process is over, the new ZIM replaces the old one on our library, and from there to our mirrors around the world. Currently the farm runs more than 4,000 recipes in 100+ languages on a half-dozen servers, 24/7. Most files are updated on a monthly basis.

Become a farmer

The good news is that you can easily help us by donating server resources: the full instructions are here but basically you will need a GNU/Linux host (works on macOS) with at least:

2GB of RAM and 3 cores available;
Docker CE server running;
Fast internet connexion (downstream and upstream);
A Zimfarm user account on your machine (with appropriate worker role);
An RSA private key with its public key uploaded to the user account;
The clock must be synchronized, e.g. using ntp.