Here is a broad update of where we stand in terms of technology for the project.
First, the chart below summarizes broadly our tech stack and may or may not help you navigate what follows:
Overall Kiwix is healthier than ever, with about 30% more code contributors, releases, tickets closed, etc. than last year. You can check the metrics dashboard at https://metrics.kiwix.org/ (select 5 years time range).
Our software portfolio and code base are pretty stable on the Kiwix (reader) side, and increased on the openZIM (content) side. This reflects our pivot towards generating more ZIM files, and to do it faster and better.
Software
The overall Kiwix software quality has improved, with a continuous and significant decrease of bugs tracked since 2019. Code coverage increased significantly as well, with an ever greater share of our repositories having a clear test / Continuous Integration / code coverage strategy. But progress on that front remains slower than we would like it to.
Work on Kiwix ports continues with regular releases, bug fixes, etc.. As the situation improves, changes are now much less disruptive than before – in particular for Kiwix Android.
Improvements over the past 10 months to the core libzim and zim-tools have been very successful. The effort is not over but we have fixed most known bugs. We also improved zim reading speed by a factor of 10. This simplifies a lot of our work on content scrapers as well as for Kiwix developers. It also allows for the industrialisation of ZIM creation. It also reduces the app’s energy footprint, which is important on lower-end devices.
Content
The operational aspect of the project, running online services, is pretty satisfying. We are reaping the benefits of investments made over the past 2-3 years with a code-driven infrastructure based on Docker. Uptimes is really good (upwards of 99%) and maintenance costs minimal. We are still investing a bit to have a better backup solution, but that’s about it.
Python-based scrapers (90% of our portfolio) have all made a giant leap forward in term of quality and reliability. This move has taken a large amount of resources, but the situation is almost perfect now. The ZIMit scraper, which should allow us to make ZIM files from “random” web sites, makes good progress. The challenge will be in achieving a universal port on all Kiwix platforms, i.e. those maintained by volunteers instead of paid staff.
What next?
The Kiwix tech stack is getting better by the day. But if our efforts to stabilize the situation over the past 18 months allow us to be optimistic, there remain a few annoying pain points:
1 – MediaWiki offliner (the Wikipedia scraper) still suffers from chronic technical debt;
2 – Library and content management are definitely behind schedule, from the low level in libkiwix to high level in Kiwix. This is visible for instance in Kiwix Desktop and Kiwix Serve;
3 – We are behind schedule on the Hotspot and Cardshop projects.
Now that most of the technical debt has disappeared we can increase our focus on the above three priorities. Considering that our baseline budget has taken a hit, it will probably be hard to continue to significantly improve the Android and Desktop apps, but this remains a very acceptable trade-off.