jigdo - Jigsaw Download | Richard Atterer (original) (raw)

Jigsaw Download, or short jigdo, is a tool designed to ease the distribution of very large files over the internet, for example CD or DVD images. Its aim is to make downloading the images as easy for users as a click on a direct download link in a browser, while avoiding all the problems that server administrators have with hosting such large files.

Article about jigdo on heise.de (c't-Magazin; German)

I have stopped development on this software long ago. But luckily, Steve McIntyre has taken over development - thanks Steve! Please head to einval.com for a forked jigdo version with new features.

Download

Help

Why Is It Needed?

jigdo was inspired by the problems that Debian Linux had with the distribution of its CD images. Debian is a 100% volunteer-driven organization which depends on donations to finance its infrastructure. For this reason, it also doesn't own any of the 300 servers around the world on which the distribution is mirrored.

For some time, the size of Debian has been a problem - Debian supports 11 different processor architectures, more than any other Linux distribution, and for every single one of these architectures, there are more than 10000 software packages - again more than any other distro. Today, a full Debian mirror needs roughly 100 GB of disc space!

Debian also offers the distribution in the form of CD images (and more recently DVD images). If these images were distributed as full .iso images on the mirrors, the size of a Debian mirror would double to 200 GB (or even triple to 300 GB if there were also DVD images) - nobody is prepared to mirror such a huge amount of data!

jigdo solves this problem by not requiring that the full image be stored on the servers. Instead, it can download the individual files from a normal Debian mirror and only assemble the image on the user's machine. Additionally, it can also do other useful things like helping with the mirror selection and upgrading images.

How Does It Work?

When someone has produced a large file they wish to distribute (for example, a CD image), they process this file with the jigdo-file tool. Among other things, jigdo-file creates a file with a .jigdo extension. To download the original file, a user only needs to tell the jigdo-lite download tool the URL of the jigdo file, it will extract all the necessary information and perform the download, fetching the data in many small pieces, possibly from several different servers.

So far, this is not exciting; there are already several other programs which allow you to split and reassemble large files, or which make their download easier. Also, at the moment only the shell script jigdo-lite can be used for downloading. Unfortunately, the jigdo GUI download manager shown below never reached a usable state.

[Screenshot]
The jigdo download tool (unfinished, not usable)

However, jigdo's approach is unique due to the way it identifies the small pieces that the large file consists of: For instance, if the file is a CD image, the individual files on the CD are contained somewhere within the image file. jigdo-file is capable of finding these files, so if copies of the individual files on the CD are stored separately on the server, jigdo-lite can be made to download these files one by one, and assemble the CD image on the fly on the user's computer. This scheme has several advantages:

If you have read this far, you may be confused now, asking yourself what jigdo does and doesn't do. :-) It seems that the jigdo concept is a bit difficult to grasp at first. If you're still interested, have a look at the respective section of the HOWTO and at the Examples section of the jigdo-file manual.

Further jigdo Resources