Distributed Mirror Project

News: Distributed Mirror Project 2

As I stated I only wanted to try this system out and I am not the right person for the job. But I did manige to attract some one who is. Jeroen van Wolffelaar has now made Distributed Mirror Project 2. A proper implementation using PHP and a database reather than my sed and perl hacking. With lists of top sites, links to google and webarchive mirrors and many other advantages it looks quite good. I'm turning off my mirror server but I'll leave the rest of the page for the sake of it.

What is it?

Distributed Mirror Project is an attempt to stop the /. effect. The first viewers of a site can download its contents and share it before the site grinds down to a halt.

How do I mirror stuff?

Firstly you will need to be running on a Unix (Linux, BSD, Hurd etc. Its much easier this way for now) To share the sites you have downloaded you will need to run a webserver, Apache is very easy to get going. You will need to create a directory from where you are happy to share files from. Once you have a directory and know where it exists on the internet (e.g. mine is at http://solem.cs.man.ac.uk:8006/leech/) you can run the script. The script expects to be in the shared directory. Run the script with two parameters: site to download and publicly visible directory.
e.g. I could run

mirror http://www.cs.man.ac.uk http://solem.cs.man.ac.uk:8006/leech/

Update: I think the script is quite poor. It doesn't deal with the most simple of pages (anything involving a ~ or a = is beond it). Instead just use your favorite method of creating a mirror and then submit it using the form. I use the line:

wget -r -N -l 1 -nr --timeout=10 --no-parent --convert-links http://www.TheSiteIWantTo.get/

Or ther is also a very nice tool called httrack. It gets entire sites with multiple connections.

I don't want to run your script.(probably a good choice)

The script simply automates the process but you can always enter the mirrors directly.

How do I find mirrors?

For now the easiest way is to type in the website into the box below.

Good and Bad

You can now rate the mirrors. Initialy mirrors are inserted with 50% score. If there are several mirrors with high scores try to go for the middle scoring ones reather than just hammering the top scoring entry. There is a bottom limit where if a mirror scores less than 10% it is deleted.

What about the server code?

The server code is available. If you want to create a server then please tell me as the current server address is quite annoying.

Does it work?

Well sort of, it is currently in the early stages. Addresses with funny characters (like ~) don't work yet. But its early days.

What next?

Well I only wanted to prove that the theory works (and it does), so I am looking for other people to take over the project.
If I am bored then I might make a Mozilla plugin for it, if it is based on leech then it will be quite easy.

Mozilla button

If you have found a slashdotted page then you can go straight to a mirror by using this link. Simply drag it to your bookmarks bar and press it when you want to see a mirror of the current page.
D-Mirror
Thanks to Dan Fandrich for the link.