download - Does aria2 support write small files in batch?
2014-07
Jon
I'm using aria2 to download 8 million jpg from flickr. Each image is about 100KB. I got a list of urls of these images in a txt file, the format is:
http://farm2.staticflickr.com/1070/1151334893_5a8e7f77f4.jpg
I'm wondering whether aria2 support writing small files in batch? Say write 100 image to disk when all of them are download in the memory, not just write every single file when the download is finished. Because I think writing in batch will better protect my hard disk.
Or do you have other software or opensource code to recommend?
Molotoff
I have a quite large file hosted on five different servers.
I would like to be able to download different parts of the file from each server and subsequently concatenate the parts, in order to produce the original file.
Is there any way to do this with curl/wget or any other OS X command-line tools?
xpixelz
aria2 is the solution for multiple downloads of same file from different locations
in your case : aria2c -s5 host1/bigfile host2/bigfile ... host5/bigfile
better late than never...
Caleb Gray
I've wanted this at least once a year for the past decade. I've taken the time to write the script; you can find the latest version here.
Sources: https://github.com/calebgray/scripts/blob/master/usr/local/bin/download-multiple
Just save it somewhere, make it executable, and give it a shot:
cd /usr/bin
curl -o download-multiple https://raw.github.com/calebgray/scripts/master/usr/local/bin/download-multiple
chmod +x download-multiple
./download-multiple http://mirror.pnl.gov/releases/oneiric/ubuntu-11.10-desktop-amd64.iso http://linuxfreedom.com/ubuntu-releases/.pool/ubuntu-11.10-desktop-amd64.iso ftp://mirror.us.leaseweb.net/ubuntu-releases/oneiric/ubuntu-11.10-desktop-amd64.iso ftp://mirror.lstn.net/ubuntu-releases/oneiric/ubuntu-11.10-desktop-amd64.iso
Sources