automation - automated ftp download, recursive folders, download specific file type for non technical user
2013-08
newuser1979
I have Windows 7 desktops and I want to check/download files from a few webservers (apache) into a specific folder on my desktop. My research seems to find it possible with paid ftp programs with scripting utilities, which I am not familiar with.
My webserver folder structure looks like this.
/home/foldera/folderb/exports/folder1/testfileabc.zip
/home/foldera/folderb/exports/folder1/testfileabc.txt
/home/foldera/folderb/exports/folder2/testfilexyz.zip
/home/foldera/folderb/exports/folder2/testfilexyz.txt
I want the ftp program to check for new *.zip files in /home/foldera/folderb/exports/ say every 6 hrs, download them to a local directory c:/datafiles/*.zip and then delete the file from the server.
Each zip file will have a unique name (there is a timestamp appended to its name). Sometimes the folder /home/foldera/folderb/exports/ wont exist (its deleted occasionally by the webserver before new data is written into it, so there is a chance it wont exist, that should not cause anything to fail).
As you can see I don't want a simple mirror application, I need the ftp program to login in, look for files recursively within the */exports/ dir and download just the zip files so that in C:/datafiles/ I just have zip files, no subfolders, and no files of any other type.
Considering I cannot batch script, but can follow instructions pretty well, how can I create an ftp script to do this? Is there one with a GUI and well documented instructions? Is there a way within windows to do this without the need for ftp software? I understand I may have to use task scheduler to trigger this.
rajeem_cariazo
I only use Windows Explorer to open FTP links. Is there a way to list all the available files and folders which I think is legal since the FTP is public? For example is this FTP location: ftp://ftpd.embarcadero.com/pub
njd
I tried that embarcadero link and the site was empty apart from the (empty) pub folder - is this what you expect?
If you visit (e.g.) ftp://ftp.mozilla.org, you will see lots of files and folders; and if you view the Folders pane in Explorer, you can browse into each folder.
The conventional way to get a list of all the files/folders on an FTP site is to download a file called ls-lR (or the compressed version: ls-lR.gz) and look at the file in a text editor.
If you look at the Mozilla FTP site, in ftp://ftp.mozilla.org/pub, you will see the ls-lR file right there.
But this is a convention which originated on Unix, so you would have to use a program which can understand Unix line-endings (Notepad does not).
I should add that even though FTP servers allow anonymous access, and are therefore "public", you can only see files which are made available through the FTP service.
You would not be able to browse back up to the root folder (or C:\ if it's a Windows system) and examine system files on the remote machine.
Update 2010-08-06
Just seen the comment about downloading a hidden file.
Well, FTP servers can be configured to hide files (although how this is done depends on the server). The files and folders will be hidden from directory listings, but are still accessible if you know the name.
It is possible to guess folder and file-names by brute-force, and people can write scripts to do this. But this would mean sending thousands of requests to the FTP server in just a few minutes, which is quite hostile; most server administrators would regard it as an attack.
Although the site may be open to the public, I would think that an administrator who has configured certain things to be hidden really doesn't want them to be publicly available.
Now it may be that the administrator really should have denied access to those files or disabled anonymous access, but the intent is clear; so I think brute-forcing the files would be considered illegal in some countries, and "legal since the FTP is public" might not be a sound defence.
So to answer your question: Yes, there are ways to find all the files and folders (even the hidden ones) on an FTP server by brute-force, but I'm not going to explain how, because it's legally dubious.