download - file downloaded via firefox and curl have different size
2014-07
Arash Mousavi
When I download a file from this link by Firefox its size is 74580 B, But when I download it by curl with exactly all of header was sent by Firefox its size is 79891 B (I copied all header from Firefox and paste it in curl command).
what is the problem?
If you need any additional data ask me in comment.
My curl command:
curl --header 'Host: members.tsetmc.com' --header 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:29.0) Gecko/20100101 Firefox/29.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: en-US,en;q=0.5' --header 'Referer: http://www.tsetmc.com/Loader.aspx?ParTree=15131F' --header 'Cookie: ASP.NET_SessionId=pwzbckbdpjlzqj45vcdbd455' --header 'Connection: keep-alive' 'http://members.tsetmc.com/tsev2/excel/MarketWatchPlus.aspx?d=0' -o 'MarketWatchPlus-1393_3_14.xlsx' -L
AFH
Glad you sorted it. I find meld on Ubuntu incredibly useful for comparing text files: it installs compare options into the Nautilus file manager context menu:-
- if two or more files are selected, the context option is Compare, and meld is scheduled to compare all of them (I don't recommend more than four);
- if a single file is selected, the context option is Compare Later, and the next file selected has the context menu Compare to previous file, as well as Compare Later to make this the new reference file for the next selection (for use if the files are on different discs or in different directories).
Arash Mousavi
Webserver that is serve that file has enabled gzip compression, Firefox uncompress the gzip, but for curl I should --compressed to uncompress it.
So my curl command is:
curl --header 'Host: members.tsetmc.com' --header 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:29.0) Gecko/20100101 Firefox/29.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: en-US,en;q=0.5' --header 'Referer: http://www.tsetmc.com/Loader.aspx?ParTree=15131F' --header 'Cookie: ASP.NET_SessionId=pwzbckbdpjlzqj45vcdbd455' --header 'Connection: keep-alive' 'http://members.tsetmc.com/tsev2/excel/MarketWatchPlus.aspx?d=0' -o 'MarketWatchPlus-1393_3_14.xlsx' -L --compressed
Behrang
Is there a way for downloading XCode using curl or wget?
EDIT: Why is the following curl snippet not working?
First, I login to Apple's site:
curl -d "theAccountName=USERNAME&theAccountPW=PASS" -c xcode-cookie
-A "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
https://daw.apple.com//cgi-bin/WebObjects/DSAuthWeb.woa/104/wo/W1df7WQblBsLekQzvS1tJ0/0.5.3.1.1.2.1.1.3.1.1"
Then, I try to download the file:
curl -b xcode-cookie -c xcode-cookie -O -v -A
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
http://adcdownload.apple.com/ios/ios_sdk_4.0.2__final/xcode_3.2.3_and_ios_sdk_4.0.2.dmg
However I get a 403 Forbidden error:
* About to connect() to adcdownload.apple.com port 80 (#0)
* Trying 60.254.148.16... % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0connected
* Connected to adcdownload.apple.com (60.254.148.16) port 80 (#0)
> GET /ios/ios_sdk_4.0.2__final/xcode_3.2.3_and_ios_sdk_4.0.2.dmg HTTP/1.1
> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1
> Host: adcdownload.apple.com
> Accept: */*
> Cookie: myacinfo=foo; DefaultAppleID=bar; ds01=baz
>
< HTTP/1.1 403 Forbidden
< Server: Apache-Coyote/1.1
< X-Powered-By: Servlet 2.4; JBoss-4.0.5.GA (build: CVSTag=Branch_4_0 date=200610162339)/Tomcat-5.5
< no-cache: Cache-Control
< no-store: Cache-Control
< Content-Type: text/html;charset=ISO-8859-1
< Date: Mon, 30 Aug 2010 05:30:42 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Connection: Transfer-Encoding
< Cache-Control: no-store
<
{ [data not shown]
100 1932 0 1932 0 0 7531 0 --:--:-- --:--:-- --:--:-- 8548* Connection #0 to host adcdownload.apple.com left intact
* Closing connection #0
purefusion
Josh thinks to simple-mindedly. The URL to download XCode is indeed http://developer.apple.com/iphone/download.action?path=%2Fios%2Fios_sdk_4.0.2__final%2Fxcode_3.2.3_and_ios_sdk_4.0.2.dmg
However, you must be authenticated to download the file.
Now both wget and curl support cookies, but I've never used curl for downloads so here's a couple lines using wget to first authenticate into the dev site (storing the auth cookies in a cookies.txt file), then another line to actually grab the download (recalling that same cookies.txt file):
wget --post-data="theAccountName=YOURUSERNAME&theAccountPW=YOURPASSWORD" --cookies=on --load-cookies=cookies.txt --keep-session-cookies --save-cookies=cookies.txt https://daw.apple.com/cgi-bin/WebObjects/DSAuthWeb.woa/67/wo/99OWdOWyhWA2pMdrPemdSw/0.5.3.1.1.2.1.1.3.1.1
wget --cookies=on --load-cookies=cookies.txt --keep-session-cookies --save-cookies=cookies.txt http://developer.apple.com/iphone/download.action?path=%2Fios%2Fios_sdk_4.0.2__final%2Fxcode_3.2.3_and_ios_sdk_4.0.2.dmg
Be sure to change YOURUSERNAME and YOURPASSWORD to the appropriate values, and change the path to the cookies.txt file if so desired. Also, note the URL of the login form might change, so be sure to check the action attribute of the login form from your browser if this fails to work.
I tested this on my server, and the download took about 40 mins. Worked a treat!
Cheers!
vdm
Single command, using stock Mac OS X 10.6.6 command line tools (bash, curl and sqlite3).
curl -b <(sqlite3 -separator $'\t' ~/Library/Application\ Support/Google/Chrome/Default/Cookies "select host_key, 'TRUE','/', 'FALSE', expires_utc, name, value from cookies where host_key like '%apple.com%'") http://adcdownload.apple.com/ios/ios_sdk_4.2__final/xcode_3.2.5_and_ios_sdk_4.2_final.dmg
Caveats:
This uses a query of Google Chrome's sqlite cookie database.
You need to login via Apple's Xcode developer page with the browser first so that the cookies get set.
Notes:
<() is bash's Process Substitution operator and does away with the instantly-stale temporary cookies file.
If, like me, you are resuming a stalled browser download, and you have the file, use these arguments:
-C - -o unconfirmed\ 21043.crdownload
Now that you have Xcode, you can install Homebrew, and then wget. ;)
Josh K
Of course, just copy the download link that you got from Apple.