$7.49 .com domain with free InstantPage Website Builder$1.99 Web Hosting   捷飞网络官方淘宝店   Godaddy 优惠码
返回列表 发帖
分享到:




[Tools] Wget Proxy: How to run Wget behind a Proxy server

Since, proxy servers are very common today - in companies, universities etc. almost all of the applications today have proxy support. As far as wget is concerned, there are two ways of doing so, either set the appropriate environment variables and wget will act according to them or configure the wget configuration file - wgetrc.+ W5 a4 o% c8 Y+ D# c  k
/ ]# a$ Q& q; f; y# L; i
Wget Proxy: Set the environment variable
- d7 F5 k9 P+ C4 y8 ]1 x! n" X- _9 k9 J
Before setting the environment variables, lets first see if there are any preset proxy variables present. We can get that information using the following command.
. z! ]# v" K3 h0 N4 a+ Z% \) d& S) |. u0 ]& M5 h" \7 r
[shredder12]$ env | grep proxy7 D8 q% X4 x# V6 W! b( J

7 W. O9 O3 n: H$ r' {http_proxy=http://foo.bar:3128" e& s1 `7 P$ R  K) S; @+ c4 ]
no_proxy=localhost,127.0.0.16 U2 n/ w& i# J( k
9 `  A1 p0 l4 V
If no proxy environment variable is set, it will go for direct connections. Lets take an example of my college. We use a HTTP proxy server. Running wget without setting the http_proxy environment variable just fails. So, lets set this env. variable first
8 I8 {) R; w( i8 _8 j
$ d3 S8 J2 ?, i& X' b! z[shredder12]$ export http_proxy="http://foo.bar:8080"- c' q* ?' m) D/ y
- s; F3 x- f5 [5 p. m5 j
Now, run Wget and it should work like a charm. If your proxy also requires authentication i.e username & password then use this format+ A/ c0 y, s& b2 C% U

5 z8 g$ X* l5 h% L& c[shredder12]$ export http_proxy="http://username:password@foo.bar:8080"
  a) H! E3 r6 R, q: b# B# f9 e1 [4 l! ~6 Z; J5 p  D
Similarly, for other type of proxies - ftp_proxy, https_proxy etc. All we need to do is set the appropriate environment variable.# e3 f. c9 ~$ A- S* O
( _* ~( ^3 y+ a9 t
Please note that, the above commands will be in effect only till the user session expires, i.e. by running the above commands we are just setting the proxy env. for a session, not permanently. And usually we wouldn't want to run the command everytime we login. So, to make it permanent, we can either set the value in wget's configuration - wgetrc file or set the environment in ~/.bashrc file.
; I3 K' R4 N$ |- k& _  s
1 U3 U# d: ?. u# F- u% c  z[shredder12]$ echo "export http_proxy=http://foo.bar:8080/" | tee -a ~/.bashrc
; Z; R5 N' g! v( W. f
& ~* C5 E7 d8 y8 dIt should be noted that, this way we might be affecting the user's proxy settings. A better way is to set the settings permanently in wget's configuration file, this way no other app. is affected.
& @! C4 S' ?& e; u
1 ^9 [% l7 W$ D- Q/ q5 w) a( eWget Proxy: Configuring the wgetrc file2 }3 ~$ x4 f/ x4 |* u+ X( r

. l6 G  s3 F) S! I5 B5 cLike most of the applications wget has a configuration file too - wgetrc:
# P: D7 e# A/ h! v; t2 u; }7 ~% m$ q* {1 c1 n6 O
/etc/wgetrc,
- X1 b/ t2 s% l; g5 u~/.wgetrc.' v, X8 I2 c2 ]9 Q
The former is for global changes and the latter one is for local settings(user specific). We will go into the details later, lets just see how to apply the proxy settings. Its similar to setting the proxy environment variable, just exclude the export command. Open the file ~/.wgetrc file. If one doesn't exist, then create it.5 g1 i5 E% ?( b4 n+ I, k9 B

" j1 g5 N& Q( s: b' @" h( o4 L[shredder12]$ vim ~/.wgetrc
8 K& f* z% |1 M$ g# g" j: K9 O; d' W9 h* p5 w/ e9 K
Now, add the following statement in the file
$ u0 ?; e0 f, b0 m9 c7 W/ }  v3 q# l; F
http_proxy=http://foo.bar:8080
$ f2 k+ }8 s: v' _& R& E. `
. t0 n* A/ s- }* K! F% P" iWget Proxy: How to disable/turn-off the proxy settings
$ ?) f9 K, }# V# M9 V& h, _$ m1 @: F3 U! f# m3 z( q3 `* N, O9 Y  n
Once you have configured the proxy settings in the wgetrc file, everytime you run wget, it will automatically connect to the proxy server. Sometimes for local/LAN based downloads, you may not need the proxy server. In such cases we can turn off the proxy settings using --no-proxy argument in the command
, e) K( B' k/ y  r8 i3 O( H# ]( @, C$ g1 E- u3 _: K; x, J
[shredder12]$ wget --no-proxy http://10.0.0.1/file.tgz& C4 X+ d5 U9 g) D7 b6 B, ?) m* w; ?6 a

& i: a" k3 r" Y9 g4 [Wget Proxy: If the proxy requries authentication - Username & Password  J$ b9 I* Z$ c/ Z* `2 A

  C) v7 P8 X$ J, CWe already know the format for setting the environment variable with username and password. Wget provides you two methods to mention the username and password for proxy server.
8 [/ Z7 n+ C/ I  Z
* E& _" K" x! d' z# mMention it in the command in the form of arguments.
. K! M( `" B6 C- RSet the values in the configuration file, /etc/wgetrc or ~/.wgetrc* E+ ~# b0 e4 G% S3 q
If the authentication you are using is static, then its better t set it in the configuration file. If its just one time thing or if you want to overwrite the settings in configuration file then you can use the --proxy-user & --proxy-password options.
) l. q% e/ d& q' }( h5 i+ @2 J/ e5 H4 p
[shredder12]$ wget --proxy-user=username --proxy-password=password http://foo.bar/file.tgz
3 q+ E: Z! c% f' ^1 @+ F! r* o4 s3 g( ^' `
For permanent changes, its always advised to change the configuration file.
' P& h# \8 S2 \# O6 c& l, X3 {8 j2 d
Once again, for global changes modify the /etc/wgetrc file otherwise go for ~/.wgetrc file. Open any of the file and append/add the following lines in it.! x, s+ O  \, X6 e% N& R

* U" m0 _, C0 o' E5 F- {$ r( J* Cproxy_user=username( \6 |4 l' x* S+ p- f* w
proxy_password=password& e0 _# b) j. S) p! \# m

* @% ~: E/ v, @! q! a; ~In a similar way, you can set the ftp proxy along with the authencation credentials.. A5 R* R; L  g! p
2 N2 F$ g8 @% t  S# t; B3 h2 |
How to ovewrite a preset proxy in Wget
2 ?7 X  y8 B9 V. X! ], l" c9 K0 u
6 |! d4 i+ e6 k7 O" |1 X2 mIf you see the first proxy example, we are actually overwriting the pre-defined environment variable(if any). But since, we are executing the export command, rather than editing any configuration file, the proxy setting will be removed as soon as we close the terminal ( in case of GUI, gnome-terminal etc.) or logout of the session.
7 Z4 Q" [! d' F0 F% c% j1 Y, @9 f8 N4 e  A" K6 s  F0 R8 D, w; g
And, if for some reason we are interested in using separate proxy for wget, then instead of overwriting the variable we can just edit the ~/.wgetrc or /etc/wgetrc file. Please note that, a local wgetrc file (~/.wgetrc) will overwrite the settings on /etc/wgetrc. Once again, remember that ~/.wgetrc is a local/user-specific setting where as /etc/wgetrc is a global one.) z1 l! N' I! s* O
& s# P7 w- G  R" y$ F
Finally, the high command in this hierarchy are the variables provided at command execution time. (--proxy-password, --no-proxy etc.), i.e.9 C* s" U5 l1 v4 o
: X/ Z1 l7 N: F1 U" F- ~
(Exec. time command options) > ~/.wgetrc > /etc/wgetrc > environment
' p* J9 d+ q& ^5 O8 L/ v" }
% \+ L* b7 n# m, E1 w9 NThis is just a big picture, only for proxy settings. It might not be the same in all the cases.8 B* u  \8 W% L9 d7 e

8 M4 m8 u2 S! R参考: http://linuxers.org/tutorial/wget-proxy-how-run-wget-behind-proxy-server




返回列表
Namecheap
Namecheap.com - Cheap domain name registration, renewal and transfers - Free SSL Certificates - Web Hosting
互联网安全