[tor-dev] Tor-stem in python script - HTTP requests issue

Sat Aug 3 20:02:06 UTC 2013

Hi Damian and thanks for your answer. Actually I only need 2 (at most)
different IPs at a time from different countries, that's why I need to use
2 instances of my script with separate circuits. I have to test a CGI
script on a server of mine and each request is delayed 2 seconds from the
next one, so I think this situation shouldn't be an issue for the TOR
network. So we don't have 400xN but 1request x 2instances every 2 seconds.
In netstat i see at most 4 established connections (about tor) and even
socklist gives me the same result.

Btw, let's consider only one instance of the script. It blocks at N~=200
requests (it makes a get first - N/2 in total - and then a post - N/2 in
total - both blocking requests). Enabling log, the last message is:
[debug] parse_socks(): socks5: ipv4 address type

Previous similar messages were followed by:
[debug] connection_ap_handshake_process_socks(): socks handshake not all
here yet. #or Client asked for [scrubbed]:80
[debug] connection_ap_handshake_process_socks(): entered.
[debug] connection_ap_handshake_process_socks(): socks handshake not all
here yet.
[debug] conn_write_callback(): socket 64 wants to write.
...

But once i got:
[warn] Your application (using socks5 to port 80) is giving Tor only an IP
address. Applications that do DNS resolves themselves may leak information.
Consider using Socks4A (e.g. via privoxy or socat) instead. For more
information, please see
https://wiki.torproject.org/TheOnionRouter/TorFAQ#SOCKSAndDNS

So it seems that the script gets stuck here:
connection_ap_handshake_rewrite_and_attach()

Thanks a lot Damian, all

Eduard

Il giorno 02/ago/2013 19:41, "Damian Johnson" <atagar at torproject.org> ha
scritto:

> Hi Eduard. On first glance I'm not aware of any resource that would be
> exhausted by this after 200 iterations. Does the issue repro if you
> just do 400 GETs or 400 POSTs, or does the problem only arise when you
> do a combination of 200 of each? Have you tried running netstat or
> another connection resolver when it gets stuck to see if you have 400
> open connections (that is to say, checking that this is terminating
> the connections as expected)?
>
> On a side note I'm a little concerned about you running multiple
> instances of this script to run through multiple relays. 400 requests
> x N instances would be quite a bit of traffic to dump on the network.
> In addition, each instance is downloading the full tor consensus and
> microdescriptors since it does not share a data directory. What
> exactly is the goal of your script?
>
> Cheers! -Damian
>
> On Thu, Aug 1, 2013 at 10:00 AM, Eduard Natale <eduard.natale at gmail.com>
> wrote:
> > Hello guys,
> >
> > I had a problem and currently I'm not able to solve it. So, here I am ;)
> I
> > have a python script that uses python-stem to create and handle a tor
> > instance (on a defined port). What it does is retrieving (using a  HTTP
> GET)
> > a web page and submitting information (using HTTP POST messages).
> > Basically i use tor because I need to test this server from different IP
> > addresses with more requests in parallel. What I also do is keeping
> trace of
> > Cookies. Here's a sample of the code I use, based on the example on stem
> > website https://stem.torproject.org/tutorials/to_russia_with_love.html(to
> > have more parallel requests, i launch the script many times with
> different
> > socks_port value):
> > ----------------------------
> > import socket, socks, stem.process
> > import mechanize, cookielib
> >
> > SOCKS_PORT = 9000
> > DATA_DIRECTORY = "TOR_%s" % SOCKS_PORT
> > socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, '127.0.0.1', SOCKS_PORT)
> > socket.socket = socks.socksocket
> >
> > tor_process = stem.process.launch_tor_with_config(
> >           config = {
> >             'SocksPort': str(SOCKS_PORT),
> >             'ControlPort': str(SOCKS_PORT+1),
> >             'DataDirectory': DATA_DIRECTORY,
> >             'ExitNodes': '{it}',
> >           },
> >         )
> >
> > # initialize python mechanize, with cookies (it works exactly like
> urllib2,
> > urllib3, etc. already tried...)
> > br = mechanize.Browser()
> > cj = cookielib.LWPCookieJar()
> > br.set_cookiejar(cj)
> > ...
> >
> > for number in num_list:
> >   req = br.open_novisit("http://example.com") #_1_
> >   res = req.read()
> >   print res
> >   req.close()
> >   req2 = br.open("http://example.com/post_to_me", data_to_post) #_2_
> >   res2 = req2.read()
> >   req2.close()
> > --------------------------------
> >
> > And that's it. The problem occurs on the lines i marked as _1_ and _2_:
> > basically when it reaches around 200 requests, it seems to block
> > undefinitely, waiting for a response that never comes. Of course,
> > wiresharking doesn't work because it's encrypted. The same stuff, without
> > TOR, works perfectly. So, why does it stuck at about 200 requests!? I
> tried
> > to:
> >
> > 1. Telnet on control port, forcing to renew circuits with SIGNAL NEWNYM
> > 2. instantiating mechanize (urllib2, 3, whatever) in the loop
> > 3. ...i don't remember what else
> >
> > I thought it could be a local socket connection limit: actually without
> TOR,
> > i see in wireshark the source port changes every time a request is
> > performed. But actually i don't know if the problem is in using the same
> > source port every time (but i don't think so) and if so, should I close
> the
> > current socket and open a new one? Should I kill the tor process? I can't
> > exaplain myself why...
> > What I only know is: *when the script stucks, if i kill the python
> process
> > (ctrl+c) and then re-launch, it starts working again.*. I've seen that
> it's
> > possible to set the value of TrackHostExitsExpire, is it useful in my
> case?
> >
> > Thanks in advance to whoever can help me!!
> > Ed
> >
> >
> > _______________________________________________
> > tor-dev mailing list
> > tor-dev at lists.torproject.org
> > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
> >
> _______________________________________________
> tor-dev mailing list
> tor-dev at lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20130803/cb3233d0/attachment-0001.html>