urequests timeouts

[kjm]

I'm running a battery powered GPY so looking to minimise the active time between deepsleeps. In regular python I'd use

r=requests.get(VERSION_API_URL + site, timeout=5)

to limit the time waiting for a server response to 5s. Unfortunately urequests doesn't accept a timeout

Traceback (most recent call last):
  File "<stdin>", line 198, in _updater
  File "/flash/lib/urequests.py", line 112, in get
TypeError: function does not take keyword arguments

and the default timeout it uses is quite long (30s). Given that a normal wake from deepsleep-attach/connect 4G modem-read sensors-send data cycle takes just 20s, the long wait for a stalled connection is a real pain.

Any suggestions from those-in-the-know for a workaround?

[kjm]

@andreas blobs are a bit beyond my skill set but ai=socket.getaddrinfo(host, 443) is using a socket so I reckon a 5s OSError socket timeout should prevent hangs if someone can figure out how to change the default timeout & recompile.

My thread attempt was pretty simple, put getaddrinfo in a thread, wait for 5s in the main program then move on if the thread had not produced a DNS by then. Instead the main program stalled after 1 iteration of the 5s delay loop and waited for the thread.

[andreas]

Dear @kjm,

@kjm said in urequests timeouts:

My current blunt force instrument is to put a wdt=machine.WDT(timeout=5*1000) in front of getaddrinfo & s.connect

This is probably the most blunt but best way to go for this, as I currently also can't see how to tweak getaddrinfo from lwIP in any way.

By [1], it looks like you could adjust the DNS_MAX_RETRIES parameter, but these things are so deep inside lwIP so they should probably not be fiddled with.

Using multithreading might sound feasible, but I can't decipher your code at [2]. Doing it that way will have to be investigated more thoroughly.

So, as long as you don't want to put more efforts into it, using the watchdog timer would probably the best workaround solution.

With kind regards,
Andreas.

[1] https://github.com/espressif/esp-lwip/blob/61d840ff4778f4946c8743f7e412345abcd537f1/src/include/lwip/opt.h#L1131-L1134
[2] https://forum.pycom.io/topic/5295/getaddrinfo-in-a-thread

[kjm]

@andreas said in urequests timeouts:

Dear @kjm,

I see that you probably already incorporated the changes from [1] into your urequests module, right?

I had that working briefly in 1.18.1.r7 Andreas, or at least I thought I did, but it certainly doesn't work anymore in 1.20.1.r1.

My current blunt force instrument is to put a wdt=machine.WDT(timeout=5*1000) in front of getaddrinfo & s.connect since everything else I've tried hasn't worked, most bizarrely https://forum.pycom.io/topic/5295/getaddrinfo-in-a-thread/4 where the main program actually pauses & waits for the thread, the exact opposite of what it's supposed to do!

[andreas]

Dear @kjm,

I see that you probably already incorporated the changes from [1] into your urequests module, right?

It also looks like @zmaier asked for the same thing at [2].

@kjm said in GPY full flash erase:

the 20s wait when a DNS lookup fails at getaddrinfo in urequests really bugs me. In reality if a DNS server hasn't replied in a tenth of that time it's not going to. I'd kill to be able to tweak that value.

Now if tweaking the urequests module works already, the next thing to tackle would be to tweak getaddrinfo appropriately, right?

As far as I can see, this seems to be tricky if possible at all. Bummer!

With kind regards,
Andreas.

[1] https://github.com/micropython/micropython-lib/pull/263/files
[2] https://forum.pycom.io/topic/4886/urequests-and-timeout

[kjm]

@jcaron Yeah I had to re do urequests. So hard for a plod to do surgery on a library file & so distracting from the main task of writing a bullet proof main.py. Who are these geniuses that write socket programs sans timeouts or try/exception protection?

[jcaron]

@kjm One option could could be to use a separate timer to cut things short if things don't go as planned, but that would have part of the issues you mentioned earlier (could abort a valid request in progress, etc.), unless you modify urequests to use that timer just for the getaddrinfo (and possibly initial connect) part, then cancel the timer.

[kjm]

OK I have a better handle on the problem now. Maybe if I explain it better some wunderkind can offer a solution? The problem is urequests uses getaddrinfo which can't accept a timeout (https://stackoverflow.com/questions/26857922/pass-timeout-to-socket-getaddrinfo/28120166#28120166). So if the DNS lookup fails urequests hangs for a bit. Strangely the hang time is different if it runs from Atom (~10s) to if it runs from flash in the GPY (~30s).

Having a battery powered GPY stalled waiting for a 30s timeout is hopeless so I'm looking for a fix. All suggestions gratefully received!

[kjm]

I worked out how to set a timeout in urequests but it's a blunt instrument. Firstly it always waits for the timeout regardless of how long it takes to get a server response. Secondly it will abort a download mid stream if the timeout expires. Surely there is something better than this?

What I need is a timeout that limits how long to wait for a server response but cancels once the server starts responding. This would give me data as soon as it was available & would stop the timeout cutting off a response mid stream.