HTTP POST request fails with "IndexError: list index out of range"

[Jan Dillmann]

I'm trying to make a HTTP POST request using the urequests library. I uploaded this file to lib/urequests.py and have a GPy with a working wifi connection (sending data via MQTT works as expected, so the internet connection is active).

from urequests import post
from network import WLAN

wlan = WLAN(mode=WLAN.STA)
print("WLAN:", wlan.isconnected())

url = "http://httpbin.org/post"
headers = {"Content-Type": "application/json"}
data = {"foo": "bar"}
response = post(url, json=data, headers=headers)
print(response.json())
response.close()

After printing WLAN: True the output hangs for about 30 seconds (at the post() call) and then fails with this error:

Traceback (most recent call last):
  File "main.py", line 10, in <module>
  File "/flash/lib/urequests.py", line 115, in post
  File "/flash/lib/urequests.py", line 100, in request
  File "/flash/lib/urequests.py", line 84, in request
IndexError: list index out of range

Any idea on where the issue might be?

A curl call from a computer on the same wifi network works as expected: curl "http://httpbin.org/post" -d '{"foo":"bar"}' -H "Content-Type: application/json"

[jcaron]

@Jan-Dillmann Can you uncomment the print(l) at line 82 and let us know what it says?

[Jan Dillmann]

Uncommenting the line only outputs b''. Does this and the long wait suggest there is no response coming from the server?

[kjm]

@Jan-Dillmann You've got further than I ever did with json posts (https://stackoverflow.com/review/suggested-edits/28626395). Maybe insert a print(data) at line83 in urequests to see what's it actually sending to the server?

[Jan Dillmann]

@kjm I changed urequests.py to contain the following:

if data:
    print(data)
    print(isinstance(data, str))
    s.write(data)

The output is {"foo": "bar"} and True, so it is a string containing the data serialized as JSON.

I also tried serializing the data to JSON myself, but the error is the same:

import ujson
from urequests import post

url = "http://httpbin.org/post"
headers = {"Content-Type": "application/json"}
data = ujson.dumps({"foo": "bar"})
response = post(url, data=data, headers=headers)
print(response.json())
response.close()

So I guess it's not a problem with the request content but rather with the underlying socket?

[jcaron]

@Jan-Dillmann It does indeed look like there's no response coming from the server.

Do GETs on the same server work? Do POSTs on other servers work?

The normal reason for a server not to reply would be that it's still waiting for the request to be finished. Some possible causes for this are:

• Mismatch in CRLF (using only CR or only LF)
• Missing double CRLF (i.e. empty line) at the end of the headers
• Mismatch on the body length (Content-Length v. actual body length)
• Data buffered but not sent

But I don't see anything that would cause this here. The only weird thing is the HTTP/1.0 version, but tests show no difference with that server.

Can you check the value of len(data) and the return value of s.write(data)?

Don't have a *Py module at hand to test. Do you have any way to capture the traffic between the module and the server? Do you know if your traffic goes through a proxy?

[Jan Dillmann]

@jcaron Okay, I finally figured it out…

Unbeknownst to me, deep in my router's settings was a "security" feature active that tries to block malicious outgoing requests. Not having a User-Agent header apparently qualifies as malicious, so the POST requests from the GPy were blocked. cURL on the other hand adds it's user agent header, so these requests from my laptop in the same wifi worked just fine.

Adding a User-Agent header (and disabling this feature in the router) did the trick – HTTP requests work as expected now.