urequest IndexError

[Tgasser]

Hello,

I have this error :

All I notice is that the problem appears when the data is very important. Do you have any idea how to solve the problem? Thank you.

Example of data :
{'1': {'mac': b'XXXX', 'reg1': {'timestamp': (2020, 1, 2, 14, 15, 1, 221154, None), 'data': b'XXXX'},'reg2': {'timestamp': (2020, 1, 2, 14, 15, 5, 78284, None), 'data': b'XXXX'}}, '0': {'mac': b'XXXX', 'reg1': {'timestamp': (2020, 1, 2, 14, 15, 1, 94182, None), 'data': b'XXXX'}}, '2': {'mac': b'XXXX', 'reg1': {'timestamp': (2020, 1, 2, 14, 15, 5, 78284, None), 'data': b'XXXX'},'reg2': {'timestamp': (2020, 1, 2, 14, 15, 5, 78284, None), 'data': b'XXXX'}}}

Code :

def request(payload):
    print("DSN :")
    print(usocket.dnsserver())
    usocket.dnsserver(0, '8.8.8.8')
    usocket.dnsserver(1, '10.0.0.1')
    print(usocket.dnsserver())
    flag=False
    while(flag==False):
        try :
            print("Using library urequest...")
            r= urequest.post("https://labs.de/node-red",json=payload)
            print("Return:", r.text) #r.content
            flag=True
            return r
        except OSError as err:
            print("OUPS ! OS Error {0}".format(err))

def deconnect_to_LTE():
    try :
        pycom.rgbled(0xd5362a) # red
        lte.disconnect()
        print("Disconnected!")
        lte.detach()
        print("Dettached!")
        lte.deinit()
    except OSError as err:
        print('Oups..')

[kjm]

urequest line 86 is where it tries to read your servers response after the data is sent. It attempts to crack open the response to extract the status code with .split(None, 2). So my guess is the problem is in the server response. Maybe try inserting a print(l) before line 86 in the library so you can see what the response is?

[andreas]

Dear @Tgasser,

two things come to mind.

1. Holding your code (you are using urequest.post) against the current version of the urequests module, it looks like they don't match. Can I humbly ask you which variant you are using?

2. Looking at the example of data:

   {'1': {'mac': b'XXXX', 'reg1': {'timestamp': (2020, 1, 2, 14, 15, 1, 221154, None), 'data': b'XXXX'}
   

   I am not sure whether native Python datetime objects like (2020, 1, 2, 14, 15, 1, 221154, None) will actually serialize well using ujson.

With kind regards,
Andreas.

[kjm]

@andreas said in urequest IndexError:

I am not sure whether native Python datetime objects like (2020, 1, 2, 14, 15, 1, 221154, None) will actually serialize well using ujson.

Never mind exotica like python datetime objects, we've found that json.dumps() in ujson just doesn't work period. If you look at the output it generates there are a few double quotes begrudgingly inserted around strings & that's about it! We've had to hand code our dictionaries into json to send them with urequests.

The one thing you can say about micropython & it's crap libraries, is it hammers home just how well python works!

[andreas]

Dear @kjm,

@kjm said in urequest IndexError:

we've found that json.dumps() in ujson just doesn't work period.

Are we talking about the canonical modujson module here? It is hard to believe that.

We've had to hand code our dictionaries into json to send them with urequests.

I feel deeply sorry for that. Do you have an example program which demonstrates the issue you have been observing?

We are using json.dumps() within the core of the Terkin Datalogger telemetry library [1] and this probably processed a few thousands of messages successfully on our workbench alone [2].

Apart from [3], I really can't find anything on the MicroPython issue tracker which tells anything about ujson.dumps() creating invalid JSON.

With kind regards,
Andreas.

[1] https://github.com/hiveeyes/terkin-datalogger/blob/master/terkin/telemetry.py#L274-L275
[2] https://swarm.hiveeyes.org/grafana/d/0CJxruMZz/amo-fipy-workbench?orgId=2&from=1559868149581&to=1576953115727
[3] https://github.com/micropython/micropython/issues/4790

[kjm]

@andreas We're gonna have Tgasser complaining about thread drift Andreas. But, by way of example, if we send a hand coded list like so;

r=get('http://our.server/api/v2/observations?request=[{"LPtank":17},{"BRpump":2020}]')

we get a status_code 200 reply from our server. But if we try;

r=get('http://our.server/api/v2/observations?request=', json=[{'LPtank':17},{'BRpump':2020}])

we get a status_code 400 reply.

[crumble]

@kjm

sounds like the second call moved the parameter from the UEL into the body.

[Tgasser]

@kjm @andreas Hello, thank you for your answers.

If I print the line 82 here's the result: b''
and after : []

I don't use a variant of the library: https://github.com/micropython/micropython-lib/tree/master/urequests. Is there a problem doing a post with this library?

There are no conversion problems with ujson.dumps. I checked the conversion result with https://jsonlint.com/.

[kjm]

@crumble said in urequest IndexError:

@kjm

sounds like the second call moved the parameter from the UEL into the body.

In python we'd do;

r=get('http://our.server/api/v2/observations',  params=[('request', json.dumps([{"LPtank":17},{"BRpump":2020}]))])

to generate the required

'http://our.server/api/v2/observations?request=[{"LPtank":17},{"BRpump":2020}]'

url. But urequests won't do params and we've never been able to figure out a Tgasser style query of the form

r=get('http://our.server/api/v2/observations?request', json=.........)

that works. I assumed it was because json.dumps was faulty but he has verified it's output so I guess the problem is elsewhere.

[andreas]

Dear @kjm,

@kjm said in urequest IndexError:

r=get('http://our.server/api/v2/observations?request=[{"LPtank":17},{"BRpump":2020}]')

we get a status_code 200 reply from our server. But if we try;

r=get('http://our.server/api/v2/observations?request=', json=[{'LPtank':17},{'BRpump':2020}])

we get a status_code 400 reply.

We would have to look at the server code why that is. I can say for sure both things are not equal. The first request uses the query parameter request where you are putting in some JSON. The second one will put the JSON into the request body.

Apart from that, submitting measurement or other data using GET requests is considered bad practice. Imagine if some crawler would pick up any URLs of you?

So, the best way to submit data using POST would be:

data = [
    {'LPtank':17},
    {'BRpump':2020}
]
response = urequests.post('http://our.server/api/v2/observations', json=data)

Of course, the server will have to accept the data in that format.

With kind regards,
Andreas.

[kjm]

@andreas I take your point Andreas. The LPtank & BRpump data is indeed put on the server by 2 other GPYs using post. The get is from a third GPY trying to access those uploads. But I fear I have hijacked Tgassers thread.

Tgasser, b"[] is pretty much a null reply. What does print(r.status_code) give for that reply?

[Tgasser]

@kjm I can't print the answer since it crashed before.

    l = s.readline()
        #print(l)
        l = l.split(None, 2)
        status = int(l[1])
        reason = ""
        if len(l) > 2:
            reason = l[2].rstrip()
        while True:
            l = s.readline()
            if not l or l == b"\r\n":
                break
            #print(l)
            if l.startswith(b"Transfer-Encoding:"):
                if b"chunked" in l:
                    raise ValueError("Unsupported " + l)
            elif l.startswith(b"Location:") and not 200 <= status <= 299:
                raise NotImplementedError("Redirects not yet supported")
    except OSError:
        s.close()
        raise

    resp = Response(s)
    resp.status_code = status
    resp.reason = reason
    return resp

[kjm]

@Tgasser Good point. Can't help I'm afraid. I don't have any background with node-red & all my woes with urequests are from no reply from the server or a non 200 status_code reply. Never had a reply without a status_code. Do you have access to the server side code?
PS: You should do an r.close() before return r

[andreas]

Dear @Tgasser,

@Tgasser said in urequest IndexError:

I don't use a variant of the urequests library.

I just was asking because in line 86, there is if len(l) > 2 and I wouldn't know how this could croak with IndexError: list index out of range.

@Tgasser said in urequest IndexError:

All I notice is that the problem appears when the data is very important.

Funny thing! Does this mean it works sometimes actually?

For me, it looks like the server doesn't answer at all but closes the connection immediately after receiving the request. Would that be possible?

With kind regards,
Andreas.

[crumble]

@andreas

I agree that the len part look s odd. But I my money on the message length. All network interfaces beside WLAN are designed for tiny messages. This one will become quite huge when you replace the anonymous fake data with real one.

I assume that this payload will be copied together with the protocol data into a buffer, which is used by the modem. This may explain why it fails only with "important" data.