Pybytes Token and data visualization

[securigy]

Is it possible to set the token in FiPy without running firmware upgrade?
Is Pybytes free if I need to send data to it and create data visualization?
What are free or inexpensive alternatives that allow data visualization?

[andreas]

@andreas said in Pybytes Token and data visualization:

Data can be shipped en bloc using JSON or in a discrete way.

See also [1] and [2].

[1] https://getkotori.org/docs/handbook/acquisition/protocol/mqtt.html#single-readings
[2] https://getkotori.org/docs/handbook/acquisition/protocol/mqtt.html#multiple-readings

[andreas]

Dear @securigy,

thanks for your interest in Kotori. While I will definitively be happy to answer your questions thoroughly, I would like to divert them to reduce the noise about it on the Pycom user forum.

You might find some information already on [1] and maybe also [2]. Saying this, I want to humbly refer to [3,4] here, where I tried to explain some basics of Kotori to other people asking about the details. Otherwise, feel free to open yet another issue on GitHub for additional questions.

I believe most of your questions can be answered positively like "Kotori's DAQ stack will be able to do it". However, I put down some words on the high-frequency data ingest which is questionable and will require more thoughts and engineering.

@securigy said in Pybytes Token and data visualization:

How do I deliver data?

@securigy said in Pybytes Token and data visualization:

Is your broker Mosquitto?

Yes, we use Mosquitto as a MQTT broker and every feature regarding authentication and beyond will be available as accustomed. Data can be shipped en bloc using JSON or in a discrete way. We designed Kotori to be flexible with respect to these details.

@securigy said in Pybytes Token and data visualization:

How do I add/create my account, register the device and then see the data

We designed Kotori deliberately to omit any device registration steps on purpose. Through the MQTT topic design, it is multi-tenant / multi-channel out of the box. So, it is really just about submitting measurement data and see it coming within Grafana in a "wash & go" style.

This style resonates very much with our use-cases of having a hackable DAQ platform without any obstacles with respect to device onboarding and all that jazz. While this is definitively the opposite direction of "enterprise-grade" here, respective features might be added on top.

@securigy said in Pybytes Token and data visualization:

If I in the future plan to deliver relatively big chunks of data, that is, the sensor with 500 Hz.

While building the DAQ stack around Kotori, we also thought about these kinds of use-cases with high-frequency measurements and their respective ingesting mechanisms. Right now, I don't believe any solution around what we are doing here will be able to deliver that kind of performance.

However, respective requirements are fulfilled through industry-grade systems and we will be happy to look at these kinds of requirements in general by eventually providing respective "fast-path" mechanisms.

@securigy said in Pybytes Token and data visualization:

Do you think MQTT will be efficient for these kind of payloads?

While we haven't been able to investigate into that, we will be happy to look more into this. I believe with high-frequency ingest, everything will also depend on efficient payload serialization and database performance. These are advanced topics.

I hope this helps for now. Going into more details will probably spiral this thread out of control. However, I will be happy to stay in touch as it looks like we share a common interest here.

With kind regards,
Andreas.

[1] https://getkotori.org/
[2] https://github.com/daq-tools/kotori/issues
[3] https://github.com/daq-tools/kotori/issues/12
[4] https://github.com/daq-tools/kotori/issues/13

[robert-hh]

@securigy No. The builds I provided are deliberately WITHOUT pybytes, in contrast to those from the Pycom download pages.

[andreas]

@securigy said in Pybytes Token and data visualization:

I am also wondering if Pybytes are enabled/included in your build(s)

To quote myself here:

@andreas said in Pybytes Token and data visualization:

These builds don't have the pybytes modules compiled in anyway.

[securigy]

@andreas If I in the future plan to deliver relatively big chunks of data, that is, the sensor with 500 Hz gives me 10 seconds of data where each point is tuple...that could be about 100 KB (floats delivered as strings for more compact payload)...
Then do you think MQTT will be efficient for these kind of payloads (I always delivered just dozens of bytes in the past) ? Is your broker Mosquitto? Or sockets would be a better way? (delivering 1 tuple at a time will be quite an overhead, I think).
If this size is a problem for MQTT Broker, I supposed I could break it to smaller chunks, just not sure what freedom/control I have to stitch them together...

[securigy]

@andreas I looked at both.... ...few problems:

1. I don't understand German ;-)
2. If I ignore #1
   A. How do I deliver data? (what are MQTT Broker settings: URL/Port, other settings? Dow it support SSL/TLS? If yes, is it Basic Auth by credentials or through security certificates?)
   B. How do I add/create my account, register the device and then see the data (json? base64encoded?) then how to I do visualization? I have zero knowledge in Grafana...

[securigy]

@andreas I am also wondering if Pybytes are enabled/included in your build that Robert built on Nov 2, 2019?
I am referring the the issue raised here:
https://forum.pycom.io/topic/5331/building-compiling-micropython-with-pybytes

[andreas]

@securigy Yes, this software stack has to be installed somewhere, it is a complete lightweight but slick DAQ backend. Don't you have any iron floating around? However, we are operating a few of these services, two of them are publicly available at [1,2]. Let me know if I can be of help in any way on that aspect.

And yes, MQTT is a first citizen here, however data ingestion can also be made through HTTP.

[1] https://swarm.hiveeyes.org/
[2] https://weather.hiveeyes.org/

[securigy]

@andreas About Kotori... I tried to read/browse through it documentation. So correct me if I am wrong: it is a framework that consists of a few tools (some of them, like MQTT, I am very familiar with) that I need to install on a Linux machine somewhere in the cloud... I did not see that it is a specific web site that I need to register my device/s with... Correct? If yes, then it can be a problem for me - the web site I have (gravitystorms.com) on GoDaddy doe not provide access to the OS guts in order to build applications but just for building the visual UI... Grafana looks good though...

[andreas]

@andreas said in Pybytes Token and data visualization:

just pass the token into the respective API methods when using the pybytes modules

Maybe looking at [1] specifically and [2] in general might help you along.

[1] https://github.com/pycom/pycom-micropython-sigfox/blob/Dev/esp32/frozen/Pybytes
[2] https://github.com/pycom/pycom-micropython-sigfox/tree/Dev/esp32/frozen/Pybytes

[andreas]

@securigy said in Pybytes Token and data visualization:

I used the alternative builds you provided instead (the ones built by Robert).

These builds don't have the pybytes modules compiled in anyway.

@securigy said in Pybytes Token and data visualization:

The most recent official FiPy firmware release 1.20.1.1r is flawed, as we all know, (and makes FiPy unusable)

I wouldn't say so. Pycom will probably not ship flawed things for building the Pybytes platform altogether. It might just be that we don't know how to exactly invoke them.

@securigy said in Pybytes Token and data visualization:

then I am out of luck, and cannot use Pybytes

I believe you will be able to just pass the token into the respective API methods when using the pybytes modules in a manual way?

[securigy]

@andreas If setting the token in FiPy firmware can be done only by firmware upgrade then I am out of luck, and cannot use Pybytes - and here is why:
The most recent official FiPy firmware release 1.20.1.1r is flawed, as we all know, (and makes FiPy unusable) and I used your new builds instead (built by Robert). When you do that using the firmware upgrade tool pycom_firmware_update_1.6.1.exe
then the box that forces Pybytes registration disappears... and all the UI that allows entering the token is skipped...

[andreas]

@securigy said in Pybytes Token and data visualization:

Is it possible to set the token in FiPy without running firmware upgrade?

I don't know about it. Maybe it can be put directly into pybytes_config.json.

[securigy]

@andreas Cool. Thank you Andreas. I will get to it shortly.
I can't wait to put PySense data in graphical form - as a proof of concept...Then I can really get to more serious high-volume data like vibrations, EKG, etc...
What about my 1st question? "Is it possible to set the token in FiPy without running firmware upgrade?"

[andreas]

@securigy said in Pybytes Token and data visualization:

What are free or inexpensive alternatives that allow data visualization?

On the one hand, I don't want to advertise our own stuff here against what Pycom is building with the Pybytes platform on the visualization aspect. On the other hand, the Pybytes platform handles many more aspects regarding device registration and onboarding, OTA etc. than just visualization.

So, you might want to look at Kotori. It really does not compete with Pybytes in any way as it really just covers the aspect of ingesting and visualizing your measurement/telemetry data with Grafana. It is free software, written in Python and you might like it.

For more information about it, you might want to search the internet for kotori {daq,grafana,mqtt}. I will be happy to answer any questions coming from this.

Disclaimer: I am the main author of Kotori.