Advice to speed up write to SD card file - Datalogger

[JSmith]

Hello,

My Wipy 3.0 with expansion board V3.1 has a datalogger mode in its use case, whereby I append/write ~32 class object attributes/variables to a csv file at a frequency that I can choose (im using an interrupt to do this).

I ideally want the logging frequency to be as good as 100Hz (10ms between writes), but I am finding it takes 100ms to write the data, with my current write to file strategy.

Below is the existing code:

def log(alarm):
    global last_time_ms
    if sd_mounted:
        if data_file_version == 0:
            setup_log_file()
            last_time_ms = chrono3.read_ms()
        timenow_ms = chrono3.read_ms()
        timenow_s = timenow_ms/1000

        ADC_mV_str = ",".join(str(inst.mV_val_WMAfilt) for inst in ADCInput._ADCregistry[1:])
        ADC_engval_str = ",".join(str(inst.engineering_val) for inst in ADCInput._ADCregistry[1:])
        Output_str = ",".join(str(inst.desired_duty_perc) for inst in output_class._Outregistry[1:])
        CAN_str = ",".join(str(inst.engineering_value) for inst in CANSPN._registry[0:])

        with open('/sd/data_log_%s.csv' % data_file_version,'a') as f:
            f.write(str(timenow_ms)+","+str(timenow_s)+","+str(timenow_ms-last_time_ms)+","+ADC_mV_str+","+ADC_engval_str+","+Output_str+","+CAN_str +"\n")
        #print("logged dT: ", str(timenow_ms-last_time_ms))
        last_time_ms = timenow_ms
    else:
        print("cannot log data, SD card not mounted succesfully on boot!")

As you can see I open the file at each interval in append mode, and write in the new variable string set.

Can anyone suggest a faster alternative (e.g. writing to the flash?, or simply adding the new string to a massive/accumulating string which I then write to the card periodically?, or just keeping the file open instead of opening each time I want to log?)

Im a novice so my current solution maybe slow, but I do like simply appending to a pre-existing file since the data is safe if the device were to malfunction or loose power.

Many thanks

[JSmith]

@robert-hh thanks for your suggestion. I have attempted to improve the minimum sapling frequency, by keeping the file open if the desired sampling freq < 100ms. Unfortunately, if I set the delta time to 50ms (updating the interrupt in via a IOT feed) the time between samples is still excessive at 90ms.

If I comment out the write operation so that the function isnt actually writing to sd, then the min delta T becomes 65ms (still long). I have found however that the bulk of this is due to the time to collect the variables (28 class attributes) and the convert to strings & concatenate.

def log(dTms):
    global last_time_ms
    global file_open_status
    global f
    if sd_mounted:
        if data_file_version == 0:
            setup_log_file()
            last_time_ms = chrono3.read_ms()
        timenow_ms = chrono3.read_ms()
        timenow_s = timenow_ms/1000

        ADC_mV_str = ",".join(str(inst.mV_val_WMAfilt) for inst in ADCInput._ADCregistry[1:])
        ADC_engval_str = ",".join(str(inst.engineering_val) for inst in ADCInput._ADCregistry[1:])
        Output_str = ",".join(str(inst.desired_duty_perc) for inst in output_class._Outregistry[1:])
        CAN_str = ",".join(str(inst.engineering_value) for inst in CANSPN._registry[0:])

        if dTms >= 100:                                                                 # non-volatile mode if samp freq <10Hz (closes file each time)
            if file_open_status:
                f.close                                                                 # this closes the file as switching from high samp freq to slow/non volatile mode
                file_open_status = False
            with open('/sd/data_log_%s.csv' % data_file_version,'a') as f:
                f.write(str(timenow_ms)+","+str(timenow_s)+","+str(timenow_ms-last_time_ms)+","+ADC_mV_str+","+ADC_engval_str+","+Output_str+","+CAN_str +"\n")
        else:                                                                           # volatile mode which keeps file open between logs in high freq use cases
            if not file_open_status:
                f = open('/sd/data_log_%s.csv' % data_file_version,'a')
                file_open_status = True
            f.write(str(timenow_ms)+","+str(timenow_s)+","+str(timenow_ms-last_time_ms)+","+ADC_mV_str+","+ADC_engval_str+","+Output_str+","+CAN_str +"\n")
        print(str(timenow_ms-last_time_ms))
        last_time_ms = timenow_ms
    else:
        print("cannot log data, SD card not mounted succesfully on boot!")

Could you advise if my approach of collecting my variables from class instance attributes from other modules is slow practice? I cannot see a more suitable approach since I want to access many variables within multiple modules.

Kind regards

[robert-hh]

@JSmith Do not open & close the file all the time. That takes a lot of time. Keeping the file open has obviously the risk of loosing data in case of a fatal error. But at least exceptions could be covered, such that the file is closed when it happens.
Buffering is already done by the FAT library, so you do not have to do that.

[claus]

I would probably try to write the data into a memory queue from the main thread and have a separate thread that regularily checks the queue and writes everything it finds to the sd, removing the saved items from the queue. You will have to use locking on the queue to avoid race conditions though.

A simpler option would be to wait for a good number of data to accumulate and write every 10/100 steps or so to the disk. This probably will not take much longer than writing a single row.