Add persistent message storage and message log frame #3784

ndnestor · 2024-05-04T22:38:03Z

Added

Persistent message storage
Message RAM cache
Message log UI frame
Message protobuf

Changed

Trunk version to latest
Trunk linters' versions to latest
Touchscreen tap and double tap input events

Removed

Text message frame that only showed last received message

Video Demonstration

Here is a video demonstration of changes made. It is slightly outdated.

Recommended Future Changes

Replace current ACK UI frame with a symbol that is inline with the ACK'd message
Add "default auto-focus state" toggle to settings
Add "delete all messages" button to settings
Add setting to change how often messages are saved
Use ostream for more efficient string building in drawMessageLogFrame
Add external storage support (e.g. SD card)
Only run through message log frame drawing logic when what is drawn needs to be changed
Warn user when running out of storage
Delete only the oldest message when running out of storage

…ge pruning

CLAassistant · 2024-05-04T22:38:08Z

All committers have signed the CLA.

CLAassistant · 2024-05-04T22:38:08Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

todd-herbert

Looking good! Do you have any plans to implement some of the features (cycle through channel?) for devices which only have a single user-button?

src/graphics/Screen.cpp

GUVWAF · 2024-05-05T06:28:24Z

Very nice.

On which devices/screens has this been tested next to the T-Deck?

src/modules/TextMessageModule.cpp

src/modules/CannedMessageModule.cpp

src/mesh/NodeDB.cpp

.trunk/trunk.yaml

src/mesh/NodeDB.cpp

HarukiToreda · 2024-05-05T23:39:03Z

Tested on Heltec V3, wireless Paper, Wireless Tracker. Although the screen on the Tracker is the same size as the V3 i can view less messages per line, this is not an issue since we have the ability to scroll with the arrow keys. However the User button does not work anymore, no matter how many times it is pressed it remains on the same Message screen on all 3 devices. Arrow keys on the CardKB are the only way to go to the next screens.

meshtastic-bot · 2024-05-07T04:41:13Z

This pull request has been mentioned on Meshtastic. There might be relevant details there:

https://meshtastic.discourse.group/t/t-deck-2-3-6-firmware-how-to-change-channel/12378/8

ndnestor · 2024-05-09T00:57:30Z

Very nice.

On which devices/screens has this been tested next to the T-Deck?

@GUVWAF
I've only tested it on the T-Deck so far since that's all I own.

todd-herbert · 2024-05-09T04:10:42Z

Very nice.
On which devices/screens has this been tested next to the T-Deck?

@GUVWAF I've only tested it on the T-Deck so far since that's all I own.

What I might actually suggest is: instead of modifying the Screen class directly, write a new class which derives from Screen, then override what you need. Then, where Screen is instantiated, use a macro to instead use your derived class.

That way, you can focus on implementing the new features for just T-Deck right now, and gradually expand to more devices later on.

ndnestor · 2024-05-10T00:41:45Z

Very nice.
On which devices/screens has this been tested next to the T-Deck?

@GUVWAF I've only tested it on the T-Deck so far since that's all I own.

What I might actually suggest is: instead of modifying the Screen class directly, write a new class which derives from Screen, then override what you need. Then, where Screen is instantiated, use a macro to instead use your derived class.

That way, you can focus on implementing the new features for just T-Deck right now, and gradually expand to more devices later on.

The issue I see with making a T-Deck-specific device is that expanding would mean making classes for other models which seems impractical in the long run as new devices come out. For maintainability, one class that can generically deal with everything would be nice.

todd-herbert · 2024-05-10T01:03:09Z

Very nice.
On which devices/screens has this been tested next to the T-Deck?

@GUVWAF I've only tested it on the T-Deck so far since that's all I own.

What I might actually suggest is: instead of modifying the Screen class directly, write a new class which derives from Screen, then override what you need. Then, where Screen is instantiated, use a macro to instead use your derived class.
That way, you can focus on implementing the new features for just T-Deck right now, and gradually expand to more devices later on.

The issue I see with making a T-Deck-specific device is that expanding would mean making classes for other models which seems impractical in the long run as new devices come out. For maintainability, one class that can generically deal with everything would be nice.

I'm still suggesting using one class for all devices, but use a macro to select the new class. If a variant is built with the macro USE_PERSISTENT_MSG (or whatever suits), it will build with the new class, otherwise it will build with the existing Screen class.

Once it's confirmed that a new device works correctly with the new class, that generic macro can be added to its specific platformio.ini file

Have a look at how the Screen constructor does it. I think something like this might be useful

ndnestor · 2024-05-10T01:08:05Z

Very nice.
On which devices/screens has this been tested next to the T-Deck?

@GUVWAF I've only tested it on the T-Deck so far since that's all I own.

What I might actually suggest is: instead of modifying the Screen class directly, write a new class which derives from Screen, then override what you need. Then, where Screen is instantiated, use a macro to instead use your derived class.
That way, you can focus on implementing the new features for just T-Deck right now, and gradually expand to more devices later on.

The issue I see with making a T-Deck-specific device is that expanding would mean making classes for other models which seems impractical in the long run as new devices come out. For maintainability, one class that can generically deal with everything would be nice.

I'm still suggesting using one class for all devices, but use a macro to select the new class. If a variant is built with the macro USE_PERSISTENT_MSG (or whatever suits), it will build with the new class, otherwise it will build with the existing Screen class.

Once it's confirmed that a new device works correctly with the new class, that generic macro can be added to its specific platformio.ini file

Have a look at how the Screen constructor does it. I think something like this might be useful

Oh I see. That makes sense to me. Will do!

caveman99 · 2024-05-11T22:01:59Z

src/platform/esp32/architecture.h

@@ -87,6 +87,7 @@
 #define HW_VENDOR meshtastic_HardwareModel_TLORA_V2_1_1P8
 #elif defined(T_DECK)
 #define HW_VENDOR meshtastic_HardwareModel_T_DECK
+#define USE_PERSISTENT_MSG
 #elif defined(T_WATCH_S3)


This define is best moved to the variant file if you really want it only for the t-deck.

@caveman99 Would you prefer to try implement it everywhere? I'll withdraw my objections if you do think it'd be better.

I second this, the Tdeck UI will be out someday, but all other devices will be left behind, this is the best option we've gotten so far. Maybe have a new module and the ability to give the choice of turning this on or not. I've been using this on my V3 and so far no issues other than the ability to press the program button. No issues with long messages either cause I can scroll up and down to see them, something I couldn't do before. please don't block this for just the Tdeck

My idea initially was to maybe get it released for T-Deck first, and add other devices once they're tested. There's definitely a lot of edge case stuff that would need to be handled, and some devices probably shouldn't get this feature at all imo, but if the devs want to go ahead and try, I'll support the effort.

caveman99 · 2024-05-11T22:22:52Z

just a general question, since you seem to go to great lengths to implement this. I am not convinced the heavy filesystem operation this relys on will work on the embedded platforms when the number of files grows. You do know realize we have a new UI for the T-Deck ind evelopment that will mimick a phone app more than the current Oled UI? This seems like double effort.

ndnestor · 2024-05-11T23:57:57Z

just a general question, since you seem to go to great lengths to implement this. I am not convinced the heavy filesystem operation this relys on will work on the embedded platforms when the number of files grows. You do know realize we have a new UI for the T-Deck ind evelopment that will mimick a phone app more than the current Oled UI? This seems like double effort.

@caveman99 Yes, I've heard about that. When I first asked about this feature on the Discord server, I was told that it would, optimistically, take a year. Based on the progress done so far, I would guess that it's going to take decently longer than that. Given the utility of this feature, I figured it is worth it to have this not-so-fancy version for now and get the fancy version later.

bobricius · 2024-05-14T10:20:13Z

What about reserve some array in RAM for cca 10 messages ?

tropho23 · 2024-05-14T13:53:46Z

Even a rolling window of 10 messages for nrf52 would be great, for now until an external storage solution is developed. Tony Good

…

________________________________ From: Peter Misenko ***@***.***> Sent: Tuesday, May 14, 2024 06:20 To: meshtastic/firmware ***@***.***> Cc: tropho23 ***@***.***>; Mention ***@***.***> Subject: Re: [meshtastic/firmware] Add persistent message storage and message log frame (PR #3784) What about reserve some array in RAM for cca 10 messages ? — Reply to this email directly, view it on GitHub<#3784 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AQ7GUPQQ6Q3RDRBFYNBDIP3ZCHQPHAVCNFSM6AAAAABHHFLYOCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBZHAZDOMBZGU>. You are receiving this because you were mentioned.Message ID: ***@***.***>

todd-herbert · 2024-05-14T14:02:43Z

Even a rolling window of 10 messages for nrf52 would be great, for now until an external storage solution is developed. Tony Good

This would be a whole lot more straight-forward! (Assuming there's RAM to spare?)

tropho23 · 2024-05-14T14:13:52Z

Of course more than 10 would better 🙂 My ideal is the last 30 messages if available memory allows. Tony Good

…

________________________________ From: todd-herbert ***@***.***> Sent: Tuesday, May 14, 2024 10:03 To: meshtastic/firmware ***@***.***> Cc: tropho23 ***@***.***>; Mention ***@***.***> Subject: Re: [meshtastic/firmware] Add persistent message storage and message log frame (PR #3784) Even a rolling window of 10 messages for nrf52 would be great, for now until an external storage solution is developed. Tony Good This would be a whole lot more straight-forward! — Reply to this email directly, view it on GitHub<#3784 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AQ7GUPRVZX5T46LK6GXRKHTZCIKRTAVCNFSM6AAAAABHHFLYOCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJQGMZDSMJZHA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

ndnestor · 2024-05-31T04:04:17Z

We've solved the throttling problem by implementing a database file for each "category" instead of individual message files in a file structure. This allows us to save messages in batches which store in a buffer that gets saved every 30 seconds to a single file, limiting our number of writes. We also now execute saving to disk asynchronously which makes the whole process safe from packet-loss.

HarukiToreda · 2024-06-05T07:38:54Z

I tried building this to test but it's not building. Missing "message.pb.h"

In file included from src/DisplayFormatters.h:2,
                 from src/DisplayFormatters.cpp:1:
src/mesh/NodeDB.h:13:10: fatal error: mesh/generated/meshtastic/message.pb.h: No such file or directory
 #include "mesh/generated/meshtastic/message.pb.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from src/GPSStatus.h:2,
                 from src/gps/GPS.h:5,
                 from src/ButtonThread.cpp:4:
src/mesh/NodeDB.h:13:10: fatal error: mesh/generated/meshtastic/message.pb.h: No such file or directory
 #include "mesh/generated/meshtastic/message.pb.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
*** [.pio\build\heltec-v3\src\DisplayFormatters.cpp.o] Error 1
*** [.pio\build\heltec-v3\src\ButtonThread.cpp.o] Error 1
In file included from src/modules/Telemetry/Sensor/TelemetrySensor.h:7,
                 from src/modules/Telemetry/Sensor/INA219Sensor.h:6,
                 from src/power.h:41,
                 from src/power.cpp:13:
src/mesh/NodeDB.h:13:10: fatal error: mesh/generated/meshtastic/message.pb.h: No such file or directory
 #include "mesh/generated/meshtastic/message.pb.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
*** [.pio\build\heltec-v3\src\Power.cpp.o] Error 1
In file included from src/mesh/Default.h:2,
                 from src/PowerFSM.cpp:11:
src/mesh/NodeDB.h:13:10: fatal error: mesh/generated/meshtastic/message.pb.h: No such file or directory
 #include "mesh/generated/meshtastic/message.pb.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
*** [.pio\build\heltec-v3\src\PowerFSM.cpp.o] Error 1
In file included from src/RedirectablePrint.cpp:2:
src/mesh/NodeDB.h:13:10: fatal error: mesh/generated/meshtastic/message.pb.h: No such file or directory
 #include "mesh/generated/meshtastic/message.pb.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
*** [.pio\build\heltec-v3\src\RedirectablePrint.cpp.o] Error 1
In file included from src/airtime.cpp:2:
src/mesh/NodeDB.h:13:10: fatal error: mesh/generated/meshtastic/message.pb.h: No such file or directory
 #include "mesh/generated/meshtastic/message.pb.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
*** [.pio\build\heltec-v3\src\airtime.cpp.o] Error 1
In file included from src/buzz/buzz.cpp:2:
src/mesh/NodeDB.h:13:10: fatal error: mesh/generated/meshtastic/message.pb.h: No such file or directory
 #include "mesh/generated/meshtastic/message.pb.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
*** [.pio\build\heltec-v3\src\buzz\buzz.cpp.o] Error 1
In file included from src/SerialConsole.cpp:2:
src/mesh/NodeDB.h:13:10: fatal error: mesh/generated/meshtastic/message.pb.h: No such file or directory
 #include "mesh/generated/meshtastic/message.pb.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
*** [.pio\build\heltec-v3\src\SerialConsole.cpp.o] Error 1

ndnestor · 2024-06-06T02:37:29Z

I tried building this to test but it's not building. Missing "message.pb.h"

In file included from src/DisplayFormatters.h:2,
                 from src/DisplayFormatters.cpp:1:
src/mesh/NodeDB.h:13:10: fatal error: mesh/generated/meshtastic/message.pb.h: No such file or directory
 #include "mesh/generated/meshtastic/message.pb.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from src/GPSStatus.h:2,
                 from src/gps/GPS.h:5,
                 from src/ButtonThread.cpp:4:
src/mesh/NodeDB.h:13:10: fatal error: mesh/generated/meshtastic/message.pb.h: No such file or directory
 #include "mesh/generated/meshtastic/message.pb.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
*** [.pio\build\heltec-v3\src\DisplayFormatters.cpp.o] Error 1
*** [.pio\build\heltec-v3\src\ButtonThread.cpp.o] Error 1
In file included from src/modules/Telemetry/Sensor/TelemetrySensor.h:7,
                 from src/modules/Telemetry/Sensor/INA219Sensor.h:6,
                 from src/power.h:41,
                 from src/power.cpp:13:
src/mesh/NodeDB.h:13:10: fatal error: mesh/generated/meshtastic/message.pb.h: No such file or directory
 #include "mesh/generated/meshtastic/message.pb.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
*** [.pio\build\heltec-v3\src\Power.cpp.o] Error 1
In file included from src/mesh/Default.h:2,
                 from src/PowerFSM.cpp:11:
src/mesh/NodeDB.h:13:10: fatal error: mesh/generated/meshtastic/message.pb.h: No such file or directory
 #include "mesh/generated/meshtastic/message.pb.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
*** [.pio\build\heltec-v3\src\PowerFSM.cpp.o] Error 1
In file included from src/RedirectablePrint.cpp:2:
src/mesh/NodeDB.h:13:10: fatal error: mesh/generated/meshtastic/message.pb.h: No such file or directory
 #include "mesh/generated/meshtastic/message.pb.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
*** [.pio\build\heltec-v3\src\RedirectablePrint.cpp.o] Error 1
In file included from src/airtime.cpp:2:
src/mesh/NodeDB.h:13:10: fatal error: mesh/generated/meshtastic/message.pb.h: No such file or directory
 #include "mesh/generated/meshtastic/message.pb.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
*** [.pio\build\heltec-v3\src\airtime.cpp.o] Error 1
In file included from src/buzz/buzz.cpp:2:
src/mesh/NodeDB.h:13:10: fatal error: mesh/generated/meshtastic/message.pb.h: No such file or directory
 #include "mesh/generated/meshtastic/message.pb.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
*** [.pio\build\heltec-v3\src\buzz\buzz.cpp.o] Error 1
In file included from src/SerialConsole.cpp:2:
src/mesh/NodeDB.h:13:10: fatal error: mesh/generated/meshtastic/message.pb.h: No such file or directory
 #include "mesh/generated/meshtastic/message.pb.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
*** [.pio\build\heltec-v3\src\SerialConsole.cpp.o] Error 1

@HarukiToreda I was told to remove that file from the PR since it can be generated using nanopb. You can manually add message.pb.c from here and message.pb.h from here by copying and pasting the file contents.

ndnestor added 9 commits March 25, 2024 22:53

Implement persistent message log

5010439

Modify protobufs submodule

57834a4

Update protobufs

06ebfc4

Add scrolling, message protobuf, cache invalidation, and stored messa…

35af455

…ge pruning

Merge branch 'master' into dev

d8c4c92

Add documentation and log statements

1fff782

Merge branch 'master' of https://github.com/meshtastic/firmware

b0c0c8c

Fix channel log frame info text formatting

62cb934

Comply with trunk formatting

1563b45

ndnestor marked this pull request as draft May 4, 2024 22:47

todd-herbert requested changes May 5, 2024

View reviewed changes

src/graphics/Screen.cpp Show resolved Hide resolved

src/graphics/Screen.cpp Show resolved Hide resolved

GUVWAF reviewed May 5, 2024

View reviewed changes

src/modules/TextMessageModule.cpp Outdated Show resolved Hide resolved

GUVWAF reviewed May 5, 2024

View reviewed changes

src/modules/CannedMessageModule.cpp Outdated Show resolved Hide resolved

GUVWAF reviewed May 5, 2024

View reviewed changes

src/mesh/NodeDB.cpp Outdated Show resolved Hide resolved

Merge branch 'master' into master

243baf5

caveman99 requested changes May 5, 2024

View reviewed changes

.trunk/trunk.yaml Outdated Show resolved Hide resolved

src/mesh/NodeDB.cpp Outdated Show resolved Hide resolved

thebentern mentioned this pull request May 5, 2024

[Feature request]: Access previously received messages on-device #1985

Open

ndnestor added 3 commits May 9, 2024 22:04

Revert trunk version to 1.20.1

3e6500f

Merge branch 'master' of https://github.com/meshtastic/firmware

9fed261

Remove unnecessary includes

362c0ee

Add descriptive comment to Message protobuf message

a9c49e8

ndnestor mentioned this pull request May 10, 2024

Add message protobuf meshtastic/protobufs#497

Open

2 tasks

ndnestor added 7 commits May 10, 2024 22:32

Remove /msgs folder on factory reset

1bbc688

Use pre-existing remove directory recursive implementation

6a9074a

Add USE_PERSISTENT_MSG macro

569082d

Fully comment message.proto and limit category int size

d533a3c

Merge branch 'master' of https://github.com/meshtastic/firmware

de32db2

Revert changes to generated files

ed50877

Merge branch 'master' of https://github.com/ndnestor/firmware

0e10832

caveman99 reviewed May 11, 2024

View reviewed changes

ndnestor and others added 4 commits May 11, 2024 21:41

Allow for dynamic primary channel in info string

6f362a8

Merge branch 'master' of https://github.com/meshtastic/firmware

47cd09c

Move USE_PERSISTENT_MSG macro from architecture.h to variant.h

e82bc8b

Merge branch 'master' into master

c2c7acf

ndnestor added 3 commits May 30, 2024 23:48

Save messages asynchronously and in batches

bc922d9

Merge branch 'master' of https://github.com/meshtastic/firmware

6a695b2

Merge branch 'master' of https://github.com/ndnestor/firmware

fddb08e

ndnestor marked this pull request as ready for review June 1, 2024 00:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add persistent message storage and message log frame #3784

Add persistent message storage and message log frame #3784

ndnestor commented May 4, 2024 •

edited

CLAassistant commented May 4, 2024 •

edited

CLAassistant commented May 4, 2024

todd-herbert left a comment

GUVWAF commented May 5, 2024

HarukiToreda commented May 5, 2024

meshtastic-bot commented May 7, 2024

ndnestor commented May 9, 2024

todd-herbert commented May 9, 2024 •

edited

ndnestor commented May 10, 2024

todd-herbert commented May 10, 2024

ndnestor commented May 10, 2024

caveman99 May 11, 2024

todd-herbert May 12, 2024

HarukiToreda May 12, 2024 •

edited

todd-herbert May 12, 2024

caveman99 commented May 11, 2024

ndnestor commented May 11, 2024

bobricius commented May 14, 2024

tropho23 commented May 14, 2024 via email

todd-herbert commented May 14, 2024 •

edited

tropho23 commented May 14, 2024 via email

ndnestor commented May 31, 2024

HarukiToreda commented Jun 5, 2024

ndnestor commented Jun 6, 2024

Add persistent message storage and message log frame #3784

Are you sure you want to change the base?

Add persistent message storage and message log frame #3784

Conversation

ndnestor commented May 4, 2024 • edited

Added

Changed

Removed

Video Demonstration

Recommended Future Changes

CLAassistant commented May 4, 2024 • edited

CLAassistant commented May 4, 2024

todd-herbert left a comment

Choose a reason for hiding this comment

GUVWAF commented May 5, 2024

HarukiToreda commented May 5, 2024

meshtastic-bot commented May 7, 2024

ndnestor commented May 9, 2024

todd-herbert commented May 9, 2024 • edited

ndnestor commented May 10, 2024

todd-herbert commented May 10, 2024

ndnestor commented May 10, 2024

caveman99 May 11, 2024

Choose a reason for hiding this comment

todd-herbert May 12, 2024

Choose a reason for hiding this comment

HarukiToreda May 12, 2024 • edited

Choose a reason for hiding this comment

todd-herbert May 12, 2024

Choose a reason for hiding this comment

caveman99 commented May 11, 2024

ndnestor commented May 11, 2024

bobricius commented May 14, 2024

tropho23 commented May 14, 2024 via email

todd-herbert commented May 14, 2024 • edited

tropho23 commented May 14, 2024 via email

ndnestor commented May 31, 2024

HarukiToreda commented Jun 5, 2024

ndnestor commented Jun 6, 2024

ndnestor commented May 4, 2024 •

edited

CLAassistant commented May 4, 2024 •

edited

todd-herbert commented May 9, 2024 •

edited

HarukiToreda May 12, 2024 •

edited

todd-herbert commented May 14, 2024 •

edited