Frozen Sys

[neilh10]

While soak testing, "tu_rc_test06" it stopped processing.
The test system was started on Apr 8th midday, and it stopped working on Apr 19th.
Its based on 0.28.3.
At a guess, based on the trace, it looks as if the RTC has never woken it up.
I also removed the uSD, and copied the files quickly before returning it to the Mayfly, and same date from .csv

I have had a debug FTDI cable monitoring it, and have a complete trace to when it stopped.
Battery was good when it went to sleep at 4.01V Apr 19th, and measured 3.95V on Apr25th

It woke on
... zzzZZ Awake @ 2021-04-19T16:07:00-08:00

and last actions as recorded on the FTDI debug terminal was to post successfully to MMW

---

POST /api/data-stream/ HTTP/1.1
Host: data.envirodiy.org
TOKEN: e89538ae-7ff0-4897-a396-9b10c0fadfc3
Content-Length: 556
Content-Type: application/json

{"sampling_feature":"31461753-cd5d-4ed0-99f7-17713c5e1e0e","timestamp":"2021-04-19T16:00:00-08:00","b404d204-f127-44d4-869c-50c058412999":1072,"d395b5f7-c9c4-4c99-a919-4c4869cbc288":3662.521,"013225a5-b2ba-4f0d-ba99-3f38599e0672":-0.223,"32b6fb1f-4688-4fb0-ada7-323753536c08":4.0138,"a6e185cb-32a4-4c05-bc7d-c14e87bc9701":4.0054,"be84dc61-86dd-4ff9-9c44-f1d25bd78f6b":4.003,"a4361b13-a4b4-4d53-b1c8-1398d73ceeec":0.14103,"b40cdca7-22c5-4d6e-91db-4eb77ebf2179":19.17,"e5a6b171-ae06-4fc0-bc3f-68820f9e448c":0.068,"9a848614-1dc2-4372-890b-68d2b48d9407":17.01}

-- Response Code -- 201 waited 2132 mS Timeout 5000
---logDataAndPubReliably Complete----------
Going to sleep. Ram( 6127 ) ZZzzz...
Watchdog disabled. barksUntilReset 150 <--WatchDogAVR

---

And then appears to have never woken up.

FROM: TU_RC_test06a_2021-04-08.csv
Data Logger: TU_RC_test06a
Data Logger File: TU_RC_test06a_2021-04-08.csv
Location: "beta test06 Monitor"
LoggingInterval (min): 15
Tz: -8
BatteryType: 2
Sw Name: tu_LT5_lte LT500,Acculevel. Sw Build: src\tu_xx01.cpp Apr 8 2021 09:47:07
ModularSensors vers 0.28.3
Board: mayfly rev:'0.5bR04' sn:'MAYFLY180347'
Digi XBee3 Cellular LTE-M Sn OK HwVer 4B48 FwVer 11417
RS485 Wingboard STC3100 Sn 14017d611 for electricEnergy
RS485 Wingboard STC3100 Sn 14017d611 for electricEnergy
SDI Sensor IN-SITU LT500 306 Sn 0000525528 SDI12 Version tbd for waterDepth
SDI Sensor IN-SITU LT500 306 Sn 0000525528 SDI12 Version tbd for temperature
Sampling Feature UUID: 31461753-cd5d-4ed0-99f7-17713c5e1e0e,

---Boot. Sw Build: src\tu_xx01.cpp Apr 8 2021 09:47:07 .
Sw Name: tu_LT5_lte LT500,Acculevel
ModularSensors version 0.28.3
TinyGSM Library version 0.10.9
From eeprom boot Name[ mayfly ] rev[ 0.5bR04 ] sn[ MAYFLY180347 ]
From eeprom common: Logger File Name[ TU_RC_test06a ]
logging interval= 15 minutes, Tz= -8 battery type= 2 battery mAhr= 4400
Loc= "beta test06 Monitor"
From eeprom Network: Network Type= 0
APN= VZWINTERNET
WiFiId= WIFIID_NONE WiFiPwd= WIFIPWD_NONE
CollectReadings= 8 SendOffset(min)= 7
From eeprom Provider: Provider Type= 1 PROVIDER_MMW CloudId= data.enviroDIY.com
PROVIDER_MMW Reg Token= e89538ae-7ff0-4897-a396-9b10c0fadfc3
PROVIDER_MMW Sampling Feature= 31461753-cd5d-4ed0-99f7-17713c5e1e0e
PROVIDER_MMW timerPostTout(ms)= 5000
PROVIDER_MMW timerPostPace(ms)= 1000
PROVIDER_MMW postMax= 0
PROVIDER_MMW UUIDs (none unless listed), Max= 12
0 ] ITROLL_DEPTH_UUID = a4361b13-a4b4-4d53-b1c8-1398d73ceeec
1 ] ITROLL_TEMP_UUID = b40cdca7-22c5-4d6e-91db-4eb77ebf2179
2 ] KellerXxlevel_Height_UUID = e5a6b171-ae06-4fc0-bc3f-68820f9e448c
3 ] KellerXxlevel_Temp_UUID = 9a848614-1dc2-4372-890b-68d2b48d9407
4 ] Batt_UUID = be84dc61-86dd-4ff9-9c44-f1d25bd78f6b
5 ] SampleNumber_UUID = b404d204-f127-44d4-869c-50c058412999
6 ] Volt0_UUID = a6e185cb-32a4-4c05-bc7d-c14e87bc9701
7 ] STC3100Volt_UUID = 32b6fb1f-4688-4fb0-ada7-323753536c08
8 ] STC3100used1_mAhr_UUID = 013225a5-b2ba-4f0d-ba99-3f38599e0672
9 ] STC3100avlbl_mAhr_UUID = d395b5f7-c9c4-4c99-a919-4c4869cbc288
BatV Good = 0.00
---parseIni Start
Parse ini; No file ms_cfg.ini
COMMON LoggerId Set: TU_RC_test06a
COMMON Logginterval: 15
COMMON Battery Type: 2
COMMON Battery mAhr: 4400
Logger timezone is set to UTC -8
NETWORK APN was 'hologram', now set to 'VZWINTERNET'
NETWORK CollectReadings 8
NETWORK sendOffset_min 7
PROVIDER_MMW CloudId data.enviroDIY.com
PROVIDER_MMW Registration Token e89538ae-7ff0-4897-a396-9b10c0fadfc3
PROVIDER_MMW Samping Feature 31461753-cd5d-4ed0-99f7-17713c5e1e0e
PROVIDER_MMW timerPost Tout(ms) 5000
PROVIDER_MMW timerPost Pacing(ms) 1000
PROVIDER_xx ITROLL_DEPTH_UUID a4361b13-a4b4-4d53-b1c8-1398d73ceeec -> ITROLLdepth
PROVIDER_xx ITROLL_TEMP_UUID b40cdca7-22c5-4d6e-91db-4eb77ebf2179 -> ITROLLtemp
PROVIDER_xx KellerXxlevel_Height_UUID e5a6b171-ae06-4fc0-bc3f-68820f9e448c -> kellerAccuDepth
PROVIDER_xx KellerXxlevel_Temp_UUID 9a848614-1dc2-4372-890b-68d2b48d9407 -> kellerAccuTemp
PROVIDER_xx Batt_UUID be84dc61-86dd-4ff9-9c44-f1d25bd78f6b -> Battery
PROVIDER_xx SampleNumber_UUID b404d204-f127-44d4-869c-50c058412999 -> SampNum
PROVIDER_xx Volt0_UUID a6e185cb-32a4-4c05-bc7d-c14e87bc9701 -> extVolt0
PROVIDER_xx STC3100Volt_UUID 32b6fb1f-4688-4fb0-ada7-323753536c08 -> Stc3100_V
PROVIDER_xx STC3100used1_mAhr_UUID 013225a5-b2ba-4f0d-ba99-3f38599e0672 -> STC3100usedmAhr
PROVIDER_xx STC3100avlbl_mAhr_UUID d395b5f7-c9c4-4c99-a919-4c4869cbc288 -> STC3100avblmAhr
---parseIni complete

RTC timezone is set to UTC
Battery Type=2
Thresholds USEABLE=3.50V LOW=3.60V MEDIUM=3.70V GOOD=3.80V
Watch-dog timeout is set for 1200 sec with the interrupt firing 150 times before the reset. <--WatchDogAVR
Watchdog Enabled, Interrupt will fire 150 times before the system resets. <--WatchDogAVR
Current RTC time is: 2021-04-08T12:07:17-08:00
All variable UUID's appear to be correctly formed.

b404d204-f127-44d4-869c-50c058412999 -> SampNum
d395b5f7-c9c4-4c99-a919-4c4869cbc288 -> STC3100avblmAhr
013225a5-b2ba-4f0d-ba99-3f38599e0672 -> STC3100usedmAhr
32b6fb1f-4688-4fb0-ada7-323753536c08 -> Stc3100_V
a6e185cb-32a4-4c05-bc7d-c14e87bc9701 -> extVolt0
be84dc61-86dd-4ff9-9c44-f1d25bd78f6b -> Battery
a4361b13-a4b4-4d53-b1c8-1398d73ceeec -> ITROLLdepth
b40cdca7-22c5-4d6e-91db-4eb77ebf2179 -> ITROLLtemp
e5a6b171-ae06-4fc0-bc3f-68820f9e448c -> kellerAccuDepth
9a848614-1dc2-4372-890b-68d2b48d9407 -> kellerAccuTemp

[neilh10]

My test06 system froze for a 2nd time. This time I pressed the User Button, which is also tied to an interrupt, and started up again. 
I have made some changes to the RTC methods that ensure that the 5 of RTC registers are as expected before sleeping.
This performs a read on the registers, and if not as expected then writes to them.
neilh10/Sodaq_DS3231@9aec45c
See LoggerBase.cpp changes
29b8a28

[neilh10]

Been running for over week with above updates, and nothing showed.

[neilh10]

[2021-05-24 08:38:02] test06 froze again, right after 8 POSTs.
Possibly because of a race condition in
initializing RTC to interrupt every minute
setting the port to create interrupt on change of status (not re-entrant)
clearing the RTC status bit to allow further interrupts

[neilh10]

Frozen on startup on test06/
[2021-10-11 07:28:00] ---Boot(0) Sw Build: a\src\tu_xx01.cpp Oct 8 2021 14:39:46 b'rel1_dvlp1m'
[2021-10-11 07:28:00] Sw Name: tu_LT5_lte LT500
[2021-10-11 07:28:00] ModularSensors version 0.30.0
[2021-10-11 07:28:00] TinyGSM Library version 0.11.4
[2021-10-11 07:28:00] From eeprom boot Name[ mayfly ] rev[ 0.5bR04 ] sn[ MAYFLY180347 ]
[2021-10-11 07:28:00] From eeprom common: Logger File Name[ TU_RC_test06a ]
[2021-10-11 07:28:00] logging interval= 2 minutes, Tz= -8 battery type= 2 battery mAhr= 4400
[2021-10-11 07:28:00] Loc= "beta test06 Monitor"

....
[2021-10-11 07:28:00] From eeprom Provider: Provider Type= 1 PROVIDER_MMW CloudId= data.enviroDIY.com
.....
[2021-10-11 07:28:01] 7 ] Volt0_UUID = 32b6fb1f-4688-4fb0-ada7-323753536c08

FAILURE FROZEN while still in startup
MANUAL RESET

[2021-10-11 09:01:27]
[2021-10-11 09:01:27] ---Boot(0) Sw Build: a\src\tu_xx01.cpp Oct 8 2021 14:39:46 b'rel1_dvlp1m'

[neilh10]

On same system as above a few dates later
[2021-10-13 19:39:00.677] ---logDataAndPubReliably Complete----------
[2021-10-13 19:39:00.706] Going to sleep. Ram( 7349 / 5892 ) ZZzzz...
[2021-10-13 19:39:00.708] Watchdog disabled. barksUntilReset 51 <--WatchDogAVR
Frozen : notified by email and saw early 14th
On 14 investigating JTAG options, but couldn’t find any

Tried pressing User button, and no action. (I think its disabled)

[2021-10-14 20:01:24.867] manually pressing RESET started it.
[2021-10-14 20:01:24.867] ---Boot. Sw Build: src\tu_xx01.cpp Oct 12 2021 17:21:15 .
[2021-10-14 20:01:24.894] Sw Name: tu_LT5_lte LT500
[2021-10-14 20:01:24.895] ModularSensors version 0.30.0
[2021-10-14 20:01:24.896] TinyGSM Library version 0.11.4
[2021-10-14 20:01:24.897] From eeprom boot Name[ mayfly ] rev[ 0.5bR04 ] sn[ MAYFLY180347 ]
[2021-10-14 20:01:24.941] From eeprom common: Logger File Name[ TU_RC_test06a ]