"watchdog-expired" on all masternodes after 12.1?

Just asked around, it sounds like it's a connectivity issue due to the network upgrading and should sort it's self out once things stabilise. It wont effect payments while enforcement is off and if they all check out ok with 'dash-cli masternode status' then all should be good. Might be worth checking them here just to see what another part of the network is seeing:
https://www.dashninja.pl/
 
  • Like
Reactions: AjM
Hmm... after 5 hours Watchdog_expired, they are now again enabled....
Dashninja still shows 50% is expired but remote wallet say they are enabled all 100%.

Edit: aaand now they are again expired, going to force start command from remote.
 
Ok, found bug, i am 95% sure, this is it.

If dash.conf rpcuser and/or rpcpassword include any special character, like !"#¤%&/,
sentinel wont work like it should be, my 50% nodes was expired because of this, others did not included special chars.

Its bed time for me, tomorrow i put bug report into GIT.
 
I'm getting the same error:

~/.dash/sentinel$ venv/bin/python bin/sentinel.py
-342: non-JSON HTTP response with '401 Unauthorized' from server
Cannot connect to dashd. Please ensure dashd is running and the JSONRPC port is open to Sentinel.

tried a number of things and still no result. tearing my bloody hair out here! lol
 
AjM said:
Hmm... after 5 hours Watchdog_expired, they are now again enabled....
Dashninja still shows 50% is expired but remote wallet say they are enabled all 100%.

Edit: aaand now they are again expired, going to force start command from remote.
Click to expand...

There shouldn't be any need to if they're ok on the servers
Walter said:
I'm getting the same error:

~/.dash/sentinel$ venv/bin/python bin/sentinel.py
-342: non-JSON HTTP response with '401 Unauthorized' from server
Cannot connect to dashd. Please ensure dashd is running and the JSONRPC port is open to Sentinel.

tried a number of things and still no result. tearing my bloody hair out here! lol
Click to expand...

At least you've got some left to tear out! ;) Are you in the .dash folder or the .dashcore folder? The new one is .dashcore.

EDIT: Guessing you're using these instructions for Sentinel, linking them just in case:
https://github.com/dashpay/sentinel

Everything is the same except the new folder name and Sentinal installation.
 
Maybe only I'm so stupid but... do You modify this below or just copy paste into cron like I do?

*/5 * * * * cd /home/YOURUSERNAME/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2&>1 >sentinel-cron.log
*/5 * * * * cd /home/---------HERE---------/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2&>1 >sentinel-cron.log
 
maky said:
Maybe only I'm so stupid but... do You modify this below or just copy paste into cron like I do?

*/5 * * * * cd /home/YOURUSERNAME/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2&>1 >sentinel-cron.log
*/5 * * * * cd /home/---------HERE---------/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2&>1 >sentinel-cron.log
Click to expand...

You need to modify it, just one line so delete the last and replace YOURUSERNAME with the name of your home folder, ie. /home/maky/.dashcore/sentinel
 
Yeah, I did it already. I hoped that this is cause of our problem but it is not. I change it to correct and masternodes go to ENABLED but minute later some of them go WATCHDOG state again...

PS.
It seems that for me problem is solved. If not I let You know.
 
maky said:
Yeah, I did it already. I hoped that this is cause of our problem but it is not. I change it to correct and masternodes go to ENABLED but minute later some of them go WATCHDOG state again...
Click to expand...

A few things to try, go into the sentinel folder and run this:
Code: 
SENTINEL_DEBUG=1 ./venv/bin/python bin/sentinel.py

That will run sentinel manually with debugging on, it should just run second with a lot of weird looking output, is so that part's ok.

Another to try is:
Code: 
dash-cli masternode status

If all is good it will give a few lines of details and the last line should be:
Code: 
"status": "Masternode successfully started"
 
@moocowmoo helped me out with the problem I had. It ended up being a syntax issue. "RPCUSER= " needs to NOT have a space between the = sign and the username supplied. I'm pretty sure that's what he said... It was a really dumb mistake tbh. He was fantastic though, really appreciated him helping like that :)
 
Last edited:
Bug confirmed, i am 100% sure about this, all my nodes are ok now.

v12.1
If in dash.conf rpcuser and/or rpcpassword include any special character, like !"#¤%&/,
sentinel wont work like it should be, and masternode status become to watchdog_expired,
only normal chars are working ok.

I did have this problem, after fixed/removed all special characters, no problem any more.

Bug report: https://github.com/dashpay/dash/issues/1323
 
AjM said:
Bug confirmed, i am 100% sure about this, all my nodes are ok now.

v12.1
If in dash.conf rpcuser and/or rpcpassword include any special character, like !"#¤%&/,
sentinel wont work like it should be, and masternode status become to watchdog_expired,
only normal chars are working ok.

I did have this problem, after fixed/removed all special characters, no problem any more.

Bug report: https://github.com/dashpay/dash/issues/1323
Click to expand...

Just to comment on this - I actually setup new MNs with brand new rpcuser and rpcpassword for 12.1 with special characters and sentinel test script was failing. It does not support special characters.

Went back to plain letters and numbers and all humming along smoothly now.
 
I've noticed "Watchdog_expired" problems too. On the latest one I logged in to my masternode and ran in the sentinel directory:

venv/bin/python bin/sentinel.py

The status of that masternode changed from "Watchdog_expired" in my local wallet to "Enabled" so that seemed to fix it.

I'm wondering whether it's anything to do with the amount of time between Sentinel polls in the cron job. I've set mine to 5 minutes in accordance with the instructions at https://dashpay.atlassian.net/wiki/display/DOC/Updating+to+12.1+-+Masternodes , ie....

*/5 * * * * cd /home/YOURUSERNAME/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2>&1 >> sentinel-cron.log

but I see that the actual sentinel github page at https://github.com/dashpay/sentinel indicates 2 minutes, ie....

*/2 * * * * cd /home/YOURUSERNAME/sentinel && ./venv/bin/python bin/sentinel.py >/dev/null 2>&1

Does anyone know whether 5 minutes or 2 minutes or some other time is best, and whether this may be anything to do with the "Watchdog_expired" issues?
 
TreasureSeeker said:
I've noticed "Watchdog_expired" problems too. On the latest one I logged in to my masternode and ran in the sentinel directory:

venv/bin/python bin/sentinel.py

The status of that masternode changed from "Watchdog_expired" in my local wallet to "Enabled" so that seemed to fix it.

I'm wondering whether it's anything to do with the amount of time between Sentinel polls in the cron job. I've set mine to 5 minutes in accordance with the instructions at https://dashpay.atlassian.net/wiki/display/DOC/Updating+to+12.1+-+Masternodes , ie....

*/5 * * * * cd /home/YOURUSERNAME/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2>&1 >> sentinel-cron.log

but I see that the actual sentinel github page at https://github.com/dashpay/sentinel indicates 2 minutes, ie....

*/2 * * * * cd /home/YOURUSERNAME/sentinel && ./venv/bin/python bin/sentinel.py >/dev/null 2>&1

Does anyone know whether 5 minutes or 2 minutes or some other time is best, and whether this may be anything to do with the "Watchdog_expired" issues?
Click to expand...

I changed the Sentinel cronjob today from 5 to 2 minutes as i had some troubles with my own masternodes, after they got disconnected somehow without Dash Central noticing (i only noticed they got kicked off the network by checking on Dashninja.pl). During cold restart i also came across the Watchdog_expired notification, setting the cronjob to 2 minutes seems to have helped with that.

It took a long time though before Dashninja.pl and later on Dash Central indicated them as active again.
(slow network propogation i guess).

Edit : i'm not using any special characters in my dash.conf by the way
Edit : also i'm using heavily modified and duplicate monit restart software, which helps with my own setup (which is one server with multiple dedicated IP addresses).
I need to keep an eye on my monit implementation to see if it is doing what i hope its doing :confused:
 
Last edited:
*/2 * * * * cd /home/YOURUSERNAME/sentinel && ./venv/bin/python bin/sentinel.py >/dev/null 2>&1

Did you change YOURUSERNAME correct and the path is live?
 
Yes, username changed and path to sentinel is correct. Haven't seen any more watchdogs expiring since changing to a 2 minute cronjob rather than 5 mins.
 
You must log in or register to reply here.
Back
Top