NEW_START_REQUIRED instantly becomes WATCHDOG_EXPIRED

camosoul

camosoul

Well-known member
Had a hard drive blow out on one of my pizza boxes. Had config backups.

Installed new drive.

Set up everything as before using the backup config files.

Everything checks out.

Broadcast MN start, they instantly go to inactive. No timer. Nothing. Instantly inactive.

I switched in fresh mnprivkeys... No effect. MNs don't even get a chance. Instaban.

What gives?

EDIT: dashninja shows the "last seen" counter resets, but stays "inactive."
 
Last edited:
I have tested.


NEW_START_REQUIRED ----> WATCHDOG_EXPIRED ---> with a mnp(ping) ---> PRE_ENABLED ---> ENABLED
 
chaeplin said:
cd /home/coind/sentinel/ && venv/bin/python bin/sentinel.py -b
Click to expand...
Code: 
Invalid Masternode Status, cannot continue.
Code: 
2017-05-24 08:31:24 CActiveMasternode::ManageStateRemote -- NOT_CAPABLE: Masternode in NEW_START_REQUIRED state
The only thing that changed was installing a new hard drive.

P.S. Thanks for the new feature on dashmnb! ;-)
 
Last edited:
This just happened for no apparent reason:
Code: 
2017-05-24 08:53:26 CActiveMasternode::ManageStateRemote -- STARTED!
But it's still inactive and not pinging.
 
7 of them claim to be relaying ping, but dashninja disagrees, therefore dashmnb disagrees; no pings seen.

None of my other nodes see these pings, either.

I deleted the chain data and allowed it to pull from the network, to prove they're not in a bubble. Took about 2 hours to pull chain... Claims to be relaying ping... But issue persists even though I did not use bootstrap.dat. They act like they're working, but both dashninja and nodes that have been running the whole time still don't see them even though they claim to be up and running and pinging.

Are they stuck in a network fragment large enough to be independent and functional?

chainz.cryptoid.info agrees with height...

Makes no damn sense. To my understanding of how this is supposed to work, what I am observing is impossible.

Noticed something odd on one of the nodes.... All of them but this one have a solid wall of ping relay messages (cat ~/.dashcore/debug.log | grep SendMasternodePing). But, this one seems to waffle back and forth:
Code: 
2017-05-23 15:44:07 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 15:54:14 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 16:04:21 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 16:14:29 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 16:24:36 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 16:28:38 CActiveMasternode::SendMasternodePing -- Too early to send Masternode Ping
2017-05-23 16:34:43 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 16:44:50 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 16:54:57 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 17:05:04 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 17:21:09 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 17:26:20 CActiveMasternode::SendMasternodePing -- Too early to send Masternode Ping
2017-05-23 17:31:33 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 17:41:40 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 17:51:46 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 18:01:53 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 18:11:59 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 18:22:06 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 18:32:13 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 18:42:20 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 18:52:26 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 19:02:32 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 19:42:08 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 19:43:40 CActiveMasternode::SendMasternodePing -- Too early to send Masternode Ping
2017-05-23 19:48:02 CActiveMasternode::SendMasternodePing -- Too early to send Masternode Ping
2017-05-23 19:52:41 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 20:02:48 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 20:12:55 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
2017-05-23 20:23:01 CActiveMasternode::SendMasternodePing -- Relaying ping, collateral=CTxIn(COutPoint(19153a0c2430508ba5df68d49d73592e3ff41d53f27cbc441f822a5c79bf26d4, 0), scriptSig=)
WTF?

dashmnb declares NEW_START_REQUIRED, but the nodes claim to be running and pinging like nothing is wrong...
 
Last edited:
On the surface, it looks like the problem is dashninja. Since dashmnb pulls it's data from dashninja, that's why it's wrong.

But.

My other, functioning nodes reflect this same problem; my other nodes don't believe that these nodes exist, either.
 
Another random thing
Code: 
2017-05-25 00:08:09 CActiveMasternode::ManageStateRemote -- NOT_CAPABLE: Broadcasted IP doesn't match our external address. Make sure you issued a new broadcast if IP of this masternode changed recently.
Untrue. I've checked it many times. The IP is just fine. The mnprivkey is not goofed, neither is the txid/index of the 1000DASH. Nothing wrong, but it keeps pretending that it doesn't match.
 
I gave up fighting against the defective network.

I moved all the collateral to new addresses. Rebuilt masternode.conf with new mnprivkeys and the new TXIDs and indexes. Put the mnprivkeys in the dash.conf files of the nodes.

Upon announce, they switch from NEW_START_REQUIRED to WATCHDOG_EXPIRED. They begin pinging, but remain WATCHDOG_EXPIRED in masternode list.

I guess I've been kicked out of DASH. I don't know how yet, but I am being tracked, uniquely identified, and prevented from running masternodes.
 
Last edited:
Just let them run for a while and they should become enabled if sentinel is (was) configured correctly (before).
 
They were ENABLED for about 2 hours, and have now gone back to NEW_START_REQUIRED with no reason or explanation. They continue to operate as if nothing is wrong, but the network silently bans them for no reason.
 
Last edited:
All I did was cuss at it, and the entire rack spontaneously went PRE_ENABLED for the first and only time since the hard drive failure. Which should not be possible since they were in NEW_START_REQUIRED status.

A simple hardware problem should not result in such bizarre behavior.

I have a theory that fits, but don't have solid evidence... May never have.

A few years back, before DRK... I had a completely false blockchain fed to my BTC client. I only discovered it when all broadcast TXes would never show up in the memory pool of any chain explorers. I enabled tor and the whole chain rewound. I ran traceroute and discovered all my traffic was being tunneled through a non-ISP gateway in Northern Virginia. The guvtrash were MIMing me with my ISP's cooperation. Sloppy job, too, or I never would have noticed. The entrance was not discernible, but the exit was. This is why I became a proponent of tor's fringe benefits; you can't MIM tor traffic even if you can perform traffic analysis on exit nodes with global observation power... I can't say that's what is happening here, but the shoe fits... If my server is in a sloppy bubble, it would explain some of this.
 
Last edited:
ok, so:
1. local wallet and remote node are synced (block count matches, sync process finished on both sides)
2. masternode start-alias <alias>
3. what's in remote logs?

EDIT: can you send remote debug.log to my email?
 
There was nothing strange in the debug output. The nodes were recognizing collateral and the announce, and they would start pining. They acted as if nothing was wrong, but the network was apparently not getting the ping messages. I dropped my rather simple firewall just in case, but it had no effect. There was something blocking certain traffic... Or, designed poorly enough that it failed to forward transparently.

It's working now. As mentioned, spontaneously. Makes no sense. It's as if someone did a new masternode start. But my node collateral addresses are in my trezor, so it could only have come from me. But, I didn't do it. I have, however, made several announcements that seemed to disappear into a black hole... Say, a stale announce that never made it to the network, recorded and played back later, so to speak...

Such as, maybe the server was firewalled by a sloppy MIM pass through and they gave up upon seeing that I was catching on and it wasn't working as transparently as they had hoped.

Again.

Still.

If guvtrash wants to spy, at least try not to suck at it...

Could have been holding the announce and released it when they gave up... Can't say for sure, but it smells very familiar.
 
Last edited:
You must log in or register to reply here.
Back
Top