Dash Core v0.14 on Mainnet

[camosoul]

I wonder if this pull request from 2 days ago is related your described "failing to propagate messaging" or if it is unrelated : https://github.com/dashpay/dash/pull/2967

Maybe two different issues... What knocks a node off may or may not be the same that that makes it permabanned with no hope even after re-register... Scrap the node and start over, hope it doesn't get singled out... But, from what I'm seeing, everything will eventually be banned until there's only one MasterNode left (maybe two fighting over which one is to be banned), it just takes a while...

 

[GrandMasterDash]

Nothing wrong here and mine are not on amazon.

Really want to hear from what DCG has to say about this.

And why is it just two hits, seems a bit harsh.

 

[Rick Seeger]

Half my nodes on Vultr are banned even though they've never experienced any down time. All of them have a PoSe penalty > 0. I parsed the debug.log to produce this chart - definitely linear, possibly exponential. I don't think it's worth the effort to re-establish my nodes until this is resolved. There's clearly a major problem here.

 

[tungfa]

Half my nodes on Vultr are banned even though they've never experienced any down time. All of them have a PoSe penalty > 0. I parsed the debug.log to produce this chart - definitely linear, possibly exponential. I don't think it's worth the effort to re-establish my nodes until this is resolved. There's clearly a major problem here.View attachment 9543

did you proper register your MN (determanistic and all) ?
BLS operator key ?

 

[AjM]

Half my nodes on Vultr are banned even though they've never experienced any down time. All of them have a PoSe penalty > 0. I parsed the debug.log to produce this chart - definitely linear, possibly exponential. I don't think it's worth the effort to re-establish my nodes until this is resolved. There's clearly a major problem here.View attachment 9543

Pinging @UdjinM6 @codablock

 

[flare]

Half my nodes on Vultr are banned even though they've never experienced any down time. All of them have a PoSe penalty > 0. I parsed the debug.log to produce this chart - definitely linear, possibly exponential. I don't think it's worth the effort to re-establish my nodes until this is resolved. There's clearly a major problem here.View attachment 9543

What is the output of one of your non-banned, PoSe > 0 nodes for

$ dash-cli masternode status

?

 

[Phil7]

From reading in here, by now it should be clear that it was gross negligence by Core, to enact & enforce this PoSe ban system on live net.

The PoSe scoring should have been introduced first without any immediate banning effects, only for the purpose to be closely
monitored & observed over the course of a grace period, whether the score is really behaving the way it is expected to work and without any flaws or severe bugs.
Only after a close observation period of at least 1-2 months, and if the score behaved just as expected, it should have been enacted & enforced with actual bans.

Seems like for some MNO this is turning out to be a real nightmare.
Especially because it seems to be hard or even outright impossible to fix, if a MN (or the PoSe system itself) produces no specific log-files in order to exactly pinpoint
to the failure (with exact incidence, timestamp etc.) and the recommended measures to repair it.

 

[codablock]

Can everyone who is affected please add these lines to dash.conf of the masternodes:

debug=llmq
debug=llmq-sigs
debug=llmq-dkg

Can you also give me protx hashes of failed MNs that you believe should not have failed? You can send these in private to me.
So far, all cases that have been investigated by core members turned out to be a misconfiguration. If there really is an issue, we'll try to figure out what it is ASAP.

 

[qwizzie]

I guess we have found at least one other metric thats influencing the PoSe Penalty score : misconfiguration of the deterministic masternode.
It will be interesting to see if this is a major cause of masternodes getting PoSe banned or if there are v14 code problems that need fixing.

In the mean time i suggest to owners of banned masternodes to double-check their deterministic masternode setup, particularly with regards to
BLS operator key setup and if their Sentinel has been updated. Also it can be handy to do a "./dash-cli masternode list full | grep -e IPADDRESS" command

 

[flare]

I guess we have found at least one other metric thats influencing the PoSe Penalty score : misconfiguration of the deterministic masternode.
It will be interesting to see if this is a major cause of masternodes getting PoSe banned or if there are v14 code problems that need fixing.

I can only speak for myself, but for me the code is working flawless. The code was tested for weeks on testnet and was deployed to mainnet when we were sure of its production quality.

On mainnet the PoSe system works as designed - and i am maintaining 100s of masternodes for my customers. Actually I had 3 cases of PoSe penalties the last days and in all cases the collateral owner accidentally changed the operator pubkey, leading to mismatching configuration.

So before calling for pitchforks: Check if the masternodeblsprivkey in your dash.conf matches the operator pubkey first.

 

[qwizzie]

I can only speak for myself, but for me the code is working flawless. The code was tested for weeks on testnet and was deployed to mainnet when we were sure of its production quality.

On mainnet the PoSe system works as designed - and i am maintaining 100s of masternodes for my customers. Actually I had 3 cases of PoSe penalties the last days and in all cases the collateral owner accidentally changed the operator pubkey, leading to mismatching configuration.

So before calling for pitchforks: Check if the masternodeblsprivkey in your dash.conf matches the operator pubkey first.

Personally i have not experienced any PoSe penalty on my masternodes, so i'm inclined to believe that unintended misconfiguration of masternodes by their owner / operator could have a large impact
on these masternodes getting banned. Makes sense to me. The PoSe Penalty score just brings visibility to it.

 

[Figlmüller]

No PoSe penalty problems here either, except for usual DoS Attacks which require proper mitigation.

 

[qwizzie]

@camosoul : i remember you using a restart script / command that lets your masternodes restart every "x" time automatically.
Is that still the case ? I'm wondering if that could have either an impact on your PoSe score or obscure a possible misconfiguration.

 

[masternube]

@flare I also had a MN banned once when I changed the operator key (thread). Could you explain the point of being able to change the operator key (using protx update_registrar) if it will get you banned anyway?

 

[Figlmüller]

@camosoul : i remember you using a restart script / command that lets your masternodes restart every "x" time automatically.
Is that still the case ? I'm wondering if that could have either an impact on your PoSe score or obscure a possible misconfiguration.

What's the point in restarting masternodes regularly? Using the stop command or by killing the process with a SIGTERM?
I generally would recommend to use scripts which only start the daemon when it is not running (based on the PID file or by process name in a single node setup).

 

[JGCMiner]

No PoSe errors for me either. Looking at Dashninja, this seems to be the case for the vast majority of other masternodes as well. So I agree that rather than grabbing pitchforks — those having issues should take advantage of @codablock’s offer if they cannot troubleshoot the issue on their own.

@flare I also had a MN banned once when I changed the operator key (thread). Could you explain the point of being able to change the operator key (using protx update_registrar) if it will get you banned anyway?

PoSe banning was enabled a few days ago but your post was from April. I don’t see the relation.

 

[splawik21]

All good here, no PoSe_penalty taken (fingers crossed)

 

[qwizzie]

What's the point in restarting masternodes regularly? Using the stop command or by killing the process with a SIGTERM?
I generally would recommend to use scripts which only start the daemon when it is not running (based on the PID file or by process name in a single node setup).

I used to stop my masternode and restart it every two weeks (before v13) as dashd would for some mysterious reason crash on me after two weeks, got restarted by my restart script and then ended up
in a false "masternode start required" state. Which then required a manual deletion of the mncache.dat file and a dashd restart to get that masternode active again.
At the time i thought the crashing of my dashd was related to my server's RAM, and as a precaution i restarted the server every two weeks to prevent these crashings and associated extra work.

Since v13 i no longer have any problems (mncache is no longer local and my dashd is running stable for long periods of time), so i just use monit to monitor and restart my dashd if necessary.
So i also dont really see a need (anymore) to restart masternodes on a regular basis.

 

[masternube]

@JGCMiner I guess an additional cause for PoSe banning was introduced recently, but my node was PoSe banned in April after I changed the operator key, so PoSe banning definitely existed. If you think it's off topic here, I'd appreciate an answer on the other thread.

 

[JGCMiner]

@JGCMiner I guess an additional cause for PoSe banning was introduced recently, but my node was PoSe banned in April after I changed the operator key, so PoSe banning definitely existed. If you think it's off topic here, I'd appreciate an answer on the other thread.

I see. However, as this is a v14 release thread I would rather keep this discussion to banning types just introduced to minimize confusion.

Hopefully a dev can get back to you in the other thread.