Dash Core Group announces first v19 chain halt

[Monotoko]

I just used Dash Core v21 as example in case the Github pull request was indeed about BLS serialisation and BLS serialisation only (which i was not sure about). If you say that the Github pull request was indeed only about BLS serialisation, then i believe that to be the case.

What i personally would have preferred to see is a focus only on supporting Evo nodes in Dash Core v19 and a focus only on supporting Dash Platform launch in Dash Core v20. Instead features have been added to both Dash Core v19 and Dash Core v20 that are not strictly necessary to support Evo nodes and the launch of Dash Platform. Having BLS upgraded from basic to legacy does little for the Dash network right now and will only be fully utilized down the road, when Dash Core Group focus on blockchain inter-exchangeability. If the BLS upgrade was pushed back to a later update, i wonder if Dash Core v19 would have been in development as long as it did. In my view (assumption) having added the BLS upgrade caused delay of Dash Core v19 and ultimately a severe bug in BLS serialisation also caused the Dash blockchain to halt.

I do think postponing the activation of Dash Core v19 and taking the time to fully fix the issue was the right action of Dash Core Group. I just think Dash Core Group is putting too many features into large Dash Core updates, often causing a Dash Platform delay and in this case even a Dash blockchain halt.

It's from legacy to basic, and it was mostly because of technical debt... we made BLS sigs before the basic/regular one came about. I even think (I'm not 100% sure on this though) Platform relies on basic rather than legacy, if you think about it you can pull an off the shelf module for basic or you can try to build one for the legacy format only Dash uses.... there's tradeoffs on all sides. There's also several backported Bitcoin PRs etc

We do have a lot of developers in the core team, and supporting 4k nodes was actually an easy thing to accomplish compared to other tasks... but those other tasks are important and if we leave them for v21 it will just add more and more technical debt to Dash and have most devs idling for 6-12 months.

I'd personally like to see more & better testing

 

[rion]

We discussed last week's network halt in today's Incubator Weekly show:

.

(several comments from this thread were referenced)

 

[TheNetworkIsBiggerThanYou]

So what is the role of Dash POW?

 

[qwizzie]

At this point i am still wondering what is best for the Dash network :

Introduce a fix for the BLS serialisation bug and pray that not any more unforeseen bugs surface on Dash Mainnet, related to either BLS serialisation or to the BLS upgrade in general. As a reminder : the BLS upgrade is merely a foundation/base that serves to develop new Dash feautures later down the road (blockchain inter exchangeability and communication for example), it is not essential to either start Evo nodes or to start Dash Platform.

Or use Dash Core v19.2 to just strip the BLS upgrade (BLS legacy --> BLS basic) from Dash Core v19 (revert back to BLS legacy) and move the BLS upgrade to a future Dash Core version, thereby providing developers more time to extensively test the BLS upgrade and its now known bug on Devnet and on Testnet.

The Dash network can then start setting up Evo nodes after activation of Dash Core v19, without the risk of possibly introducing yet another unforeseen BLS bug that could threaten the safety of the Dash network.

Preferably the BLS upgrade would then take place after the launch of Dash Platform (after Dash Core v20 activation).
Question is how interwoven the BLS upgrade (BLS legacy --> BLS basic) got by now (code-wise) and if it can be postponed that easily.

 

[xkcd]

At this point i am still wondering what is best for the Dash network :

Introduce a fix for the BLS serialisation bug and pray that not any more unforeseen bugs surface on Dash Mainnet

Qwizzie I am start to detect hessitation from yourself and others in the network on future upgrades, we must work to avoid being like Bitcoin and becoming too scared to ever attempt an upgrade again. I would also like to point out that v18 (quorum rotations) and v0.17, v0.16, v0.15, etc etc were all hardforks too and carried the same risks as this one. We survived all those, I am sure we will survive this one too. Also, thanks to your testing and mine and many others we have thoroughly tested v19. I have confidence that the chances of another similar bug are less than 0.01% at this stage.

 

[qwizzie]

Qwizzie I am start to detect hessitation from yourself and others in the network on future upgrades, we must work to avoid being like Bitcoin and becoming too scared to ever attempt an upgrade again. I would also like to point out that v18 (quorum rotations) and v0.17, v0.16, v0.15, etc etc were all hardforks too and carried the same risks as this one. We survived all those, I am sure we will survive this one too. Also, thanks to your testing and mine and many others we have thoroughly tested v19. I have confidence that the chances of another similar bug are less than 0.01% at this stage.

No, i am not hesitating about future upgrades, i am wondering why the BLS upgrade (which is not essential to setup Evonodes and not essential to start Dash Platform) can not simply be postponed to a later update, so devs can thoroughly test it on Devnet and Testnet.
That way we can launch Evo nodes and start Dash Platform on Dash Mainnet with a whole lot less risk.
And frankly i think that should have been the plan all along.

Dash Core v19 : Setup Evo nodes
Dash Core v20 : Launch Dash Platform & DashPay

Everything else (including BLS upgrade to BLS basic) : should have been planned for later Dash Core updates.

Now we are getting delay after delay after delay throughout this year and even a Dash blockchain stall, because of the BLS upgrade and other feautures that were added / are being added. And why ? Because Dash Core Group needs to keep developers busy ? Is that really in the best interest of the Dash network ?

 

[xkcd]

No, i am not hesitating about future upgrades, i am wondering why the BLS upgrade (which is not essential to setup Evonodes and not essential to start Dash Platform) can not simply be postponed to a later update, so devs can thoroughly test it on Devnet and Testnet.
That way we can launch Evo nodes and start Dash Platform on Dash Mainnet with a whole lot less risk.
And frankly i think that should have been the plan all along.

Dash Core v19 : Setup Evo nodes
Dash Core v20 : Launch Dash Platform & DashPay

Everything else (including BLS upgrade to BLS basic) : should have been planned for later Dash Core updates.

Now we are getting delay after delay after delay and even a Dash blockchain stall, because of the BLS upgrade and other feautures that were added / are being added. And why ? Because Dash Core Group needs to keep developers busy ? Is that really in the best interest of the Dash network ?

I mean the BLS upgrade was originally slated for the v20 upgrade along with the Platform activation code! But thankfully, QE insisted it be brought forward to v19 and clear the path for a slightly less complex v20. I am satisfied with this approach and as I said before, I am fine with the level of testing we have already done on v19, we should push it and it get over and done with.

 

[qwizzie]

This makes me wonder if the BLS upgrade was indeed introduced in v20 (so after Evo nodes were already setup and running), if that BLS serialization bug would have had the same dramatic effect.

At least we would not have had a Dash blockchain stall during activation of Dash Core v19 (i am assuming here that BLS legacy on itself does not contain blockchain halting bugs).

 

[vazaki3]

You are all claiming that it was a BLS error that caused the blockchain halt (I am still not conviced 100% about it, when I read the code of the bug fix), but lets we suppose it was solely a BLS bug. Why do you assume that BLS is a good thing when used in order to make all the masternodes deterministic?

Nobody asks, why the masternodes must be deterministic?
Why all the masternode IPS must be stored in irrefutable merkle trees, so that no masternode can deny it belongs to that tree?

Well this is yet another red line imposed by the agents, in order to control the Dash network.
The agents want to be 100% sure about the IPs that belong to the masternodes, so that it will be easy for them to shut down the dash network in case it turns against their interests.

I had a big fight with @codablock when the deterministic masternodes, the merkle trees holding IPs and the chainlocks were introduced, but I cannot recall it because the forum is not searchable anymore.
@codablock's visible posts are only 6 among the 100 he posted.

 

[GrandMasterDash]

I think Evan originally had the element of randomness because of a possible DDoS attack, and the deterministic masternodes prove that it never really was an issue. Payment schedules are more accurate now.

But I agree, the dependence on collateral and a table of IPs is looking old now. And although I would appreciate more privacy, the most important thing to do right now is to stop DCG publishing code and enable at least two groups of independent developers write code for the protocols written by DCG. Trying to convince MNOs to vote against DCG though is the real challenge.

 

[qwizzie]

Looks like the orginal fix (draft) to the Dash blockchain halt (https://github.com/dashpay/dash/pull/5392) is now superseded by the following github pull request (not draft) : https://github.com/dashpay/dash/pull/5403

I guess the waiting is now on the creation of Dash Core v19.2 branche / tag on Github.

Source : https://github.com/dashpay/dash/milestone/40

 

[slamdunk]

nothing's perfect, and maybe there is blame to be had, but I was encouraged with the response to the catastrophe. The problem was identified and corrected and the chain resumed. I'm not taking this for granted.

 

[qwizzie]

nothing's perfect, and maybe there is blame to be had, but I was encouraged with the response to the catastrophe. The problem was identified and corrected and the chain resumed. I'm not taking this for granted.

Not yet. They only postponed the activation of Dash Core v19 and got PoW blocks working again.
Basically with Dash Core v19.1 they just stabilized the Dash netwerk to a state prior to Dash Core v19 activation.

The fix itself to the Dash blockchain stall during v19 activation (Dash Core v19.2) is still being tested, some Github pull requests related to v19.2 are even still in draft mode (see post above). There is currently not even a Dash Core v19.2 branche / tag on Github :

just some referencing to it in the Github pull requests that focus on fixing the issue.

 

[GrandMasterDash]

nothing's perfect, and maybe there is blame to be had, but I was encouraged with the response to the catastrophe. The problem was identified and corrected and the chain resumed. I'm not taking this for granted.

The DCG update video was informative. However, the glaring omission was any apology or remorse. No talk of fundamental changes needed that such events will never happens again.

 

[qwizzie]

The DCG update video was informative.

I agree, very informative.

What i found interesting :

Four chains were detected.

One chain (v19.1) conflicted with latest chainlock and had to be discarded under DIP8
One chain (v19.0) was invalid (this chain had some blocks mined by a specific miner, but majority of nodes after reindexing rejected this chain
One chain (v19.1) had only 1 block mined and was later orphaned by a longer chain
One chain (v19.1) ended up accumulating the most work and later got ChainLocked

Source : www.youtube.com/watch?v=dACDhgztzUw (timestamp 14:20)

Also the feedback from Pasta explaining how developers made a mistake in Dash Core v19 with regards to BLS legacy scheme and BLS basic scheme (where devs mixed up some key usage that should have been done on BLS legacy mode, but instead was done on BLS basic mode and then could not be found in a certain map).

Source : www.youtube.com/watch?v=dACDhgztzUw (timestamp 32:50)

Still strange for me to hear, taking into account how three devs have looked at that specific code on Github (orginal dev that coded it, plus two more devs that is needed to review and approve the code on Github) and this mix-up still managed to slip through. I am not sure how much influence this mistake exactly had on the stall of the Dash blockchain during activation.

 

[vazaki3]

Still strange for me to hear, taking into account how three devs have looked at that specific code on Github (orginal dev that coded it, plus two more devs that is needed to review and approve the code on Github) and this mix-up still managed to slip through. I am not sure how much influence this mistake exactly had on the stall of the Dash blockchain during activation.

If I dont see the final bug fix, I believe nothing.
I am still suspecting that EVO and the DashPlatform is involved in the bug, and they dont want to admit it (or they dont know it yet).

 

[xkcd]

Just an FYI, the evo folder was introduced more than 4 years ago.

dash/src/evo at v0.13.x · dashpay/dash

Dash - Reinventing Cryptocurrency. Contribute to dashpay/dash development by creating an account on GitHub.

github.com

It's even in the Pivx project.

PIVX/src/evo at master · PIVX-Project/PIVX

Protected Instant Verified Transactions - Core wallet. - PIVX/src/evo at master · PIVX-Project/PIVX

github.com

Hopefully this dismisses the myth that the bug is directly related to the upcoming Platform release.

 

[vazaki3]

Hopefully this dismisses the myth that the bug is directly related to the upcoming Platform release.

I invented this myth, and I insist on it until the final bug fix is released.

@odysseasg said https://discord.com/channels/484546513507188745/484546513935269918/1114128285106974720

The original problem that caused the chain stall was partially reproduced: when disabling completly caching of MN lists, the problem was succesfully reproduced while testing. With caching on, we haven't found yet a pattern that causes the problem because of the differences between Mainnet and regtest/devnets and even Testnet) The good news is that we have a fix (currently under review by the whole team) https://github.com/dashpay/dash/pull/5403 which resolves the issue. We are getting closer to 19.2 release.

The 4k implementation was added in DIP3 here, last month, Apr 17 2023. So the DIP3 specification changed to support 4k collateral.

Are you sure that the 4k HCMN (HPMN) are not affected by the BLS signatures, as the new specs of DIP3 require?

In the case of type 1:

1. collateralOutpoint `hash` is null but an output with 4000 DASH is not present at position `n` of the ProRegTx outputs

2. collateralOutpoint `hash` is not null but an output with 4000 DASH can't be found in the UTXO specified by the `hash` and `n`

The new rules to determine the next block’s payee are:

1. Take the last block's payee. If it is a regular masternode (type 0), go to step 5.

2. If the last HPMN payee was already paid 4 blocks in a row, go to step 5.

3. If the last HPMN is not present in the current masternode list anymore, go to step 5.

4. Select last HPMN payee as the next block's payee. Return.

5. Take the valid masternode subset of the previous block

6. Sort the set in ascending order by "testHeight" and “ProRegTx hash”. “testHeight” is determined for each individual entry and equals the “last paid height” (or “registered height” if the masternode was never paid). If the masternode was PoSe-banned before and revived later, the “revival height” of the masternode is used instead of the “registered height”. If the “testHeight” of two masternodes is identical, the ProRegTx hash acts as a deterministic tie breaker.

7. Take the first entry of the resulting list and use this as the next block’s payee.

This calculation is performed for every block to enforce proper payment of the block's masternode reward (both payee and amount). Previously, masternode payment enforcement was skipped for superblocks, so miners received the full mining reward. With DIP3, masternode payment enforcement is also performed on superblocks.

I suspect that the (faulty?) new specs of DIP3 caused the blockchain halt.
Someone independant of DCG has to review the DIP3 specs and give an opinion on whether these specs may cause a blockchain halt.

 

[vazaki3]

Having said all the above, I am really astonished of the unprofessional practice of the DCG when they edit the specs of the old DIPs!

If you want to improve one or more DIPs, you propose it in a new DIP, so that your changes can be reviewed and be tested by independent reviewers/testers.