6/20 RC3 Post-mortem

[flare]

Ah... we have...

I have reindexed 3 testnet nodes.
One of them got this.

ubuntu@ip-172-31-30-7:~$  grep REORGANIZE .darkcoin/testnet3/debug.log
2014-06-21 09:24:50 REORGANIZE: Disconnect 1 blocks; 00000000fd93e980730f288f1a640fc524b86f5e2fc99005bdf5b3a1d4d7aeea..
2014-06-21 09:24:50 REORGANIZE: Connect 2 blocks; ..000000018a59904ad1e02ae1cfdf30b17fb6f740c574ff31c46ab1e0b754b1d6

SetBestChain: new best=00000000fd93e980730f288f1a640fc524b86f5e2fc99005bdf5b3a1d4d7aeea  height=19451  log2_work=44.693617  tx=33603  date=2014-06-14 17:43:42 progress=0.993643

SetBestChain: new best=000000018a59904ad1e02ae1cfdf30b17fb6f740c574ff31c46ab1e0b754b1d6  height=19453  log2_work=44.693717  tx=33608  date=2014-06-14 17:47:22 progress=0.993646

So regular reindexing of the testnet clients and grepping for REORGANIZE should now be part of the test plan :smile:

 

[stilgars]

I remember that quick fork, THAT fork went smoothly actually...smoothest one of them all perhaps. The forks with the long announcement times have failed the worst no?

Look, Mintpal caused a panic today being not fully prepared and they had plenty of time...like a month! They probably could have done as well with a last minute phone call. So It's not a lead time issue right?

I'm concerned if the plan from here is actually just "Third times a charm!"

Exactly, non responsive under-concerned ppl will still be non responsive under-concerned ppl in two weeks time.
In fact, waiting for so long is underproductive since the migration effort will lose some momentum and fall under the radar in the next coming days, and awareness should be raised *again* from scratch within the last couple of days.

EDIT: but I missed the part stating that testnet will now reproduce more accurately mainnet. Good news, just a little puzzled since I thought rigorous testing was performed after last time already.

 

[flare]

EDIT: but I missed the part stating that testnet will now reproduce more accurately mainnet. Good news, just a little puzzled since I thought rigorous testing was performed after last time already.

It was, but obviously we missed a testcase with producing blocks with identical hash but different payee - will be adressed now.

Any suggestions for further testcases you can think about? Constructive input welcome ;-)

 

[JGCMiner]

It was, but obviously we missed a testcase with producing blocks with identical hash but different payee - will be adressed now.

Any suggestions for further testcases you can think about? Constructive input welcome ;-)

Well is it about testcases -- or testnet not being sufficient to serve as a testbed for mainnet?

The first time the fork failed a much larger effort was made on testnet (more testcases, purposely trying to break it, etc.). In a sense that effort paid off because the cause of the issue this time was not outdated clients -- an improvement, but in the end the fact is that the fork still failed.

What makes you think that once again re-doubling efforts will improve the situation? Especially since the issue this time was the exact type of issue that is hard to find on testnet because of the lack of hashpower.

Just like there was a difference in testing effort between attempt 1 and attempt 2, there needs to be a difference this time as well. Otherwise there is no reason to think that the next payment fork will be any different.

IMO, the first thing to do is somehow make testnet more similar to mainnet. AlexGR suggested merged mining -- not sure how doable that is, but it would be excellent if possible. That may not be the only method, but just relying on volunteer miners is not likely to improve the current situation by very much.

tl;dr You can't expect the next fork attempt to be any different if the same level of testing on the same platform is carried out. Testnet needs to be improved. Not just for the masternode payment testing, but for all the anonymity experiments down the road.

Edit: Sorry for overstepping my bounds as someone who hasn't put in the same amount of time as you and others, but I felt that the above needed to be stated strongly.

 

[Bizmonger]

No more launches ok, just stop with the launching stuff...you used up all your launches...stop now.
And no more dates, youre out of dates too..

From now on, how about you do "Live testing" or maybe "active audits"..."security simulation".... Something so that when you find a bug, its a success and not another one of these Epic Crypto Fails.

Im only saying this cause i love you, but yeah, dont put us through another one of these launch date face-smashers. At this point we have to assume the next release will also have problems, so basically im suggesting you build that into the plan.

Red-Shionobi has a point.
Why not reconsider the way that updates are released.
Why not deliver to mainnet more frequently with just devs and testers up-to-date with the network at first and eventually the secret is out. So what if it's hard on the network.
It's called continuous-integration. If something breaks, then simply revert. It's okay.

 

[luke997]

Red-Shionobi has a point.
Why not reconsider the way that updates are released.
Why not deliver to mainnet more frequently with just devs and testers up-to-date with the network at first and eventually the secret is out. So what if it's hard on the network.
It's called continuous-integration. If something breaks, then simply revert. It's okay.

No.

Continuous integration is not the same as frequent releases of production software.
But anyway, frequent released can be applied to not distributed systems, or to distributed system with strict update control.
If you don't control updating process of the clients, you have a allow a liberal amount of time to update.

Would you want to have, for example a 3 versions of accounting system clients sending different data structure to the back end?

 

[flare]

Well is it about testcases -- or testnet not being sufficient to serve as a testbed for mainnet?

Both :smile:

What makes you think that once again re-doubling efforts will improve the situation? Especially since the issue this time was the exact type of issue that is hard to find on testnet because of the lack of hashpower.

Any additional effort will improve the situation, as long you follow some kind of plan - that's called continous improvement process: Plan - Do - Check - Act. That's the way you improve situations.

Just like there was a difference in testing effort between attempt 1 and attempt 2, there needs to be a difference this time as well. Otherwise there is no reason to think that the next payment fork will be any different.

That's why we add the testcases we missed to the testsuite and improve testnet to match mainnet even closer.

IMO, the first thing to do is somehow make testnet more similar to mainnet. AlexGR suggested merged mining -- not sure how doable that is, but it would be excellent if possible. That may not be the only method, but just relying on volunteer miners is not likely to improve the current situation by very much.

I fully agree with you here, and started a private discussion with eduffield and AlexGR two hours ago, since i really like the approch of making testnet a clone of mainnet - if it's feasabe to have merged mining for testcoins, that's the way to go.

tl;dr You can't expect the next fork attempt to be any different if the same level of testing on the same platform is carried out. Testnet needs to be improved. Not just for the masternode payment testing, but for all the anonymity experiments down the road.

I fully agree with you, never said that testing/testbed should not be improved.

Edit: Sorry for overstepping my bounds as someone who hasn't put in the same amount of time as you and others, but I felt that the above needed to be stated strongly.

You are welcome, as long as the discussion is focused on moving forward and improving things, i'll take any kind of constructive criticism.

Thanks for you input. :smile:

 

[flare]

Red-Shionobi has a point.
Why not reconsider the way that updates are released.
Why not deliver to mainnet more frequently with just devs and testers up-to-date with the network at first and eventually the secret is out. So what if it's hard on the network.
It's called continuous-integration. If something breaks, then simply revert. It's okay.

You are mixing the concept of continous integration with continous delivery.

I am in the process of setting up a continous integration server for darkcoin, which will additionally solve the problem of missing build for Linux 32bit and MacOSX. Below you can have a sneak preview on my ci-server, but i still have to wrap my head around the outdated build system darkcoin is using (bitcoin moved to autotools/autoconf since several month) As soon as i resolve them, Darkcoin is set for producing deterministic software builds, which can be reproduced and signed by independent parties.

Sneak preview at http://bamboo.darkcoin.qa/

Continous delivery is an interesting concept which i agree on, but you still have to plan on how you do it - otherwise in will end in chaos.

You see: things are moving. Not at the speed we all want it to be, but Rome wasn't built in a Day either.

 

[JGCMiner]

Both :smile:

Any additional effort will improve the situation, as long you follow some kind of plan - that's called continous improvement process: Plan - Do - Check - Act. That's the way you improve situations.

That's why we add the testcases we missed to the testsuite and improve testnet to match mainnet even closer.

I fully agree with you here, and started a private discussion with eduffield and AlexGR two hours ago, since i really like the approch of making testnet a clone of mainnet - if it's feasabe to have merged mining for testcoins, that's the way to go.


I fully agree with you, never said that testing/testbed should not be improved.


You are welcome, as long as the discussion is focused on moving forward and improving things, i'll take any kind of constructive criticism.

Thanks for you input. :smile:

Thanks for the detailed response.
I misinterpreted your earlier responses as "well let's do the same thing again but get it right this time". I am very happy to see that improvements are already being discussed. :smile:

 

[tungfa]

RC 3 launch yesterday was definitely a bummer !
i do not know enough about the technical part to participate in this discussion and believe evan and the team know what they are doing and i totally trust them.

i have a couple of pointers (which in my opinion) have to be discuss,
hopefully solved before the next release of any kind.
(i am not sure where else to put them, so i post them here, hope you guys do NOT mind !!)

- Communication
there has to be a proper chain of command be set up for updates to the community and public !
it can NOT be that 2 hours into the release there are NO updates what is going on !!! no updates only fuel the trolls and all the other nonsense on the chats. people get confused and worried and that is NOT necessary or helpful (for confidence in DRK and the price) !
we are by now talking about a 41 Mill market, this is not just evan with some boys in the garage coding away, DRK has to be treated as a professional product and its community and "customers" have to be updated and maintained as they deserve it.
even after the 1st updates and change of wallet messages, there was no (until 10 hours after) update if we are on a wrong fork, or maybe it is all good now (as IRC copies suggested evan can fix it fast), or not ... you Dev guys might have it all figured out at that point (or not) but you HAVE to tell the community (treat them as your customers, thats what they are!) where DRK is standing and going, good or bad , failed or running ... there is only good PR in the long run, but NO PR and communication is totally unprofessional and not acceptable in a 41 Mill market. as we (DRK) want to play in the big boys (BTC, LTC,..) league, we have to act like it !!!

- Secure communications for the Dev team
the chatter on IRC between evan and others of the Dev team has to move to a "saver" location !
chats and communications from evan and others get copied from IRC and show up all over BItcointalk with positive or negative comments.
i know AlexGR did this to calm the community and bring necessary updates (when it was needed), but this can NOT be and happen as it was totally unaproved by Evan and others copied there ! (that chatter is till now floating around out there)
this is really unprofessional (from a PR standpoint) and can go terrible wrong in the future as more and more people find their way to IRC, use quotes they copy from there, and might even change the content they copy.
PLEASE move to a saver location (non public) as i can see the PR disaster coming from this.

- Release Dates
i am only approaching this from a PR side of things .... if a RC date is set (27th June) and then the date is moved forward (20th) due to tons of testing and confidence of the developer, and then the RC still fails ... well bummer eh !
we gotta stick to our set dates, test the sh..t out of it before launching, and launch when 100% ! do not get pushed into earlier releases by the market or whatever pushed you, stick to YOUR timeline ! and the market will follow and appreciate it. (sure now we are behind with releases and the pressure is on.... but you EVAN are the MAN, thats why we are all here, so you tell the community and market how long it takes to be ready for whatever launch !!!! )

- New Developer Team
i am very happy to see that evan extended his development team !!!! finally !
extend your team further !! a couple of guys can NOT do this business by them self !! delegate to others !! i know that started already and there was a long communication from eltito (who is very good), regarding communication going on, but this has to go even further. treat it as a company and a product !

these are only my 2 cent of knowledge and thoughts in all of this
i am happy to be part of DRK and i know we will launch to further highs, but i believe it is time to change certain parts of the "team" to be as professional as "our" coin DRK is or will be soon !

 

[DRKLord]

Hey everyone,

Pleased to be here and be part of the dev team! Been wanting to get on board for a while, and yesterday's network difficulties finally made it official. Like many of you, I'm sitting on a little pile of DRK of my own and want to do anything I can to make our coin flourish and prosper.

IMO, the market wildly over-reacted yesterday. People were panicking and trying to dump DRK. I saw some people exclaiming "Oh no! <insert exchange> disabled deposit so I can't sell my <less than $100 worth> DRK!". You would've thought the sky was falling. And yet what we saw yesterday was actually progress. We experienced a glitch and some minor forking, we know what caused it and we're going to fix it. That being said, I think Red-Shionobi does have a point. We should probably change the way we deal with major updates and rebrand them as something other than an "epic masternode launch". When the public is expecting an "epic masternode launch" and there is a bug, it's viewed as a failure by the dev team rather than progress in developing a brand-new, cutting-edge technology. And then we see panics and flash crashes like yesterday. That being said, I think just renaming it something like a "public network test" would work wonders in the PR and investor relations departments...

Anyway... Preparing myself for a marathon session to dig deep into the Darkcoin source. Got a lot of catching up to do and a lot of old rust to scrape off...

@ Evan:
Sorry I passed out on you yesterday, bro! I had been awake over 24hrs and after I ate dinner it was lights out, haha

Regards,

The DRK Lord

 

[eltito]

Red-Shionobi has a point.
Why not reconsider the way that updates are released.
Why not deliver to mainnet more frequently with just devs and testers up-to-date with the network at first and eventually the secret is out. So what if it's hard on the network.
It's called continuous-integration. If something breaks, then simply revert. It's okay.

Respectfully, no, it's not ok. There is ~$50M of other people's money involved here, not including the business models built around the coin (e.g. exchange fees). It would be grossly irresponsible to be anything other than as clear and upfront as possible about what is going on with the network now and what will be going on with the network in the near future.

 

[eltito]

That being said, I think Red-Shionobi does have a point. We should probably change the way we deal with major updates and rebrand them as something other than an "epic masternode launch". When the public is expecting an "epic masternode launch" and there is a bug, it's viewed as a failure by the dev team rather than progress in developing a brand-new, cutting-edge technology. And then we see panics and flash crashes like yesterday. That being said, I think just renaming it something like a "public network test" would work wonders in the PR and investor relations departments...

To my knowledge, it was never billed (by us) as "epic masternode launch," though I certainly understand that it took on that level of hype on its own. This is exactly what I meant earlier when I said there is room for improvement in how we manage the expectations we set.

That said, a slick rebranding of hard fork launches, IMHO, would be quite meaningless. Call it whatever you want, it's still the same thing and everyone knows it - and the effect will be exactly the same if it fails. Potentially worse, as a matter of fact, because now we've given ammunition to antagonists to allow them to say we were intentionally misleading people about the seriousness of what we were doing by trying to call it something other than what it really was.

The better course of action is to give a more thorough accounting of what should happen, what may happen, and what actions we will take if something goes wrong. Events cause panic when they are unexpected, not necessarily just because they happen. If you tell people "if we notice a problem, here is what we will do" and the DO what you said you will do, nobody panics.

 

[derk]

The better course of action is to give a more thorough accounting of what should happen, what may happen, and what actions we will take if something goes wrong.

Yes please! Will someone draft a few lines doing this?

 

[DRKLord]

@ eltito:
While I'm forced to agree with you in many ways, what I'm suggesting is that we tread more carefully when dealing with the public concerning major changes to the network. Chosing the right words is vital. We're a bunch of geeks and programmers, not PR or investor relations specialists, not CEOs or CFOs and not advertising managers. But we have to start thinking about the way we communicate with the userbase to avoid causing unnecessary turmoil in the markets.

I would never suggest lying to the public and our users or hiding what's really going on. But we should take steps to avoid "hyping up" a release or modification to the network...

EDIT:
I'm not saying anyone intentionally "hyped up" the masternode launch or did any such thing. Just saying we see how wild people can get about this, so let's try to keep things calmer in the future...

 

[DRKnite]

Hi all. I usually lurk here, but I registered to make this post.

Being highly interested in Darkcoin myself and concerned for its future, I agree with many of the points in this thread. It is understandable that software is complicated and sometimes, issues will arise despite extensive testing and every precaution in the world. With cryptocurrency, these become very public and have a real, immediate financial effect on everyone involved. That being said, I think the spotty communication with the general public during yesterday's release had the biggest negative impact on the perception of the coin, allowing anyone with a vested interest in competing crypto to spread copious FUD on every chat and forum.

This has to improve if we are to build up an image of a bulletproof coin that is safe and stable. Further, I also agree that developer testing / troubleshooting talk should be moved out of a public forum.

If the team is at all interested, I'd be happy to contribute to provide timely updates and dispel any misconceptions on public forums. What about putting up a page, say status.darkcoin.io, that lists some key metrics of the network and, more importantly, has fast updates regarding new releases, forking, and any other issues that may come up? With this available to anyone who is concerned with the current status, they can easily see that the development team is on top of maintaining stability and will work to resolve any problems immediately. It would be a single point of communication with John Q. Dark, and he no longer has to venture into IRC or dig through the FUD to try and figure out what the hell is going on. I think it'd give off a more professional image and just maybe prevent the mass hysteria panic dumps that we saw yesterday.

Give it some thought. Let's work together to make Darkcoin the next generation crypto of choice by polishing its image in the view of the general public, and, needless to say, ongoing development to iron out the kinks.

 

[tungfa]

DRKLord
Chosing the right words is vital. We're a bunch of geeks and programmers, not PR or investor relations specialists, not CEOs or CFOs and not advertising managers. But we have to start thinking about the way we communicate with the userbase to avoid causing unnecessary turmoil in the markets.
------
exactly thank you !!!
and the "hype" come from the market (which is a good thing) as it is exciting times, we are in the eye of the market, exactly where we want to be !!!
so we gotta make the best out of it and use it for our advantage !!!
chain of comand ... PR setup .... team work !
....>>>
(you guys can program in your garage, and a team out here is dealing with the real world and media, but there has to be a working connection between that garage and here !!)

 

[TanteStefana]

Hi all. I usually lurk here, but I registered to make this post.

Being highly interested in Darkcoin myself and concerned for its future, I agree with many of the points in this thread. It is understandable that software is complicated and sometimes, issues will arise despite extensive testing and every precaution in the world. With cryptocurrency, these become very public and have a real, immediate financial effect on everyone involved. That being said, I think the spotty communication with the general public during yesterday's release had the biggest negative impact on the perception of the coin, allowing anyone with a vested interest in competing crypto to spread copious FUD on every chat and forum.

This has to improve if we are to build up an image of a bulletproof coin that is safe and stable. Further, I also agree that developer testing / troubleshooting talk should be moved out of a public forum.

If the team is at all interested, I'd be happy to contribute to provide timely updates and dispel any misconceptions on public forums. What about putting up a page, say status.darkcoin.io, that lists some key metrics of the network and, more importantly, has fast updates regarding new releases, forking, and any other issues that may come up? With this available to anyone who is concerned with the current status, they can easily see that the development team is on top of maintaining stability and will work to resolve any problems immediately. It would be a single point of communication with John Q. Dark, and he no longer has to venture into IRC or dig through the FUD to try and figure out what the hell is going on. I think it'd give off a more professional image and just maybe prevent the mass hysteria panic dumps that we saw yesterday.

Give it some thought. Let's work together to make Darkcoin the next generation crypto of choice by polishing its image in the view of the general public, and, needless to say, ongoing development to iron out the kinks.

This is so true, and I feel so guilty! I made an appointment to help my family friends, deep friends, like family, yesterday before the hard fork was announced and I couldn't back out, but then it ended up being an all day affair!

Otherwise, I love "chatting" on the forums and should have been there to answer questions. It was actually the plan, and I was hoping I would be back home within a couple of hours, but that didn't happen. Around 2:30, I took a peek at the forum, saw something went wrong, ugh, and had to get back to work. I don't remember my password, it's written on a card under my keyboard, LOL. So I really should have been there, and I'm sorry I wasn't.

Next time I will be, but next time, this is going to work.

 

[DarkWater]

Just wanted to say despite all the hardfork issues, I still have the most confidence that DRK will come on top and stay there. Crypt was literally destroyed today, and Cloakcoin looks like a rip off of DRK and is doing jack squat. If anything im just buying up more coins now so when the masternode system is live I can jump in! But I do think there should be a softening of the testing phase till we know that everything is live and up and running.

With the competition being eliminated DRK still has a chance.

 

[MykelSIlver]

This may have been already mentioned in the forum but did you notice XMR over on Poloniex?
Another spook to investors could cause them to dump out in another wave.

My opinion, its not all about investors. Those who really believe in the fundamentals and long term possibilities of the coin stay in.