Dashmate (discussion)

@pshenmic

sudo apt-get install docker-compose-plugin=2.20.2-1~ubuntu.22.04~jammy

Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Version '2.20.2-1~ubuntu.22.04~jammy' for 'docker-compose-plugin' was not found

I will wait for dashmate v0.24.24 which will have the Docker Compose fix. Just give an update in here, after dashmate v0.24.24 is released and upgradeable through npm (npm update -g dashmate).
 
Last edited:
Hmm, maybe this package named differently on your system

Maybe this will work:
sudo apt-get install 'docker-compose-plugin=2.20.*'
Click to expand...
So basically, you need any docker-compose-plugin before 2.21 for a temporary fix. You can also downgrade it manually via this guide

qwizzie said:
I will wait for dashmate v0.24.24 which will have the Docker Compose fix. Just give an update in here, after dashmate v0.24.24 is released and upgradeable through npm (npm update -g dashmate).
Click to expand...
Thank you for your patience. I will let you know here (y)
 
pshenmic said:
Hmm, maybe this package named differently on your system

Maybe this will work:

So basically, you need any docker-compose-plugin before 2.21 for a temporary fix. You can also downgrade it manually via this guide


Thank you for your patience. I will let you know here (y)
Click to expand...
sudo apt-get install 'docker-compose-plugin=2.20.*' did the trick.
'dashmate stop' command works again & 'dashmate status' and 'dashmate status masternode' are showing Core & Sentinel details again.

Thank you.
 
Dashmate v0.25.3 was released (y)

Platform team finally merged v0.25 development branch into master and that makes it a first stable release since latest v24 build. Upgrade your dashmate with any of your preferred methods, and don't forget to update all images with `dashmate update` :ninja:

@qwizzie You can now safely upgrade your docker-compose-plugin system dependency.

Noticeable changes:
* Testnet Dash Core upgraded to v20.0.0-beta.4
* Platform testnet chain updated to `dash-testnet-26`
* Support all docker compose ps command output (fix fatal error reported earlier by @qwizzie)
* Fix and stabilize dashmate ssl obtain and renew commands (renew certs in the background by a scheduler)
* Expose tenderdash prometheus metrics (enabled via config)
* Improve Tenderdash & Drive logging (primarily for extended debugging of the networks deployed by dash-network-deploy)
 
@pshenmic

dashmate stop
npm update -g dashmate
dashmate update
dashmate start

Dashmate update works as intended (updated Dashmate helper to 0.25.3) after doing some system updates, but i am still unable to upgrade npm to 10.2.0

command npm update -g dashmate shows the following at the end :
npm notice New major version of npm available! 9.8.1 -> 10.2.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v10.2.0
npm notice Run npm install -g [email protected] to update!

npm install -g [email protected]

npm ERR! code EBADENGINE
npm ERR! engine Unsupported engine
npm ERR! engine Not compatible with your version of node/npm: [email protected]
npm ERR! notsup Not compatible with your version of node/npm: [email protected]
npm ERR! notsup Required: {"node":"^18.17.0 || >=20.5.0"}
npm ERR! notsup Actual: {"npm":"9.8.1","node":"v16.20.1"}
Click to expand...

VPS : Ubuntu 20.04.6 LTS

FYI
 
Last edited:
qwizzie said:
@pshenmic

dashmate stop
npm update -g dashmate
dashmate update
dashmate start

Dashmate update works as intended (updated Dashmate helper to 0.25.3) after doing some system updates, but i am still unable to upgrade npm to 10.2.0

command npm update -g dashmate shows the following at the end :


VPS : Ubuntu 20.04.6 LTS

FYI
Click to expand...
Sorry, I forgot to mention that Node v16 has reached end of life maintenance support, and we dropped its support in the v25. Although it could possibly work on Node 16, dashmate may behave differently and unstable, because there are breaking changes in the interface of library, that issuing http requests (fetch). Platform code now expects at least Node.js v18 sdk api.

Many npm modules now fails to work and install until you make an upgrade, and the npm is too.

nodejs.dev

Releases

Releases | Node.js
nodejs.dev nodejs.dev
 
Last edited:
@pshenmic

I got a few crashes and restarts of my dashd.pid today that is managed through Dashmate (Mainnet, Evonode).
dashd first trips over a bad entry in a block :

2023-10-19T23:55:30Z

************************
"scheduler" raised an exception
Exception: type=dbwrapper_error, what="Fatal LevelDB error: Corruption: bad entry in block"
No debug information available for stacktrace. You should add debug information and then run:
dashd -printcrashinfo=bvcgc43iinzgc43ijfxgm3ybaacwiyltnbsfwrlymnsxa5djn5xduidupfygkplemj3xeylqobsxex3fojzg64rmeb3wqyluhuremylumfwcatdfozswyrccebsxe4tpoi5caq3pojzhk4dunfxw4oramjqwiidfnz2he6janfxcaytmn5rwwiqo4dfvuaaaaaaabyrkeiaaaaaaacsq4yqaaaaaaafjbviaaaaaaaac5qbfaaaaaaaag3hckaaaaaaabsgyeuaaaaaaaaergwyaaaaaaacegvlaaaaaaaabdsmaaaaaaaaat27iaaaaaaaabpd2diaaaaaaacjq7viaaaaaaag3y3vdgnzjaaaaa===

2023-10-19T23:55:30Z std::terminate() called due to unhandled exception
Exception: type=dbwrapper_error, what="Fatal LevelDB error: Corruption: bad entry in block"
No debug information available for stacktrace. You should add debug information and then run:
dashd -printcrashinfo=bvcgc43iinzgc43ijfxgm3ybaacwiyltnbsi443umq5du5dfojwws3tborssqkjamnqwy3dfmqqgi5lfeb2g6idvnzugc3tenrswiidfpbrwk4dunfxw4csfpbrwk4dunfxw4oraor4xazj5mrrho4tbobygk4s7mvzhe33sfqqho2dboq6sertborqwyicmmv3gk3ceiiqgk4tsn5zduicdn5zhe5lqoruw63r2ebrgczbamvxhi4tzebuw4idcnrxwg2zcb3qmwwqaaaaaaahcfiraaaaaaaakkdtcaaaaaaaavegvaaaaaaaaalwaeuaaaaaaaa3m4jiaaaaaaagi3asqaaaaaaaase23aaaaaaaaiq2vmaaaaaaaaeojqaaaaaaaacpl5aaaaaaaaaf4pinaaaaaaaajgd6vaaaaaaaa3pdoumzxfeaaaaa=
2023-10-19T23:55:30Z Posix Signal: Segmentation fault
No debug information available for stacktrace.
Click to expand...

And then dashd continue to trip a couple of times over block checksum mismatches of different ldb files (so not continuously the same one):
2023-10-19T23:56:21Z CMasternodeSync::SwitchToNextAsset -- Completed MASTERNODE_SYNC_BLOCKCHAIN in 13s
2023-10-19T23:56:21Z CMasternodeSync::SwitchToNextAsset -- Starting MASTERNODE_SYNC_GOVERNANCE
2023-10-19T23:56:21Z LevelDB read failure: Corruption: block checksum mismatch: /home/dash/.dashcore/indexes/txindex/032752.ldb
2023-10-19T23:56:21Z Fatal LevelDB error: Corruption: block checksum mismatch: /home/dash/.dashcore/indexes/txindex/032752.ldb
2023-10-19T23:56:21Z You can use -debug=leveldb to get more complete diagnostic messages
2023-10-19T23:56:21Z

************************
"scheduler" raised an exception
Exception: type=dbwrapper_error, what="Fatal LevelDB error: Corruption: block checksum mismatch: /home/dash/.dashcore/indexes/txindex/032752.ldb"
No debug information available for stacktrace. You should add debug information and then run:
dashd -printcrashinfo=bvcgc43iinzgc43ijfxgm3ybaacwiyltnbsjcrlymnsxa5djn5xduidupfygkplemj3xeylqobsxex3fojzg64rmeb3wqyluhuremylumfwcatdfozswyrccebsxe4tpoi5caq3pojzhk4dunfxw4oramjwg6y3lebrwqzldnnzxk3janvuxg3lborrwqoraf5ug63lff5sgc43if4xgiyltnbrw64tff5uw4zdfpbsxgl3upbuw4zdfpaxtamzsg42teltmmrrcedxaznnaaaaaaaaoekrcaaaaaaaauuhgeaaaaaaabkinkaaaaaaaaaxmajiaaaaaaabwzysqaaaaaaamrwbfaaaaaaaabejvwaaaaaaaarbvkyaaaaaaaai4taaaaaaaaae6x2aaaaaaaaaly6q2aaaaaaaasmh5kaaaaaaabwzgbim4akiaaaaa====

2023-10-19T23:56:21Z std::terminate() called due to unhandled exception
Exception: type=dbwrapper_error, what="Fatal LevelDB error: Corruption: block checksum mismatch: /home/dash/.dashcore/indexes/txindex/032752.ldb"
No debug information available for stacktrace. You should add debug information and then run:
dashd -printcrashinfo=bvcgc43iinzgc43ijfxgm3ybaacwiyltnbsmi43umq5du5dfojwws3tborssqkjamnqwy3dfmqqgi5lfeb2g6idvnzugc3tenrswiidfpbrwk4dunfxw4csfpbrwk4dunfxw4oraor4xazj5mrrho4tbobygk4s7mvzhe33sfqqho2dboq6sertborqwyicmmv3gk3ceiiqgk4tsn5zduicdn5zhe5lqoruw63r2ebrgy33dnmqgg2dfmnvxg5lnebwws43nmf2gg2b2eaxwq33nmuxwiyltnaxs4zdbonugg33smuxws3temv4gk4zpor4gs3temv4c6mbtgi3tkmronrsgeiqo4dfvuaaaaaaabyrkeiaaaaaaacsq4yqaaaaaaafjbviaaaaaaaac5qbfaaaaaaaag3hckaaaaaaabsgyeuaaaaaaaaergwyaaaaaaacegvlaaaaaaaabdsmaaaaaaaaat27iaaaaaaaabpd2diaaaaaaacjq7viaaaaaaag3eyfbtqbjaaaaa===
2023-10-19T23:56:21Z Posix Signal: Segmentation fault
No debug information available for stacktrace.
Click to expand...

My question : why is it looking for /home/dash/.dashcore/indexes/txindex/032752.ldb in the first place, when Dashmate puts the data directory under var/lib/docker/volumes ? I don't even have /home/dash/.dashcore, i just have /home/user/dashmate

Upon restart of dashd, the configuration is also set for /home/dash/.dashcore with regards to data directory :

2023-10-20T00:15:48Z Ignoring unknown configuration value rpcwallet
2023-10-20T00:15:48Z Dash Core version v19.3.0 (release build)
2023-10-20T00:15:48Z InitParameterInteraction: parameter interaction: -externalip set -> setting -discover=0
2023-10-20T00:15:48Z InitParameterInteraction: parameter interaction: -masternodeblsprivkey set -> setting -disablewallet=1
2023-10-20T00:15:48Z Assuming ancestors of block 00000000000000075300e852d5bf5380f905b2768241f8b442498442084807a7 have valid signatures.
2023-10-20T00:15:48Z Setting nMinimumChainWork=000000000000000000000000000000000000000000008677827656704520eb39
2023-10-20T00:15:48Z fDisableGovernance 0
2023-10-20T00:15:48Z Using the 'standard' SHA256 implementation
2023-10-20T00:15:48Z Wallet disabled!
2023-10-20T00:15:48Z Default data directory /home/dash/.dashcore
2023-10-20T00:15:48Z Using data directory /home/dash/.dashcore
2023-10-20T00:15:48Z Config file: /home/dash/.dashcore/dash.conf
Click to expand...

Why set it to a data directory that does not even exist on my system ?

I will do a 'dashmate core reindex' command and hope this will fix things. At least it gives me the chance to test Dashmate handling of a reindex on Mainnet. If it does not work (maybe the database got corrupted), what Dashmate command can i use to delete the blockchain and download it again from scratch ?
 
Last edited:
Hey @qwizzie

The error looks like a dash core issue, probably not yet known. I would report it to the dashcore team

The path that you can see in the log files is related to the container internal disk space. Dashmate is a simply a docker compose file that is being configured and run with JS in a nice interface. So is, `/home/dash/.dashcore` is a path inside core container, and its mounted as a named docker volume from /var/lib/docker/...

Yes, `dashmate core reindex` is the command written up just for this and should be able to resolve that. If not, you can reset your chain via `dashmate reset` command. It will drop all your L1 storage
 
Hi @pshenmic , quick question:

When installing dashmate onto an existing core installation (to add Platform), is it best to delete the previous installation entirely and let dashmate install dash core and sync the blockchain from scratch? Or is it okay to "add" dashmate to an existing fully-synced node? Thanks!
 
Last edited:
Bridgewater said:
Hi @pshenmic , quick question:

When installing dashmate onto an existing core installation (to add Platform), is it best to delete the previous installation entirely and let dashmate install dash core and sync the blockchain from scratch? Or is it okay to "add" dashmate to an existing fully-synced node? Thanks!
Click to expand...

Hey @Bridgewater, the officially supported way right now is to just sync from scratch. There are couple of ways how you could import data chain into dashmate node, but we haven't tested this. The best way is just wait for the sync. It is okay to start dashmate together with your node, as long as port bindings does not interfere.
 
pshenmic said:
Hey @Bridgewater, the officially supported way right now is to just sync from scratch. There are couple of ways how you could import data chain into dashmate node, but we haven't tested this. The best way is just wait for the sync. It is okay to start dashmate together with your node, as long as port bindings does not interfere.
Click to expand...
Thanks!
I don't have enough space for all the blocks of two full nodes, so I'll just delete the current .dashcore folder and let dashmate download+install core.
 
pshenmic said:
Sure! If you are on Ubuntu or Debian, I would suggest you installing it from .deb package that can be found on https://github.com/dashpay/platform/releases, so you don't need to install any additional tools like Docker & Node.js (they are included in the package)
Click to expand...
I installed the .deb package and ran dashmate setup and it almost worked. Got the welcome menu, but below it has an error message that docker is not installed. Do I still need to install that manually?
 
Bridgewater said:
I installed the .deb package and ran dashmate setup and it almost worked. Got the welcome menu, but below it has an error message that docker is not installed. Do I still need to install that manually?
Click to expand...
Yeah, looks like I was wrong. I just checked and can confirm that it asks for an explicit Docker install in the system on Ubuntu 22.04. I will release a video instructions soon.

vazaki3 said:
can I run dashmate by using bocker instead of docker?
Click to expand...
No, this thing doesn't have compose, port forwarding and data volumes according their documentaiton. Also, it haven't been updated in 8 years, so i'm not sure it even works.
 
Dashmate v0.25.13 was released :eek: (y)

I'm happy to announce another stable build of the dashmate that includes several interesting things.

Noticeable changes:
* Config migration unit test (checks that dashmate can successfully migrate and have correct image versions)
* Extended masternodes count in the status to include evnonodes
* Fix payment queue position in the mastenode status
* Add `dashmate docker build` command to build services images
* Add `dashmate core cli` command to send Core command


Dashmate core cli command

This build introduces a new `dashmate core cli` command that you can use to execute Core RPC commands to your node.

Screenshot 2023-11-06 at 22.35.46.png



Disable config auto render

Dashmate earlier had a pretty simple and straight forward solution on rendering config files, it simply re-rendered all the configs on each command run to ensure all configs are in sync. Although such approach has some advantages, it is not quite efficient and already caused us few troubles in making things idempotent in the production deployment.
We decided to completely disable that and re-render all the configs only when it is definetly necessary, for example on setup, when config is changed, or new version was just installed. No more unneeded writes and must be less execution time.


Extended enabled count and fixed payment queue position

More masternode count fields were added like total evonodes count and enabled evnodes count. Also, the payment calculation was revised and fixed to correctly include evonodes.



Config migration unit test

This thing is very great thing that already helped us tighten our builds in the latest development branch. The idea is simple, we mocked a first dashmate v25 config, and each build we check if the migration process finished succesfully and config version matches expected. This prevents human errors during development and reduces broken builds number.
 
pshenmic said:
No, this thing doesn't have compose, port forwarding and data volumes according their documentaiton. Also, it haven't been updated in 8 years, so i'm not sure it even works.
Click to expand...
It works BECAUSE it havent been updated for 8 years.
Something that is always updated, it is obviously either a failure or a security risk.
 
Last edited:
Do I still need the cron job to monitor dashd and restart in case of a crash, or does dashmate do this automatically?
 
You must log in or register to reply here.
Back
Top