v12 Regularly Crashing on Raspberry Pi 2

Syntax360

Syntax360

Member
I've been running a MN on a Raspberry Pi 2 for right around 6 months now. This setup has run swimmingly on every prior version of Dash, right up until the first version of v12 when I began to experience frequent masternode crashes. The problem recurred very often on the earlier minor versions of v12 - the daemon would mysteriously vanish every 24 hours or less - but each new minor version release seemed to get me a little more daemon uptime before things would go south. This observation may completely be in my head - I wasn't tracking data and the observation is purely anecdotal.

I've brought the matter up several times on IRC and most folks immediately suspect the rPi2 hardware may not be up to snuff - but I really do not think that is the issue. I have kept an eye on system resource consumption over the course of many days and dashd consistently consumes between 20-30% CPU and MEM - the daemon really does not appear to be very taxing on the rPi2 hardware.

So here's what I know:

I use github to pull source with every new version release; I compile with the following commands (it takes about 2 hours):

sh autogen.sh
./configure --with-incompatible-bdb
make && sudo make install

When the daemon crashes, my amateur eyes can detect no clues regarding the source of my troubles in ~/.dash/debug.log - the last lines will just be normal activity one would expect to find there.

I have discovered the following in /var/log which I believe might be useful in figuring out what is going on:

syslog.1:Sep 9 22:29:41 [HOSTNAME] dashd: *** glibc detected *** dashd: free(): invalid next size (fast): 0x58040da8 ***

user.log:Sep 9 22:29:41 [HOSTNAME] dashd: *** glibc detected *** dashd: free(): invalid next size (fast): 0x58040da8 ***

user.log.1:Sep 5 01:45:18 [HOSTNAME] dashd: *** glibc detected *** dashd: double free or corruption (!prev): 0x5683d140 ***


I experienced daemon crashes on September 5th and again late last night (September 9th). I wish I would have been doing a better job looking for this sort of stuff previously, but those logs are all I have to go on for now.

A quick Google search for those types of errors turn up other instances of other problematic daemons, and there seemed to be an even split of people blaming corrupt memory or problems within the daemon itself. I certainly cannot rule out the hardware angle, but it seems awful coincidental that this trouble manifested immediately after the release of v12 and has been lingering ever since...


Help! Has anyone else been having seemingly random troubles on Raspberry Pi 2 hardware? Does anyone have any idea what might be going on, or any suggestions that might help narrow down the search for a resolution?
 
Im running on a pi2 as well for ages now.

I get a v.53 crash once in a while. Been good for couple of days now. My supersimple cronjob crashdump script spits out the last 40 lines of the debug before restarting dashd. This is what i got:

CFinalizedBudgetVote::SignatureValid() - Unknown Masternode
2015-09-10 09:14:35 CFinalizedBudgetVote::SignatureValid() - Unknown Masternode
2015-09-10 09:14:35 CFinalizedBudgetVote::SignatureValid() - Unknown Masternode
2015-09-10 09:14:35 CFinalizedBudgetVote::SignatureValid() - Unknown Masternode
2015-09-10 09:14:35 CFinalizedBudgetVote::SignatureValid() - Unknown Masternode
2015-09-10 09:14:35 CFinalizedBudgetVote::SignatureValid() - Unknown Masternode
2015-09-10 09:14:35 CFinalizedBudgetVote::SignatureValid() - Unknown Masternode
2015-09-10 09:14:35 CFinalizedBudgetVote::SignatureValid() - Unknown Masternode
2015-09-10 09:14:35 CFinalizedBudgetVote::SignatureValid() - Unknown Masternode
2015-09-10 09:14:35 CFinalizedBudgetVote::SignatureValid() - Unknown Masternode
2015-09-10 09:14:35 CFinalizedBudgetVote::SignatureValid() - Unknown Masternode
2015-09-10 09:14:35 CFinalizedBudgetVote::SignatureValid() - Unknown Masternode
2015-09-10 09:14:35 ProcessNewBlock : ACCEPTED
2015-09-10 09:14:39 CheckBlock() : skipping transaction locking checks
2015-09-10 09:14:39 CheckBlock() : skipping transaction locking checks
2015-09-10 09:14:39 CheckBlock() : skipping transaction locking checks
2015-09-10 09:14:39 UpdateTip: new best=000000000002cd6c17afb67f96eec5ca9d4408abcb2f5750bb9ff130f50cfd1b height=333931 log2_work=61.700501 tx=1339645 date=2015-09-10 09:14:43 progress=1.000000 cache=3364
2015-09-10 09:14:39 ProcessNewBlock : ACCEPTED
2015-09-10 09:15:09 CheckBlock() : skipping transaction locking checks
2015-09-10 09:15:09 CheckBlock() : skipping transaction locking checks
2015-09-10 09:15:09 CheckBlock() : skipping transaction locking checks
2015-09-10 09:15:09 UpdateTip: new best=00000000000eebdf6f512df0e0d5984c1c491dd5b0c511c15b43dbce8172b5b4 height=333932 log2_work=61.700505 tx=1339646 date=2015-09-10 09:14:39 progress=0.999999 cache=3415
2015-09-10 09:15:10 ProcessNewBlock : ACCEPTED
2015-09-10 09:15:30 dstx: Got Masternode transaction ff9c9d7b7351c75943f2d98b41b19ba08b2646870310c8aec55bf3ad445bc48c
2015-09-10 09:15:30 PrioritiseTransaction: ff9c9d7b7351c75943f2d98b41b19ba08b2646870310c8aec55bf3ad445bc48c priority += 1000.000000, fee += 0.10
2015-09-10 09:16:36 CheckBlock() : skipping transaction locking checks
2015-09-10 09:16:36 CheckBlock() : skipping transaction locking checks
2015-09-10 09:16:36 CheckBlock() : skipping transaction locking checks
2015-09-10 09:16:36 UpdateTip: new best=000000000004e8a70549ddbfb501835a6690a22014d6bc2eb04129e3e2718027 height=333933 log2_work=61.70051 tx=1339653 date=2015-09-10 09:16:05 progress=0.999998 cache=3442
2015-09-10 09:16:36 ProcessNewBlock : ACCEPTED
2015-09-10 09:17:57 CheckBlock() : skipping transaction locking checks
2015-09-10 09:17:57 CheckBlock() : skipping transaction locking checks
2015-09-10 09:17:57 CheckBlock() : skipping transaction locking checks
2015-09-10 09:17:57 UpdateTip: new best=000000000012a6a3cc88f4ea7b63df3a5ab8c3184ad5246a87f1031de26c16a1 height=333934 log2_work=61.700514 tx=1339655 date=2015-09-10 09:17:31 progress=0.999999 cache=3464
2015-09-10 09:17:57 ProcessNewBlock : ACCEPTED
2015-09-10 09:18:51 CheckBlock() : skipping transaction locking checks
2015-09-10 09:18:51 CheckBlock() : skipping transaction locking checks
2015-09-10 09:18:51 CheckBlock() : skipping transaction locking checks
2015-09-10 09:18:51 UpdateTip: new best=000000000004d58459aa02ef224b0e5dd992d8b43c23a43f7526654cc83cf54e height=333935 log2_work=61.700519 tx=1339656 date=2015-09-10 09:17:57 progress=0.999997 cache=3488
2015-09-10 09:18:51 ProcessNewBlock : ACCEPTED
2015-09-10 09:19:22 CActiveMasternode::SendMasternodePing() - Relay Masternode Ping vin = CTxIn(COutPoint(64493231c1843b9a0579fce24c07422acccde32166ed04724ac35164f37b9dff, 1), scriptSig=)
2015-09-10 09:19:31 receive version message: /darkcoinseeder:0.1.2/: version 70103, blocks=320000, us=65.234.653.234:9999, peer=249
2015-09-10 09:20:15 socket send error Network is unreachable (101)
2015-09-10 09:20:15 receive version message: /Dash Core:0.12.0.53/: version 70103, blocks=333935, us=65.234.653.234:45268, peer=250
2015-09-10 09:20:15 Added time data, samples 49, offset +0 (+0 minutes)
2015-09-10 09:20:15 nTimeOffset = +0 (+0 minutes)
2015-09-10 09:21:10 receive version message: /Dash Core:0.12.0.53/: version 70103, blocks=333935, us=65.234.653.234:9999, peer=251
2015-09-10 09:21:10 Added time data, samples 50, offset +227 (+3 minutes)
2015-09-10 09:21:25 mnb - Got NEW Masternode entry - c1d42434ce8de9b2204d034f9459876afa6b7a031556b654072a321cd7af49d3 - 54.149.187.55:9999 - CTxIn(COutPoint(7652f86d64359a09072be8d589c880174f75db376e67c6e498375dd220f889b1, 0), scriptSig=) - 1439995070
2015-09-10 09:21:47 mnb - Got NEW Masternode entry - 66d1329edda01bdeb3b5dffaa10e479b69e049fb71a876c7ce386691e76123d7 - 108.61.126.208:9999 - CTxIn(COutPoint(b3e1883f1312f1fc2df4ac9b21cabac77647b7216b48c5951af8e47d07c7e821, 1), scriptSig=) - 1441876905
Click to expand...

As far as i can tell no errors here. Next time Ill check the /var/log/syslog when next crash occur and compare the times.


the script is this:

#!/bin/bash

if [ -z `pidof dashd` ]; then

echo "************************************************************************************" >> crash.log
echo "Crashed!" >> crash.log
date >> crash.log
echo -e "\n" >> crash.log
tail -n 50 ./.dash/debug.log >> crash.log
echo -e "\n" >> crash.log
echo "************************************************************************************" >> crash.log
echo -e "\n" >> crash.log
echo -e "\n" >> crash.log

./dashd

fi
Click to expand...
 
Since 53 update - I'm good - no crash or restart needed
 
fuzzyduck said:
Code: 
#!/bin/bash

if [ -z `pidof dashd` ]; then

echo "************************************************************************************" >> crash.log
echo "Crashed!" >> crash.log
date >> crash.log
echo -e "\n" >> crash.log
tail -n 50 ./.dash/debug.log >> crash.log
echo -e "\n" >> crash.log
echo "************************************************************************************" >> crash.log
echo -e "\n" >> crash.log
echo -e "\n" >> crash.log

./dashd

fi
Click to expand...

Simpler version:

Code: 
#!/bin/bash

DASHDIR=$HOME/.dash

if [ -z `pidof dashd` ]; then

cat >> $DASHDIR/crash.log << EOF
************************************************************************************
Crashed!


`tail -50 $DASHDIR/debug.log`


************************************************************************************


EOF

$DASHDIR/dashd

fi
 
I might add that I also have 3GB of SWAP
- if that matters
 
You must log in or register to reply here.
Back
Top