Filling up my debug.log: ProcessMessages(version, 138 bytes) FAILED ...

[t0dd]

I'm in the process of upgrading production masternodes to 0.15. I see this in a 0.14 MN. (Note that the IP address here is not the IP address of my MN).

I upgraded one MN to 0.15 and it is seeing similar messages but without the explicit addr=<IP>.

I thought it would clear out eventually, but 24 hours later it had not (I am unsure when it started, probably long ago). So, with the 0.14 MN, I stopped dashd, nuked the peers.dat file, and then restarted it. The log lines began again.

So, any ideas?

2020-02-20 22:50:11 ProcessMessages(version, 138 bytes) FAILED peer=76
2020-02-20 22:50:11 ThreadSocketHandler -- removing node: peer=76 addr=77.81.226.146:9999 nRefCount=2 fInbound=0 fMasternode=1
2020-02-20 22:50:12 ProcessMessages(version, 138 bytes) FAILED peer=77
2020-02-20 22:50:12 ThreadSocketHandler -- removing node: peer=77 addr=77.81.226.146:9999 nRefCount=2 fInbound=0 fMasternode=1
2020-02-20 22:50:13 ProcessMessages(version, 138 bytes) FAILED peer=78
2020-02-20 22:50:13 ThreadSocketHandler -- removing node: peer=78 addr=77.81.226.146:9999 nRefCount=2 fInbound=0 fMasternode=1
2020-02-20 22:50:14 ProcessMessages(version, 138 bytes) FAILED peer=79
2020-02-20 22:50:14 ThreadSocketHandler -- removing node: peer=79 addr=77.81.226.146:9999 nRefCount=2 fInbound=0 fMasternode=1
2020-02-20 22:50:15 ProcessMessages(version, 138 bytes) FAILED peer=80
2020-02-20 22:50:15 ThreadSocketHandler -- removing node: peer=80 addr=77.81.226.146:9999 nRefCount=2 fInbound=0 fMasternode=1
2020-02-20 22:50:17 ProcessMessages(version, 138 bytes) FAILED peer=81
2020-02-20 22:50:17 ThreadSocketHandler -- removing node: peer=81 addr=77.81.226.146:9999 nRefCount=2 fInbound=0 fMasternode=1
2020-02-20 22:50:18 ProcessMessages(version, 138 bytes) FAILED peer=82
2020-02-20 22:50:18 ThreadSocketHandler -- removing node: peer=82 addr=77.81.226.146:9999 nRefCount=2 fInbound=0 fMasternode=1
2020-02-20 22:50:19 ProcessMessages(version, 138 bytes) FAILED peer=83
2020-02-20 22:50:19 ThreadSocketHandler -- removing node: peer=83 addr=77.81.226.146:9999 nRefCount=2 fInbound=0 fMasternode=1
2020-02-20 22:50:20 ProcessMessages(version, 138 bytes) FAILED peer=84
2020-02-20 22:50:20 ThreadSocketHandler -- removing node: peer=84 addr=77.81.226.146:9999 nRefCount=2 fInbound=0 fMasternode=1
2020-02-20 22:50:21 ProcessMessages(version, 138 bytes) FAILED peer=85
2020-02-20 22:50:21 ThreadSocketHandler -- removing node: peer=85 addr=77.81.226.146:9999 nRefCount=2 fInbound=0 fMasternode=1
2020-02-20 22:50:22 ProcessMessages(version, 138 bytes) FAILED peer=86
2020-02-20 22:50:22 ThreadSocketHandler -- removing node: peer=86 addr=77.81.226.146:9999 nRefCount=2 fInbound=0 fMasternode=1
2020-02-20 22:50:23 ProcessMessages(version, 138 bytes) FAILED peer=87
2020-02-20 22:50:23 ThreadSocketHandler -- removing node: peer=87 addr=77.81.226.146:9999 nRefCount=2 fInbound=0 fMasternode=1
2020-02-20 22:50:24 ProcessMessages(version, 138 bytes) FAILED peer=88
2020-02-20 22:50:24 ThreadSocketHandler -- removing node: peer=88 addr=77.81.226.146:9999 nRefCount=2 fInbound=0 fMasternode=1

 

[t0dd]

Okay. After 2900+ repetitions (the 0.14 MN still), it stopped logging the failure. I don't understand what just happened.

First failure was:

2020-02-20 22:49:19 UpdateTip: new best=0000000000000005be39b2e4fe54b9a8f895c321916f69b51aba1acc6c308de6 height=1224859 version=0x20000000 log2_work=77.37096404 tx=21660739 date='2020-02-20 22:49:18' pro
gress=1.000000 cache=0.0MiB(357txo) evodb_cache=0.0MiB
2020-02-20 22:49:19 ProcessNewBlock : ACCEPTED
2020-02-20 22:49:19 ProcessMessages(version, 138 bytes) FAILED peer=28
2020-02-20 22:49:19 ThreadSocketHandler -- removing node: peer=28 addr=77.81.226.146:9999 nRefCount=1 fInbound=0 fMasternode=1

Last failure was:

2020-02-20 23:42:34 ProcessMessages(version, 138 bytes) FAILED peer=2954
2020-02-20 23:42:34 ThreadSocketHandler -- removing node: peer=2954 addr=77.81.226.146:9999 nRefCount=2 fInbound=0 fMasternode=1

 

[t0dd]

I did the same thing with the 0.15 MN and the issue did not go away with it. I continue to troubleshoot.

. . .
2020-02-21 04:08:49 ProcessMessages(version, 138 bytes) FAILED peer=7046
2020-02-21 04:08:49 ThreadSocketHandler -- removing node: peer=7046 nRefCount=2 fInbound=0 fMasternode=1
2020-02-21 04:08:50 ProcessMessages(version, 138 bytes) FAILED peer=7047
2020-02-21 04:08:50 ThreadSocketHandler -- removing node: peer=7047 nRefCount=2 fInbound=0 fMasternode=1
2020-02-21 04:08:51 ThreadSocketHandler -- removing node: peer=7048 nRefCount=1 fInbound=0 fMasternode=1
2020-02-21 04:08:52 ProcessMessages(version, 138 bytes) FAILED peer=7049
2020-02-21 04:08:52 ThreadSocketHandler -- removing node: peer=7049 nRefCount=2 fInbound=0 fMasternode=1
2020-02-21 04:08:53 ProcessMessages(version, 138 bytes) FAILED peer=7050
2020-02-21 04:08:53 ThreadSocketHandler -- removing node: peer=7050 nRefCount=2 fInbound=0 fMasternode=1
2020-02-21 04:08:55 ThreadSocketHandler -- removing node: peer=7051 nRefCount=1 fInbound=0 fMasternode=1
2020-02-21 04:08:56 ThreadSocketHandler -- removing node: peer=7052 nRefCount=1 fInbound=0 fMasternode=1
2020-02-21 04:08:57 ThreadSocketHandler -- removing node: peer=7053 nRefCount=1 fInbound=0 fMasternode=1
2020-02-21 04:08:58 ProcessMessages(version, 138 bytes) FAILED peer=7054
2020-02-21 04:08:58 ThreadSocketHandler -- removing node: peer=7054 nRefCount=2 fInbound=0 fMasternode=1
2020-02-21 04:09:00 ThreadSocketHandler -- removing node: peer=7055 nRefCount=1 fInbound=0 fMasternode=1
2020-02-21 04:09:01 ThreadSocketHandler -- removing node: peer=7056 nRefCount=1 fInbound=0 fMasternode=1
2020-02-21 04:09:02 ThreadSocketHandler -- removing node: peer=7057 nRefCount=1 fInbound=0 fMasternode=1
2020-02-21 04:09:03 ThreadSocketHandler -- removing node: peer=7058 nRefCount=1 fInbound=0 fMasternode=1
2020-02-21 04:09:04 ThreadSocketHandler -- removing node: peer=7059 nRefCount=2 fInbound=0 fMasternode=1
2020-02-21 04:09:04 ProcessMessages(version, 138 bytes) FAILED peer=7059
2020-02-21 04:09:05 ThreadSocketHandler -- removing node: peer=7060 nRefCount=1 fInbound=0 fMasternode=1
2020-02-21 04:09:06 ProcessMessages(version, 138 bytes) FAILED peer=7061
2020-02-21 04:09:06 ThreadSocketHandler -- removing node: peer=7061 nRefCount=2 fInbound=0 fMasternode=1
2020-02-21 04:09:08 ThreadSocketHandler -- removing node: peer=7062 nRefCount=1 fInbound=0 fMasternode=1
2020-02-21 04:09:09 ProcessMessages(version, 138 bytes) FAILED peer=7063
2020-02-21 04:09:09 ThreadSocketHandler -- removing node: peer=7063 nRefCount=2 fInbound=0 fMasternode=1
2020-02-21 04:09:10 ProcessMessages(version, 138 bytes) FAILED peer=7064
2020-02-21 04:09:10 ThreadSocketHandler -- removing node: peer=7064 nRefCount=2 fInbound=0 fMasternode=1
2020-02-21 04:09:11 ThreadSocketHandler -- removing node: peer=7065 nRefCount=1 fInbound=0 fMasternode=1
2020-02-21 04:09:12 ProcessMessages(version, 138 bytes) FAILED peer=7066
2020-02-21 04:09:12 ThreadSocketHandler -- removing node: peer=7066 nRefCount=2 fInbound=0 fMasternode=1
2020-02-21 04:09:13 ThreadSocketHandler -- removing node: peer=7067 nRefCount=1 fInbound=0 fMasternode=1
2020-02-21 04:09:14 ProcessMessages(version, 138 bytes) FAILED peer=7068
2020-02-21 04:09:14 ThreadSocketHandler -- removing node: peer=7068 nRefCount=2 fInbound=0 fMasternode=1
2020-02-21 04:09:16 ProcessMessages(version, 138 bytes) FAILED peer=7069
2020-02-21 04:09:16 ThreadSocketHandler -- removing node: peer=7069 nRefCount=2 fInbound=0 fMasternode=1
2020-02-21 04:09:17 ProcessMessages(version, 138 bytes) FAILED peer=7070
. . .

Is this normal?

 

[t0dd]

No response from anyone here in the forums, but for completeness:

It all eventually cleared out. But it had been doing this for quite some time. For how long? I don't know. But what I did was delete all the .dat files shut down for a bit, and then cranked everything back up. The FAILED messages actually came back for awhile and then disappeared.

I don't know if my MNs where being targeted or if the associated IP eventually got banned? I don't know. But it cleared out for all of the masternodes that were experiencing this.