• Forum has been upgraded, all links, images, etc are as they were. Please see Official Announcements for more information

Ongoing DDoS attack on masternode network

UdjinM6

UdjinM6

Official Dash Dev
Dash Core Group
Dear community,

We have an ongoing attack on masternode network for about 3 hours now. The attack is a mix of SYN flood, UDP flood with empty payload and protocols like sFlow and GRE. So far the only result attacker was able to achieve is that he caused higher CPU and bandwidth usage for most of masternodes which in its turn cased ~100 of masternodes to go down. We believe these masternodes were hosted on extremely low end VPS, like $1/mo, so it's not that surprising to see some of them down now. We encourage all masternode owners whose nodes were affected to move to a better hardware or upgrade their hosting plan to ensure that your masternode doesn't fall out of payment queue during such events.

Meanwhile, you can mitigate the attack by following steps in iptables setup part of this guide https://www.dash.org/forum/threads/how-to-set-up-ec2-t1-micro-ubuntu-for-masternode-part-2-3.241/

UPDATE:
Incident report: https://www.dash.org/2017/03/08/DDoSReport.html
 
Last edited:
UdjinM6 said:
Dear community,

We have an ongoing attack on masternode network for about 3 hours now. The attack is a mix of SYN flood, UDP flood with empty payload and protocols like sFlow and GRE. So far the only result attacker was able to achieve is that he caused higher CPU and bandwidth usage for most of masternodes which in its turn cased ~100 of masternodes to go down. We believe these masternodes were hosted on extremely low end VPS, like $1/mo, so it's not that surprising to see some of them down now. We encourage all masternode owners whose nodes were affected to move to a better hardware or upgrade their hosting plan to ensure that your masternode doesn't fall out of payment queue during such events.

Meanwhile, you can mitigate the attack by following steps in iptables setup part of this guide https://www.dash.org/forum/threads/how-to-set-up-ec2-t1-micro-ubuntu-for-masternode-part-2-3.241/
Click to expand...
Interesting. I thought this might happen when the value increased. I wonder if it was an economic attack by other MNs or possibly short sellers.

Would the tails IP blinding solution be a londer term response?
 
halso said:
Interesting. I thought this might happen when the value increased. I wonder if it was an economic attack by other MNs or possibly short sellers.

Would the tails IP blinding solution be a londer term response?
Click to expand...
Probably Moronsons trying their last effort to tackle Dash, us.:D
 
Attack is causing my mnodes cpu load going to up 70% now, no problem so far any of my nodes.

Clipboard01.png
 
How costly is this ongoing DDoS attack for the attacker and how long can we exspect such an attack to continue ?

Looks like this attacks also causes some incorrect feedback on port checks in dashninja.pl
(closed ports instead of open ports). At least i hope its feedback is incorrect.
(Dash Central does not give any errors)
 
Last edited:
qwizzie said:
How costly is this ongoing DDoS attack for the attacker and how long can we exspect such an attack to continue ?

Looks like this attacks also causes some incorrect feedback on port checks in dashninja.pl
(closed instead of open). At least i hope its feedback is incorrect.
Click to expand...
No idea how costly it is.
Yep, port checker is confused because all ports are used by the attacker if there is no defense in place on that masternode.
 
UdjinM6 said:
No idea how costly it is.
Yep, port checker is confused because all ports are used by the attacker if there is no defense in place on that masternode.
Click to expand...

Will we be safe if we set the firewall on remote as follows ?

apt-get update

You will see a lot of things happening, just wait for them to stop.

apt-get install ufw

Enter the following commands EXACTLY (in this order) to set up your firewall:
Please note: Make sure you enter the code in this order! If you do not, the program will not work! If need be you can disable your firewall by entering (as root;)): ufw disable.

ufw allow ssh/tcp

ufw limit ssh/tcp ---this command limits SSH connections to 6 every 30 seconds for greater security---

ufw allow 9999/tcp

ufw logging on

ufw enable


Check your firewall's status by entering the following command:

ufw status
Click to expand...

Or do we still need to add those iptables ?
 
qwizzie said:
Will we be safe if we set the firewall on remote as follows ? Or do we still need to add those iptables ?
Click to expand...
I'm not sure about ufw capability to stop this, I do know that configuring iptables the way it's described in that guide helps.
Let's ping our network guru @chaeplin for more info :)
 
UdjinM6 said:
I'm not sure about ufw capability to stop this, I do know that configuring iptables the way it's described in that guide helps.
Let's ping our network guru @chaeplin for more info :)
Click to expand...

Good idea, and if it is indeed needed maybe someone can provide an easy to use Linux 14.04 specific up to date guide to installing these iptables on remote.
 
From ubuntu wiki:

The Uncomplicated Firewall (ufw) is a frontend for iptables and is particularly well-suited for host-based firewalls. ufw provides a framework for managing netfilter, as well as a command-line interface for manipulating the firewall. ufw aims to provide an easy to use interface for people unfamiliar with firewall concepts, while at the same time simplifies complicated iptables commands to help an adminstrator who knows what he or she is doing.
 
I'm trying to find a graph where we can observ the attack, but i cant really seem to find it. This one looks normal but then again it is only focussing on transactions :

LHjpgJP.png


Are there other graphs networkwide that shows the attack more clearly ?
 
Last edited:
qwizzie said:
I'm trying to find a graph where we can observ the attack, but i cant really seem to find it. This one looks normal but then again it is only focussing on transactions :

Are there other graphs networkwide that shows the attack more clearly ?
Click to expand...

Capture.PNG


Source: @chaeplin :)

You can see the MNs that dropped off center of the top row, at the attacks peak (t+5 hrs) the # of nodes offline was 114 or ~2.8% of the network, probably the nodes with the least performance / provision / cost. Now in hour 6 and the nodes are recovering, 2.1% are still down.
 
Last edited:
I'm observing not only SYN-Floods, but valid connection attempts to exhaust the connection limit of the dashd. Except for limiting the connection rate per client, I think there is currently no way to stop a part of the attacks without some behaviour-analysis-based firewall. Luckily, the DDoS mitigation of the provider kicked in now, so everything is back to normal.
 
Figlmüller said:
I'm observing not only SYN-Floods, but valid connection attempts to exhaust the connection limit of the dashd. Except for limiting the connection rate per client, I think there is currently no way to stop a part of the attacks without some behaviour-analysis-based firewall. Luckily, the DDoS mitigation of the provider kicked in now, so everything is back to normal.
Click to expand...

Yes DDOS protection, the IP tables/hardening that @UdjinM6 posted above and making sure your node is high enough spec / on decent infrastructure is the best protection.

This kind of attack can be done on any Crypto p2p network and it's clearly not very effective against Dash considering only 2.8% of service was disrupted vs the cost of flooding 4,100 nodes with that much bandwidth for 6 hrs.
 
The fact that only 2.8% of nodes have been affected to the point of crashing is great news for the network... We're in good shape! :)

We now know that it would take a significant DDOS attack to cause any kind of statistically meaningful attack on the functions and operation of network, and is very unlikely to succeed in disrupting the network. For sure, some MN operators that took the low bid on their VPS provisioning are probably a bit pissed but that's the true cost of cheap.. Lesson learned hopefully!

Let's see if this is just an exploratory DDOS and whether we get more targetted and higher intensity attacks going forwards..

Get yer hard hats on MN operators!

Walter
 
@qwizzie : You can only estimate it from the CPU and network load.

If you want to monitor it, you would need to have the firewalling iptables-rules log into a file or so, something which would make the attack even worse.

In other words, I could do it for one of my nodes, but I won't :p
 
good test to remind all the 1U$ vultr nodes to migrate to more solid services i guess ;)
 
You must log in or register to reply here.
Back
Top