Website under attack?

[SteveJohnson]

People,

Recently, in half an hour of time my CPU usage shot upto over 80% in usage (usually it is way way lower and i use a VPS). During this time, there were no posts made, no comments done.. and im sure that there were no DB transactions at that time.

The RAM usage was fine, but the Disk I/O utilization and the Network I/O shot up just as dramatically as the CPU. My Cloudflare's analytics shows a spike in visitors too, but i still have to wait for Google's analytics till tomorrow.

As of now my website is down, and i can't even check my error logs. I'm posting here to ask for any suggestions, and what measures should i take to check if it really is an attack, or some cron job, or a program misbehaving.

Regards,

[malikin]

just set Captcha for all visitors in Cloudflare

[SteveJohnson]

just set Captcha for all visitors in Cloudflare

The typical setting in such a scenario is "i'm under attack mode", not a captcha. Though the website WAS in "i'm under attack mode" it still did not help in any way. I'm still in the process of interacting with my provider, and cloudflare to know what exactly happened.

[SmaRTeY]

Hi,

are you using Windows or Linux VPS?
Do you have your firewall setup correctly?
Do you have any "Ban" system in place?
What servers do you have running on your VPS?
Check your Mailserver logs to see if it was a mail attack (possible spam through your email server)

Regards,
Eric

[Aficionado]

People,

Recently, in half an hour of time my CPU usage shot upto over 80% in usage (usually it is way way lower and i use a VPS). During this time, there were no posts made, no comments done.. and im sure that there were no DB transactions at that time.

The RAM usage was fine, but the Disk I/O utilization and the Network I/O shot up just as dramatically as the CPU. My Cloudflare's analytics shows a spike in visitors too, but i still have to wait for Google's analytics till tomorrow.

As of now my website is down, and i can't even check my error logs. I'm posting here to ask for any suggestions, and what measures should i take to check if it really is an attack, or some cron job, or a program misbehaving.

Regards,

First defence it to block them using .htaccess. PLENTY of information on the Internet on how to.

[SteveJohnson]

Hi,

are you using Windows or Linux VPS?
Do you have your firewall setup correctly?
Do you have any "Ban" system in place?
What servers do you have running on your VPS?
Check your Mailserver logs to see if it was a mail attack (possible spam through your email server)

Regards,
Eric

Hey Eric, how've you been
Linux ubuntu
firewalls - check
Do you mean something for the ssh? fail2ban? - check

My server monitor showed a 100% CPU usage, and it was my php-fpm that was using all the resources.. so should not be a mail attack

[SteveJohnson]

First defence it to block them using .htaccess. PLENTY of information on the Internet on how to.

Though we're discussing on the other thread and you're aware, but i'll just reply here too.
I use nginx, so no htaccess for me.

[SmaRTeY]

Hi,

thanks, never went away just low profile for a while
Okay so you got most things in place, you mention php-fpm, what php version were / are you running?
Did you get a chance yet to check all your log files?

Regards,
Eric

Hi,

are you using Windows or Linux VPS?
Do you have your firewall setup correctly?
Do you have any "Ban" system in place?
What servers do you have running on your VPS?
Check your Mailserver logs to see if it was a mail attack (possible spam through your email server)

Regards,
Eric

Hey Eric, how've you been
Linux ubuntu
firewalls - check
Do you mean something for the ssh? fail2ban? - check

My server monitor showed a 100% CPU usage, and it was my php-fpm that was using all the resources.. so should not be a mail attack

[SteveJohnson]

Hi,

thanks, never went away just low profile for a while
Okay so you got most things in place, you mention php-fpm, what php version were / are you running?
Did you get a chance yet to check all your log files?

Regards,
Eric

Eric,
I currently run PHP 5.5.9, which comes by default with ubuntu 14.04. I don't prefer to mess with the versions that don't come with the OS.. i've had pain in the butthole in the past regarding this.

I am disappointed with Cloudflare. I atleast expected an alert from them.

The nginx log files show a part where i'd like input from others. Here it is -

Code: [Select]

2016/04/02 17:20:57 [error] 31836#0: *11631 access forbidden by rule, client: my.ip.address, server: mysite.com, request: "POST / HTTP/1.1", host: "mysite.com", referrer: "Osclass (v.361)"
2016/04/02 17:20:57 [error] 31836#0: *11497 FastCGI sent in stderr: "PHP message: PHP Warning:  session_start(): The session id is too long or contains illegal characters, valid characters are a-z, A-Z, 0-9 and '-,' in /usr/share/nginx/html/oc-includes/osclass/core/Session.php on line 47" while reading response header from upstream, client: another.IP.address, server: mysite.com, request: "GET /user/login HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "mysite.com", referrer: "http://mysite.com/user/login"
2016/04/02 17:20:57 [error] 31836#0: *11497 FastCGI sent in stderr: "PHP message: PHP Warning:  Unknown: The session id is too long or contains illegal characters, valid characters are a-z, A-Z, 0-9 and '-,' in Unknown on line 0
PHP message: PHP Warning:  Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php5) in Unknown on line 0" while reading upstream, client: another.IP.address, server: mysite.com, request: "GET /user/login HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "mysite.com", referrer: "http://mysite.com/user/login"
Let me know what you think.

[malikin]

set .htaccess

ErrorDocument 403 /specific_page.html
order deny,allow
deny from all
allow from 111.222.333.444your ip


and check if cpu is still 100%

[dev101]

Hi, sorry for the late reply, If your server/website(s) are under some heavy traffic attack, there are only few things you can do, really:

[1] Best option - run DDoS protection, this depends on your hosting, most of the time it is not free, because the excessive traffic spends bandwidth, and bandwidth is not free, someone has to pay for it. Some (rare) hosting companies provide it for "free", at the expense of more $/month included in the plan etc.

[2a] Second option is of limited effectiveness, and you need to work directly with your server firewall to prevent traffic from certain IPs or ranges, IF they can be identified and streamlined. Otherwise, it may be pointless.

[2b] Blocking these abnormal requests/sec on the Apache or PHP level, for example, is not as effective as dedicated DDoS protection, simply because it will still use your's server bandwidth and resources to a point, your server (Apache/Ngnix/etc.) and PHP will still perform some initial workload that counts. I mean, it surely will help (for example, I use self-developed protection of this sort for all of my websites as a first-line of defense, automatically monitoring requests and rejecting abnormal traffic amount), but much better option is [1].

[3] CloudFlare offers some basic protection on free accounts, might be worth a trial during the attacks. But otherwise, there are mixed results from using CF service, and you will probably have to disable a lot of their optimizations, because it may break your scripts, themes, plugins etc.

Regards

[SteveJohnson]

Hi, sorry for the late reply, If your server/website(s) are under some heavy traffic attack, there are only few things you can do, really:

[1] Best option - run DDoS protection, this depends on your hosting, most of the time it is not free, because the excessive traffic spends bandwidth, and bandwidth is not free, someone has to pay for it. Some (rare) hosting companies provide it for "free", at the expense of more $/month included in the plan etc.

[2a] Second option is of limited effectiveness, and you need to work directly with your server firewall to prevent traffic from certain IPs or ranges, IF they can be identified and streamlined. Otherwise, it may be pointless.

[2b] Blocking these abnormal requests/sec on the Apache or PHP level, for example, is not as effective as dedicated DDoS protection, simply because it will still use your's server bandwidth and resources to a point, your server (Apache/Ngnix/etc.) and PHP will still perform some initial workload that counts. I mean, it surely will help (for example, I use self-developed protection of this sort for all of my websites as a first-line of defense, automatically monitoring requests and rejecting abnormal traffic amount), but much better option is [1].

[3] CloudFlare offers some basic protection on free accounts, might be worth a trial during the attacks. But otherwise, there are mixed results from using CF service, and you will probably have to disable a lot of their optimizations, because it may break your scripts, themes, plugins etc.

Regards

Thank you for the reply. I have recently installed "zbblock", which is doing a great job is keeping the bad bots away. I use a basic CLoudflare plan, which comes with a basic ddos protection (so they say). But as of now, things seem to be going well.

Could you tell me what this section of my nginx log means? -

Code: [Select]

[b]2016/04/02 17:20:57 [error] 31836#0: *11631 access forbidden by rule, client: my.ip.address, server: mysite.com, request: "POST / HTTP/1.1", host: "mysite.com", referrer: "Osclass (v.361)"
2016/04/02 17:20:57 [error] 31836#0: *11497 FastCGI sent in stderr: "PHP message: PHP Warning:  session_start(): The session id is too long or contains illegal characters, valid characters are a-z, A-Z, 0-9 and '-,' in /usr/share/nginx/html/oc-includes/osclass/core/Session.php on line 47" while reading response header from upstream, client: another.IP.address, server: mysite.com, request: "GET /user/login HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "mysite.com", referrer: "http://mysite.com/user/login"
2016/04/02 17:20:57 [error] 31836#0: *11497 FastCGI sent in stderr: "PHP message: PHP Warning:  Unknown: The session id is too long or contains illegal characters, valid characters are a-z, A-Z, 0-9 and '-,' in Unknown on line 0
PHP message: PHP Warning:  Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php5) in Unknown on line 0" while reading upstream, client: another.IP.address, server: mysite.com, request: "GET /user/login HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "mysite.com", referrer: "http://mysite.com/user/login"[/b]
Many thanks

[Aficionado]

Code: [Select]

[b]2016/04/02 17:20:57 [error] 31836#0: *11631 access forbidden by rule, client: my.ip.address, server: mysite.com, request: "POST / HTTP/1.1", host: "mysite.com", referrer: "Osclass (v.361)"
2016/04/02 17:20:57 [error] 31836#0: *11497 FastCGI sent in stderr: "PHP message: PHP Warning:  session_start(): The session id is too long or contains illegal characters, valid characters are a-z, A-Z, 0-9 and '-,' in /usr/share/nginx/html/oc-includes/osclass/core/Session.php on line 47" while reading response header from upstream, client: another.IP.address, server: mysite.com, request: "GET /user/login HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "mysite.com", referrer: "http://mysite.com/user/login"
2016/04/02 17:20:57 [error] 31836#0: *11497 FastCGI sent in stderr: "PHP message: PHP Warning:  Unknown: The session id is too long or contains illegal characters, valid characters are a-z, A-Z, 0-9 and '-,' in Unknown on line 0
PHP message: PHP Warning:  Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php5) in Unknown on line 0" while reading upstream, client: another.IP.address, server: mysite.com, request: "GET /user/login HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "mysite.com", referrer: "http://mysite.com/user/login"[/b]
Many thanks

It could mean that you are not under attack of any kind, and all this is just a badly configured server.

[SteveJohnson]

For these particular log entries ...
 my cache-headers were set to "no", and the expires header was set to a value in the past (1981)..i dont know why my default nginx settings were these. So, it was a contradiction, when the cache was expiring as soon as it used to hit the server.. and the cache-control was basically not saving anything.

[dev101]

line 1. probably bot or someone sniffing around, don't worry
line 2 & 3. probably bot, and well-known problem in Osclass, still not resolved
4. badly configured server, seems that you have much to learn about proper configurations

Regards