Consolidated Wordpress learnings HISTORY OF ATTACKS 3 DEALING WITH A PHISHING ATTACK 3 Step 1: Check all incoming mail to identify path. Step 2: ZIP the offending path and save it for your record, then delete Step 3: Delete all email so you will know that the solution has worked Step 4: Full check of the system 3 3 3 3 RECOVERY FROM HACKING OR DAMAGE OF BLOG 3 Step 1: Download and backup whatever you can Step 2: Troubleshoot the source of the problem 4 4 1) The SQL database (user) might have got corrupted Scenario 1: "Sorry, but you don’t have the administrative privileges needed to do this." Scenario 2: Can’t log in and can’t reset email Solution: Fix the user in phpMyAdmin SQL database 4 4 4 4 2) Restoring the SQL database Step 1: Upload the database using FTP Step 2: Restore using Shell 5 5 5 3) Wordpress might have got corrupted Step 1: Reinstall WordPress Step 2: Upload wp-content files from Amazon S3 backup. Note for Indiapolicy.org Step 3: Upload the SQL file saved in Amazon S3 5 6 6 6 6 3) Theme might have got corrupted Solution: Rename to whitehouse2, reinstall fresh theme 6 6 4) Defective .php file 6 5) Plugin incompatibility 7 6) Upgrade of WordPress does not work properly 7 7) Funny characters in the blog after upgrade 7 8) Language setting can get disrupted 7 9) Default search feature comes back (needs to be disabled) 8 STRENGTHEN BLOG SECURITY 9 Step 1: Find bad files 9 Step 2: Install security plugins 9 Step 3: Keep Wordpress installations up to date 9 Step 4: General analysis of the website for security gaps DATA, VIDEO, AND GENERAL WEB BACKUP 9 10 Personal data Critical files: Videos: 10 11 11 Web backup Control Panel backup Full FTP backup Compressed directory backup 11 11 11 11 WORDPRESS BACKUP: THREE THINGS 11 Wordpress Files a)Wordpress export: b) Automatic backup to Amazon S3: 12 12 12 Wordpress database 12 Root files 13 History of attacks From February 2011 onwards, my domains have been hacked repeatedly. Twice in the space of one month the IPI website was hacked and "phishing attack" launched, requiring complete reinstallation. But not just IPI, all domains have had multiple hackings. (Note: There were lots of files in /public_html/indiapolicy.org that have modification timestamp of Feb 7. And in those 7th Feb files there was a malicious index.php (default file when browser opens http://indiapolicy.org/). Possibly this site was hacked on Feb 7.) Dealing with a phishing attack That is when someone is using your website to attack other websites/ extract passwords, etc. Step 1: Check all incoming mail to identify path. Use HORDE: The path from where these mails are bouncing is usually indicated in these emails. Step 2: ZIP the offending path and save it for your record, then delete Once identified first download, zip and backup these suspicious files – and save in E:\WEB-BKUP\WEB-ATTACKS which will be automatically backed up on Carbonite – in case there is need for an audit trail (I don't have backups of whatever was planted on our server prior to 10 April 2011). Apparently some of these files can provide info on the attacker's identity. Then delete them from the website. Note that this does not mean they won't come in again! They are driven by a process that may source them from somewhere on the internet. Step 3: Delete all email so you will know that the solution has worked Make sure you delete and empty the Horde folder so that you can monitor if any other emails are bouncing. This is a very useful to detect whether the attack has been stemmed. Step 4: Full check of the system From the shell, check all zip files (the phishing attack usually puts in a zip file that expands into the fake website). Recovery from hacking or damage of blog WordPress disasters are just too frequent. WordPress is not like an HTML system which is easy to restore. Wordpress is very vulnerable to all kinds of mishaps. This system is a nightmare! Step 1: Download and backup whatever you can Start a download from FTP, just to be safe (if you can!). This should not be necessary, since you should have an active backup system, anyway. Step 2: Troubleshoot the source of the problem 1) The SQL database (user) might have got corrupted Hackers tend to change username and email of the blog user. Locked out of the blog! Here's what to do if you are locked out of the blog. Scenario 1: "Sorry, but you don’t have the administrative privileges needed to do this." Same instructions as below. Scenario 2: Can’t log in and can’t reset email What is to be done if a hacker steals your password or otherwise hacks into the blog and prevents you from logging in? This happened to me on 26 March 2011. I suddenly found I could not log into sabhlokcity.com and was told that my password is wrong. Password reset would not work since the hacker had changed the email. Solution: Fix the user in phpMyAdmin SQL database Fix the password (details here) (select user/ browse/ edit). In my case the hacker had changed email to something in france (?). I changed back to my email ID, then used the login screen to reset the password. Then go into the blog and change to another password. DETAILS Note the image of the hacked screen (I've BLANKED OUT MY USERNAME and other security information) but allowed the hacker's fake ID to be visible). Through phpMyAdmin Begin by logging into phpMyAdmin and click databases. Image #3 All the tables in your database will appear. If not, click Structure. Look for wp_users. Click on the icon for browse. Locate your Username under user_login Click edit Image #5 Check that your password is actually correct, and that MD5 is in the box. Click the 'Go' button to the bottom right. Test the new password on the login screen. If it doesn't work, check that you've followed these instructions exactly. MAKE SURE TO USE THE MD5 OPTION – without it the password WON’T WORK 2) Restoring the SQL database Typically you lose widgets/ widget area after a hacking incident. Reconstructing them is not possible without uploading back an old database. But the database greater than 50MB can’t be restored using phpmyadmin. Tables are individual entities in a database. eg: in the wordpress database tables are users, posts, settings etc. Step 1: Upload the database using FTP Step 2: Restore using Shell Use restore.sh in ~/sabhlok-backup. To restore dbdata.sql run: ./restore.sh dbdata.sql. Note: the full command is outlined here. An old database was uploaded, but unfortunately it did not have the widget arrangement. One last go – at using a later version of the backup. 3) Wordpress might have got corrupted In some cases the hacker not merely replaces the user name and account but deletes key files. That can made the wordpress installation defunct. Step 1: Reinstall WordPress This may not always be necessary. At times this may require deleting existing installation and reinstalling it. Step 2: Upload wp-content files from Amazon S3 backup. It is crucial to upoload the wp-content folder and uploads folder from the Amazon backup. (if prosumer theme doesn't work, then reinstall it) Note for Indiapolicy.org Download and install prosumer theme again. The two key files are head.jpg (the main header) and you.jpg which replaces the woman with a cherry. Fortunately the Amazon backup contains these files. That had to be FTP'd, and for some reason it would not overwrite existing (new) files. That meant a forced upload. Step 3: Upload the SQL file saved in Amazon S3 Go to myPhPadmin, then relevant database, and then upload the SQL file that has been saved under uploads on Amazon. This works for small database files. 3) Theme might have got corrupted Solution: Rename to whitehouse2, reinstall fresh theme Go to myPhPadmin, then relevant database, and then upload the With great effort I found the core file (under the wordpress theme – whitehouse) and changed the functions_libraray.php file. That, however, did not help. So I renamed whitehouse theme to whitehouse2, and lo and behold worpress came alive. That was the problem, so I reinstalled a whitehouse version. And saved the XML file. This happened again recently when all widgets seemed to disappear. The solution was: Rename to whitehouse2, install a fresh version, and delete whitehouse2. Sabhlokcity.com instantly came back to life. Note that when the theme is re-installed, some settings disappear. Reset them inside the theme options: Header: http://sanjeev.sabhlokcity.com/PICS/sabhlok-blog.jpg Favicon: This is best fixed by going to the wp-content backup and over-writing the previous .ico image. If you are copying all older files across this should occur automatically. 4) Defective .php file The last time I had a peculiar problem I had to overwrite a main directory (not subdirectory) .php file (I forget which). Everything came back. NOTE THAT THE MOST IMPORTANT FILE IS CONFIG.PHP 5) Plugin incompatibility A typical problem is plugin mismatch. When that happens, all sorts of weird things can happen. Solution: 1) FTP to the site and rename the plugin folder as something else. This inactivates all plugins. 2) Re-active plugins one at a time and check which is causing grief. Since I have 50 odd plugins this is quite a nuisance. It pays to DELETE plugins that you are not using. In my case, I'm building this blog post to record all the ones that I'll be using. Everything else needs to be deleted. 6) Upgrade of WordPress does not work properly I upgraded WordPress (it asked me to do so!) and now I get this msg: "Sorry, but you don’t have the administrative privileges needed to do this." That was a disaster! I searched the internet and found this solution. Go into Themes (under wp-content) to your theme. Within that you'll find 'core' then under /functions you'll find the functions_library.php file. Open the file for editing. You will see Quote: function checkauthority(){ if (!current_user_can('edit_themes')) wp_die('Sorry, but you dont have the administrative privileges needed to do this.'); } Replace it with Quote: function checkauthority(){ if (!current_user_can('edit_themes')) ; } This worked for me. I find this whole WordPress thing such a mess, but one has to learn to live with it. 7) Funny characters in the blog after upgrade In the wp-config.php, change define(’DB_CHARSET’, ‘utf8′) to define(’DB_CHARSET’, ‘utf-8′) 8) Language setting can get disrupted Open up ‘wp-config.php’ from the root directory of your WordPress installation.Add ‘//’ at the very beginning of these two lines:define(‘DB_CHARSET’, ‘utf8′);define(‘DB_COLLATE’, ”);So that section should now look like this://define(‘DB_CHARSET’, ‘utf8′);//define(‘DB_COLLATE’, ”); I accidentally put the double // on a previous line. Be careful. This does work. 9) Default search feature comes back (needs to be disabled) 1) Remove everything from searchform.php <form method="get" id="searchform" class="" action="<?php bloginfo('home'); ?>/"> <fieldset> <input type="text" value="<?php _e('Search',TDOMAIN);?>" name="s" id="s" onfocus="if (this.value == '<?php _e('Search',TDOMAIN);?>') {this.value = '';}" onblur="if (this.value == '') {this.value = '<?php _e('Search',TDOMAIN);?>';}" /> <input type="image" value="Go" src="<?php echo THEME_IMAGES;?>/searchbtn.png" class="submit btn" /> </fieldset> </form> 2) Delete the stuff in red from I forget which php file. <div id="sidebar" role="complementary"> <?php global $sidesearch; if($sidesearch):?> <div id="sidesearch" class="fix"> <?php include (THEME_LIB . '/_searchform.php'); ?> </div> <?php endif;?> <div id="widgets"> <?php if(VPRO) include(THEME_LIB.'/_grandchildnav_pro.php');?> <?php if(pagelines('the_sidebar', $post->ID) == 'secondary'):?> <?php if ( !function_exists('dynamic_sidebar') || !dynamic_sidebar('Secondary Sidebar') ) : ?> <?php _e('The secondary sidebar has been selected but doesn\'t have any widgets. Add some widgets to your secondary sidebar in the admin under appearance > widgets.',TDOMAIN);?> <?php endif; ?> <?php elseif(pagelines('the_sidebar', $post->ID) == 'short'):?> <?php if ( !function_exists('dynamic_sidebar') || !dynamic_sidebar('Short Sidebar') ) : ?> <?php _e('The short sidebar has been selected but doesn\'t have any widgets. Add some widgets to your short sidebar in the admin under appearance > widgets.',TDOMAIN);?> <?php endif; ?> <?php else:?> <?php if ( !function_exists('dynamic_sidebar') || !dynamic_sidebar() ) : ?> <?php if(!pagelines('sidebar_no_default')) include(THEME_LIB.'/_defaultsidebar.php');?> <?php endif; ?> <?php endif;?> </div> </div> Strengthen blog security Step 1: Find bad files Using shell, look for suspicious .exe /zip file and delete them (keep backup). Step 2: Install security plugins - WP Security Scan(which pointed out I had a number of security holes, particularly not having .htaccess files) - BulletProof Security – this installs .htaccess files. If plugin directories are visible to external people they can crack the security loophole of the lowest security protected plugin (i.e. older version plugins). Once they can get in, apparently then can get shell access. Step 3: Keep Wordpress installations up to date Make sure that all installations are up to date. Step 4: General analysis of the website for security gaps Start by looking at what files were modified around March 23rd when the first phish attack occurred – might give some insight into how the system was compromised. Check the shell histories of all your users; its possible it will contain the steps the attacker took to setup a backdoor onto the system to maintain control. I'd also change the passwords of all accounts on the system, etc. I've often found its a piece of PHP software that is vulnerable – I've seen hackers use vulnerabilities in phpMyAdmin, dokuwiki, phpBulletin, etc.. to gain access to the operating system. Websites that provide hints on security: How to recover from sql injection attack useful blog post on WP security another one. very useful information Stop badware blog Code injection SQL injection Cross-site scripting types of vulnerabilities: Web Servers, HTTP Methods, SQL Injection, CRLF Injection, Cookie Manipulation, Script Language Error, cross-Site Scripting Commercial web security services FREE http://nmap.org/ Full list: http://www.alken.nl/online-security-check.htm http://www.alken.nl/online-security-check.htm PAID http://www.mavitunasecurity.com/netsparker-web-app-scanner/ Data, video, and general web backup Personal data ESTIMATE OF THE VALUE OF FAMILY PHOTOS AND PERSONAL DOCUMENTS Technology provides benefits but imposes significant costs (in money and time). PHOTOS: When film rolls were expensive, each camera would take about 20 rolls of film on average. So if a cheap camera cost USD$100 (in those days) and each roll including processing cost $15, then 20 rolls (say 500 shots in all) cost $400 in all, or $1 per shot. In today's terms, each shot is worth at least $5 in current value. Once digital computers came, then photos multiplied. By now I have 14,000 pictures in my personal records. Even if the marginal cost of these pictures is very low, the fact is that at least 10 cameras have been used by now, and plenty of other raw material (rolls, batteries, albums, etc. etc.). Total value is at least $10,000 if not more. If I add the cost of videos (on Sony Handycam, and the new one) then the cost of the capital and raw material is at least another $2500 in today's value. Altogether around $12,500 in COST in today's value. But what if the house were burnt and these things lost? What would be the value you'd be willing to pay to recover these? At least a quarter of my remaining savings. So the true value of these things is enormous. And what about one's writings? While books are replaceable, one's unpublished writings are not. The total value of these things is huge. Fortunately much of this material is now available in digital form. In brief, it is OF GREAT VALUE in BACKUPING THIS MATERIAL DILIGENTLY AND, to the extent possible, AUTOMATICALLY. So far I've been buying spare hard disks (costing each time well over $200 in current value), and these last for about 2-3 years. The problem is that this material is perishable as well. What if a fire takes place? Then everything will be lost. What if I could dispense with hard disks almost entirely, and resort entirely to the cloud? What would I be willing to pay for the power to store my valuable data on the cloud? My guess is that I'd be willing to pay QUITE A BIT, at least $100 per year in current dollar value. Carbonite: ALL PERSONAL BACKUPS [unlimited, trying first year for around $55 or so]. This includes ALL pictures, videos, and data including full FTP backups. Critical files: DriveHQ: ALL CRITICAL FILES [ up to 1 GB is free ] it is genuine FTP and very convenient – better than Cloudberry. Videos: Carbonite is slow, so back up on HDD. S3 might be useful once its prices fall down further. Web backup An AMAZING amount of time is used to build websites/ improve them/ add data/ etc. But there is little or no time spent on backing them up. That is find in most cases, but there are instances when webhosting companies collapse. In that case a systematic alternative backup process is of great importance. The only sensible way to backup the website is to use FTP. The automatic backup system of the web hosting company DOES NOT DO A FULL BACKUP. Control Panel backup There is an automatic control panel backup that can be used in an emergency. This assumes that (a) this is a full backup (often it is not) and (b) the web hosting company does not collapse. In the case of downtownhost, I've got access to a CDP server (R1 Soft Restore backups) [see the tutorial here] I've used this twice and it has been a saviour. But web hosting companies collapse (has happened twice already!), and other errors can occur. So it is crucial to have alternative backups Full FTP backup This is recommended once every often (although it can take a huge amount of time). This will also help when the web hosting company collapses, which is not an infrequent phenomenon (Note that CDP server (R1 Soft Restore backups) [see the tutorial here] IS NOT A FULL BACKUP – as I've found from sad experience). Compressed directory backup This can be done for separate main directories, periodically. Wordpress backup: THREE THINGS WordPress blogs are particularly finicky. A small change in php disrupts the entire system. Backups are absolutely critical. A) WORDPRESS FILES WordPress Core Installation WordPress Plugins WordPress Themes Images and Files Javascripts, PHP scripts, and other code files Additional Files and Static Web Pages B) WORDPRESS DATABASE C) ROOT FILES ALL THREE MUST BE BACKED UP. Wordpress Files a)Wordpress export: Regularly export the XML file containing the blog posts. This file is NOT backed up automatically. The export function is just not that reliable. If you have a large database, it won’t work sometimes. Also, it doesn’t backup the images, so you have to copy the entire wp-content folder instead of just the theme directory. I have found this to be TOTALLY PROBLEMATIC. Half the time when it exports it creates spurious duplicate posts (with -2 appended in the URL). Once it imported only a few blog posts because others were in the ‘trash’ area. Another time it converted tens of published posts into “draft” posts. A total mess. In particular, if a post is in the ‘trash’ section, the import function thinks it has already imported the blog post – and so unless you clean out the trash (an enormous challenge when blog posts increase in size), you are effectively going to face a HUGE manual task. And at the end you don’t know what you’ve lost. The amount of time I’ve lost using the pathetic “export”/import function in Wordpress boggles the mind. NEVER use it if you can avoid it. b) Automatic backup to Amazon S3: ALL MAIN BLOGS are backup up using WP S3 backups (S3 backup widget: Automatic backup doesn't work) to Amazon S3 [up to 5 GB is free]. Cloudberry gives regular access. To that is attached a free Amazon website. Here is the post that describes it. Limitations: Amazon backup doesn’t work well. Amazon backup showed 24 March but the data was restored only till 21 January. I don't know what's going wrong. Clearly the S3 Backup plugin is NOT working. That is a serious problem. Wordpress database Periodically take a backup of the datbase using phpMyAdmin Root files There is NO system available to backup root files. You need to use FTP regularly. The good thing is that the root files don't change much. Second, you can restore them in desperate condition by deleting the WordPress installation and then reinstalling a fresh version.