After watching the final U.S. presidential debate on Thursday, I checked my RSS reader instead of going straight to bed. I saw a notification from MODX releasing 2.8.1. I was a bit surprised because I just upgraded to 2.8.0 only a few weeks ago. MODX usually takes at least 6 months to put out a minor release. It must be some security issue that they needed to patch.
MODX is the content management system that powered the Scalia Law website. Since it was midnight so no one would be logging into the CMS to make any updates; therefore, it was a good time to do the upgrade. It would only take me about 15 minutes to complete. I followed the procedure like I had always done in the past. Something didn’t go right when the files which were supposed to be merged had prompted me to replace. When I allowed the files to be replaced, the website went down. The files that used to take 15 minutes to merge were now estimated about 55 minutes.
I started to panic, stopped the upload, and tried to figure out what went wrong. I went into our daily backups and re-uploaded the directories I had made the upgrade. The process alone took about half an hour. The website was still down. At this point, I re-upload the entire site as well as the database from two days before. This process took a couple of hours. It was already three in the morning and nothing worked. I called GoDaddy, our hosting provider, to see if there’s anything they could do. The first time I called, the technician recommended that I try to re-upload the site from a week old. If that would also fail, I could request a disaster recovery.
I went on Twitter and tweeted for help from the MODX community. I received no response. Its community is way too small. Around six in the morning I started to get messages from colleagues telling me that the site had been down. I explained the situation to my supervisor and her supervisor. They completely understood. Since I could not get any help the last resort was to ask GoDaddy to perform a disaster recovery. The site remained down until around 4:45 pm on Friday. The site went down for 16 hours and I feel horrible. It was all my responsibility alone to bear. I did not sleep that night and kept checking my phone for a miracle to happen.
Although the website is now backed up working, we are set back 3 weeks. GoDaddy only keeps a snapshot every two to three weeks. Now I have to restore what was missing and it is a pain. I am also causing other people to redo some of their work as well.
After this incident, I now determine that we will need to get off MODX. This CMS is dying. It could never get beyond a couple of thousand enthusiasts. I should have called the shot a long time ago, but I held on the hope for it. Now it has become cleared that I need to make an exit strategy. It’s time to migrate to WordPress.