Converting latin1 to utf8

If you want to find the quick tutorial, scroll down to the bottom of the article. Here’s some background of the issue I faced.

Last night before going to bed around 10:30 pm, I made a tiny mistake that kept me up until three something in the morning. I logged into iLoveNgocLan.com’s WordPress Admin to post an article. Instead of doing what I intended to do, I changed the encoding for pages and feeds from iso-8859-1 to utf-8. I also set the template header to <meta charset="utf-8">. Vietnamese text still rendered fine until I made a new post. Every word contained question marks. When I edited an existing article, it also turned into question marks.

I freaked out a bit and went back to change back the encoding for pages and feeds. I freaked out a bit more when the option had disappeared in the Settings Reading Screen. It turned out that WordPress has removed that option in version 3.5.0. I suspect that the option was still there because it was set to iso-8859-1 instead of the default utf-8. I edited wp-config.php to change the encoding back to define('DB_CHARSET', 'latin1');. To my dismay, all the Vietnamese texts displayed black diamonds with question marks. It appeared that going back was not possible; therefore, I googled to find out how to convert latin1 to utf8, something I should have done a long time ago.

In my search, I came across Varun Shrivastava’s “How to Fix Weird Characters Seen on WordPress Blog?”, which seems to be straightforward. I follow his instructions using phpMyAdmin. I exported the existing database into UTF-8 encoding. I created a new database with UTF-8 Collation. Then I imported the data back into the new database. Unfortunately, that didn’t solve the issue. The question marks still showed up on new posts, edited articles, and new comments.

Then I followed WordPress’s documentation on “Converting Database Character Sets” using the new database I just created. I ran the following SQL command in phpMyAdmin to change the default charset of the database:

ALTER DATABASE MyDatabaseName CHARACTER SET utf8;

It worked, so I ran the following SQL command to change the default charset of individual tables:

ALTER TABLE wp_posts CHARACTER SET utf8;

Then I ran the following SQL command to run individual columns:

alter table wp_posts change post_content post_content LONGTEXT CHARACTER SET utf8;

Unfortunately, I have to run TEXT, LONGTEXT, TINYTEXT, VARCHAR, ENUM for each individual column for each individual table. It would be a lot of work to do manually and when I got to TINYTEXT, it wiped out all the body text from the article. I gave up and went to bed around 3 something in the morning.

I woke up again around 7 am and tried to figured out what I else I could do. I opened up the SQL file I exported a few hours earlier and performed a find-and-replace to convert latin1 into utf8. 19 instances were found and replaced. I created a new database in phpMyAdmin and imported the new search-and-replaced file. To my surprise, it worked as expected. Take a look at iLoveNgocLan.com. I created new posts, edited old posts, added new comments, and the text came up normal, no more question marks and no more black diamonds.

Essentially, to convert latin1 to utf8, this is all you need to do in phpMyAdmin:

  1. Export your existing database to your local machine
  2. Find latin1 and replace with utf8 on your local machine
  3. Create a new database in phpMyAdmin with UTF-8 Collation
  4. Import the SQL file from your local machine to the new database in phpMyAdmin
  5. Run this SQL command in phpMyAdmin: ALTER DATABASE MyDatabaseName CHARACTER SET utf8;
  6. Edit your wp-config.php to point to the new database and edit define('DB_CHARSET', 'utf8');
  7. Update the meta tag in your theme’s header: <meta charset="utf-8">

That’s it. I don’t know why this method isn’t available already. I wonder if there’s any drawback of using this method, but it seems to work for me. I am not a MySQL expert. In fact, I don’t have much confidence in messing around with it. I lost several hours of sleep, but the learning experience is worth it.

No JavaScript No Content

On my iPhone, I continue to use Safari without JavaScript to browse the web. In addition to hamburger menus not working, I have come across a bigger issue. Some of the sites I visited are completely blank with JavaScript turned off. What even more shocking is that sites with typography are completely empty.

I started off with Frere-Jones Type and it is completely blank. Then I headed over to Kilotype, which designed by the same company that created Frere-Jones Type, and it is also completely blank. H&Co is completely blank. Future Fonts is completely blank. Even Google Fonts is completely blank.

I don’t have the courage to go on. As CSS fonts continue to advance and mature, I wonder why these sites rely solely on JavaScript to deliver their content. I will continue to use Safari on my iPhone to browse sites and read news. I won’t turn on JavaScript unless I absolutely have to.

JavaScript and Hamburger Menus

A couple of weeks ago, I came across a tweet encouraging users to turn off JavaScript on iOS Safari to browse the web on mobile devices. Until I read that tweet, I didn’t know that I could disable JavaScript on my iPhone. I didn’t use Safari much on my phone unless I tapped on a link in email, which would bring up Apple’s default browser. I used mostly Chrome for my own convenience. My bookmarks, histories, and passwords were all synced and managed by Chrome. I tried to deactivate JavaScript on Chrome on my iPhone as well, but I couldn’t find a way to do it.

In the past few weeks, I have been browsing the web using Safari on my phone with JavaScript turned off and it has been such a pleasurable experience. Sites, particularly news, loaded much faster and without annoying ads popping up. On some photo-heavy sites, images didn’t even load without JavaScript. I actually preferred more white space than images. Hero sliders didn’t work. I just saw all the slides stacking on top of each other.

The biggest issue I ran into was the hamburger menus. Without JavaScript, these menus simply didn’t work and I could not go beyond the homepage. Everything hidden under the three horizon lines was inaccessible. We should move away from using JavaScript for hamburger menus or avoid sweeping everything under hamburger menus for the sake of convenience. I have seen sites that used the hamburger menu for just one or two items like “about” and “contact.” For longer navigation, we can use CSS grids, flex boxes, and variable fonts to control the items. A variable font with the width axis can be useful for menu items. Its narrow width can save some space on small mobile devices. It can get wider for larger screens.

For sites with tons of menu items, I don’t know what the solution is. I implemented the anchor-link technique for the Scalia Law School website years ago. Basically, I placed the menu items in the footer for mobile devices and positioned it to the top with CSS for larger screens. It still works fine without JavaScript.

My Relationship With WordPress

Over the weekend, our Senior Associate Dean asked me to come up with a message board for the Scalia Law School’s class of 2020. Since the graduates won’t get to have a formal graduation ceremony in May, she would like us to create a special page to let faculty, staff, and administrative members post messages to the graduates. My immediate solution was to create a WordPress page with comment section enabled.

On Monday, I spun up a new site from our WordPress multisite and activated the Scalia Law 2019 theme, which is based off WordPress’s Twenty Nineteen official theme. When Twenty Nineteen first released, I created a child theme to have our own brand, which includes typography, colors, and our logo. For everything else, I depended on the parent theme. Twenty Nineteen is beautiful out of the box and it uses the new Gutenberg block editor.

Within a few hours, I created a page addressing our graduates, “Congratulations Class of 2020! An Extraordinary Class in Extraordinary Times.” I used a big cover image and made the text huge. We launched the site on Tuesday and the messages have been rolling in. I love reading them even though I am not a graduate. The messages are wonderful.

WordPress has been a great asset to my professional career. It has helped me provide many solutions to the needs of the school. Now that the entire network of over 30 sites is hosted on WP Engine, courtesy of the University, I don’t have to worry about the backend. I still have full control the themes, plugins, and full SFTP access. Some IT members at the Law School had criticized me for giving up hosting the server part of the sites, but there’s no way I can run the server as reliable as WP Engine. It would be a huge undertaking and I am not a server administration. If the University offers this huge service at no cost, why not taking advantage of it?

WordPress is great at solving problems that do not required original design. I could get pretty far with some changes to make the templated design suits my brand. Of course, I could create the entire WordPress theme from scratch, but that would required tremendous time and technical investments. For my own personal use, WordPress is far too complex. This blog, for instance, probably uses about five percent of WordPress’s powerful features. This blog has been powered by WordPress since 2003 and it hasn’t changed much over the years. I am still using the classic editor. I still code the theme using HTML and CSS and with only a minimum amount of PHP. I have control of every code I input. Developing a new theme, even from a starter theme, isn’t as simple anymore; therefore, I no longer offer WordPress for freelance clients. The amount of customization is just too much. Of course, I can still do it if I get big projects, but not for my typical clients.

I still love WordPress, but my development has changed. I am now happier to use WordPress as a tool to solve technological solutions instead of trying to offer WordPress as design solutions for client projects. It is a change in perspective that I have to come to term with if I continue to use WordPress. There are other choices out there, but WordPress remains a tough contender in the web space.

My choice is either WordPress or hand code HTML and CSS with some PHP to keep the pages manageable. I am missing the entire trend of static site generators. When web designers and developers moved their sites or blogs to Jekyll, Hugo, or Eleventy, I still keep my blog on WordPress. When they spent countless of hours moving from one static site generator to another or back to WordPress, I am still on WordPress.

Moving From WordPress to Kirby for Client Websites

This blog still runs on WordPress. It’s a theme I have designed and developed years ago in B2, which was the father of WordPress. I coded the theme from scratch using only PHP hooks specifically for my blog. Even to this day, my theme has three files: index.php, style.css, screenshot.png.

These days WordPress has become way too complex to start from scratch. I can still take a starter theme like Gutenberg and go from there, but it already packed too many things I don’t need. I prefer to have control of WordPress instead of the other way around. I want to know exactly how my HTML ended up in the browser. I tried not to sweat it and just lived with whatever an existing theme spits out, but it just feels wrong.

I would love to learn how to make a WordPress theme from scratch using the Gutenberg’s blocks. I have not found any tutorial like that. If you do, please let me know.

Because WordPress has lost me, I can no longer develop clients’ website with it. I turn to Kirky instead. Kirby is not free, but it is worthwhile paying for. Kirby allows me to stand up a site quickly and doesn’t get into the way I design the website. Every piece of HTML is rendered exactly the way I have coded. The best part is that the panel knows which piece can be updated by content editors. As a result, Kirby is an ideal CMS for a small websites.

Catching Up With Gutenberg

I am glad that the release of WordPress 5.0 has been pushed back. After testing out Gutenberg, the new, controversial editor, I can see why Matt Mullenweg is pushing hard to get it into core. Gutenberg will define WordPress as a powerful CMS and no longer just a blogging platform. It gives users more flexibility to create richer experience.

I can also see why it is alienating many designers and developers. For this blog, which I have intended to keep simple from the start, I won’t be using Gutenberg. I have no photos, no video, no audio, and no gallery. It is just simply text. I do put up large hero images once in a while, but they do not go into the database. I want the complete control of the text. My theme still just have 3 files. I don’t see the need to use Gutenberg, but I will see if I can develop a theme from scratch like I always had in the past decade just for this site.

I played with the new Twenty Nineteen. It looks impressive, but the theme has tons of files. I might be wasting my time developing a theme from scratch. I just have to roll with what already developed and just customize it for my needs. Once WordPress 5.0 is out and Twenty Nineteen is officially released, I’ll use it to develop a theme for Scalia Law School. It will be beneficial for the school sites than my own personal site.

Gutenberg isn’t solid yet, but it is definitely a step in the right direction. It is understandable that people don’t like new changes, but they will get around to it.

Busy at Work

The past two weeks have been crazy at work. We managed to upgrade to PHP 7.2 on our server for Scalia Law site. I worked with a talented developer who pulled a copy of our current and tested on PHP 7.2 to make sure MODX worked correctly. I am glad we got it done before he left the University to move on to something else.

Today we completed the migration of Scalia Law School’s WordPress Multisites to WP Engine. It was a massive undertaking, but we worked with a vendor to do so.

As far as technical things, we are now in a good spot. I am looking forward to some winding down time over the summer.

Build a Static Blog with Mavo.io

In a three-part video series, Brian Haferkamp walks through the process of building a static blog using Mavo.io. Part three is particularly helpful to see how Mavo.io works. I need to learn this new tool to build websites that will allow clients to update their own content with ease.

Upgrade Law School Website to MODX Revolution

On the front-end, nothing has changed to the Law School website, but the backend is completely upgraded. We migrated the entire site to a brand new dedicated server, upgraded to the latest version of MODX Revolution, and switched to HTTPS. Anthony Harvey is the mad genius who has made all of these happened. He is such an amazing developer. If it was not for him, we would still be stuck in the good ole MODX Evolution. He responded to tons of my questions with patience. I am so grateful for his work.