Converting latin1 to utf8

If you want to find the quick tutorial, scroll down to the bottom of the article. Here’s some background of the issue I faced.

Last night before going to bed around 10:30 pm, I made a tiny mistake that kept me up until three something in the morning. I logged into iLoveNgocLan.com’s WordPress Admin to post an article. Instead of doing what I intended to do, I changed the encoding for pages and feeds from iso-8859-1 to utf-8. I also set the template header to <meta charset="utf-8">. Vietnamese text still rendered fine until I made a new post. Every word contained question marks. When I edited an existing article, it also turned into question marks.

I freaked out a bit and went back to change back the encoding for pages and feeds. I freaked out a bit more when the option had disappeared in the Settings Reading Screen. It turned out that WordPress has removed that option in version 3.5.0. I suspect that the option was still there because it was set to iso-8859-1 instead of the default utf-8. I edited wp-config.php to change the encoding back to define('DB_CHARSET', 'latin1');. To my dismay, all the Vietnamese texts displayed black diamonds with question marks. It appeared that going back was not possible; therefore, I googled to find out how to convert latin1 to utf8, something I should have done a long time ago.

In my search, I came across Varun Shrivastava’s “How to Fix Weird Characters Seen on WordPress Blog?”, which seems to be straightforward. I follow his instructions using phpMyAdmin. I exported the existing database into UTF-8 encoding. I created a new database with UTF-8 Collation. Then I imported the data back into the new database. Unfortunately, that didn’t solve the issue. The question marks still showed up on new posts, edited articles, and new comments.

Then I followed WordPress’s documentation on “Converting Database Character Sets” using the new database I just created. I ran the following SQL command in phpMyAdmin to change the default charset of the database:

ALTER DATABASE MyDatabaseName CHARACTER SET utf8;

It worked, so I ran the following SQL command to change the default charset of individual tables:

ALTER TABLE wp_posts CHARACTER SET utf8;

Then I ran the following SQL command to run individual columns:

alter table wp_posts change post_content post_content LONGTEXT CHARACTER SET utf8;

Unfortunately, I have to run TEXT, LONGTEXT, TINYTEXT, VARCHAR, ENUM for each individual column for each individual table. It would be a lot of work to do manually and when I got to TINYTEXT, it wiped out all the body text from the article. I gave up and went to bed around 3 something in the morning.

I woke up again around 7 am and tried to figured out what I else I could do. I opened up the SQL file I exported a few hours earlier and performed a find-and-replace to convert latin1 into utf8. 19 instances were found and replaced. I created a new database in phpMyAdmin and imported the new search-and-replaced file. To my surprise, it worked as expected. Take a look at iLoveNgocLan.com. I created new posts, edited old posts, added new comments, and the text came up normal, no more question marks and no more black diamonds.

Essentially, to convert latin1 to utf8, this is all you need to do in phpMyAdmin:

  1. Export your existing database to your local machine
  2. Find latin1 and replace with utf8 on your local machine
  3. Create a new database in phpMyAdmin with UTF-8 Collation
  4. Import the SQL file from your local machine to the new database in phpMyAdmin
  5. Run this SQL command in phpMyAdmin: ALTER DATABASE MyDatabaseName CHARACTER SET utf8;
  6. Edit your wp-config.php to point to the new database and edit define('DB_CHARSET', 'utf8');
  7. Update the meta tag in your theme’s header: <meta charset="utf-8">

That’s it. I don’t know why this method isn’t available already. I wonder if there’s any drawback of using this method, but it seems to work for me. I am not a MySQL expert. In fact, I don’t have much confidence in messing around with it. I lost several hours of sleep, but the learning experience is worth it.

No JavaScript No Content

On my iPhone, I continue to use Safari without JavaScript to browse the web. In addition to hamburger menus not working, I have come across a bigger issue. Some of the sites I visited are completely blank with JavaScript turned off. What even more shocking is that sites with typography are completely empty.

I started off with Frere-Jones Type and it is completely blank. Then I headed over to Kilotype, which designed by the same company that created Frere-Jones Type, and it is also completely blank. H&Co is completely blank. Future Fonts is completely blank. Even Google Fonts is completely blank.

I don’t have the courage to go on. As CSS fonts continue to advance and mature, I wonder why these sites rely solely on JavaScript to deliver their content. I will continue to use Safari on my iPhone to browse sites and read news. I won’t turn on JavaScript unless I absolutely have to.

Typesetting for Dark Mode

Designing for the dark mode had been an afterthought for me. I simply changed the text and background colors and leave everything else intact—even typography. Because I still preferred reading dark text on light background over light text on dark background, I spent most of my time designing for light mode and I had always loved serif text faces. If I have to read text on dark background, however, I prefer sans serif over serif. As a result, I thought why not changing up the typography as well for dark mode.

If you are reading this site with dark mode, the main text is set in Amica Pro, by Dave Rowland. Last year, I had the opportunity to advise Dave on Vietnamese for Amica Pro. I am not sure if he had a chance to implement my recommendations. The version I am using has not been updated. Although some minor tweaks would be better, but the diacritic design over all is pretty solid. For display text, I am using David Jonathan Ross’s Dattilo DJR Banner, which has a variable font.

If you read this site on dark mode, please let me know your feedback.

Mavo-Less

I am working on a small website for a freelance project. While all the pages are static, I would like to give my client an easy way to update the news section of the site. Implementing a content management system, such as WordPress or Kirby, just to do that is unnecessary.

Over the weekend, I spent an hour revisiting Mavo, which seems like a perfect solution for what I wanted to accomplish. Developed in the Haystack Group at MIT CSAIL and led by Lea Verou, Mavo allows an easy way to update website content with no backend database and without a single line of JavaScript. I gave it a try a few years ago, but got hung up on creating the JSON file in Github. This time, I set up the JSON file in Dropbox and it worked. I was able to make part of the web page editable. Furthermore, I added TinyMCE to give my client an easy way to update the content.

I was thrilled to see how friendly and intuitive it is to update website content with Mavo. This is something I have wanted for many years to offer to my freelance clients, but I don’t have the skill to develop myself. Unfortunately, the huge disappointment came when I viewed the source code. The content section was blank. When I turned off JavaScript, the content also disappeared on the page. I realized that the content was being injected using JavaScript.

If the main content of the page doesn’t show up when JavaScript is turned off, that is a major accessibility issue. Who has JavaScript turned off these days? I have been trying Safari on my iPhone without JavaScript and the experience has been so good that I am continuing to do so. For the website project, I am back to square one again. I should have known that Mavo is a JavaScript library; therefore, it depends on JavaScript to work. Although I can’t implement Mavo, my time spent was not wasted. It was a good learning experience.

JavaScript and Hamburger Menus

A couple of weeks ago, I came across a tweet encouraging users to turn off JavaScript on iOS Safari to browse the web on mobile devices. Until I read that tweet, I didn’t know that I could disable JavaScript on my iPhone. I didn’t use Safari much on my phone unless I tapped on a link in email, which would bring up Apple’s default browser. I used mostly Chrome for my own convenience. My bookmarks, histories, and passwords were all synced and managed by Chrome. I tried to deactivate JavaScript on Chrome on my iPhone as well, but I couldn’t find a way to do it.

In the past few weeks, I have been browsing the web using Safari on my phone with JavaScript turned off and it has been such a pleasurable experience. Sites, particularly news, loaded much faster and without annoying ads popping up. On some photo-heavy sites, images didn’t even load without JavaScript. I actually preferred more white space than images. Hero sliders didn’t work. I just saw all the slides stacking on top of each other.

The biggest issue I ran into was the hamburger menus. Without JavaScript, these menus simply didn’t work and I could not go beyond the homepage. Everything hidden under the three horizon lines was inaccessible. We should move away from using JavaScript for hamburger menus or avoid sweeping everything under hamburger menus for the sake of convenience. I have seen sites that used the hamburger menu for just one or two items like “about” and “contact.” For longer navigation, we can use CSS grids, flex boxes, and variable fonts to control the items. A variable font with the width axis can be useful for menu items. Its narrow width can save some space on small mobile devices. It can get wider for larger screens.

For sites with tons of menu items, I don’t know what the solution is. I implemented the anchor-link technique for the Scalia Law School website years ago. Basically, I placed the menu items in the footer for mobile devices and positioned it to the top with CSS for larger screens. It still works fine without JavaScript.

My Relationship With WordPress

Over the weekend, our Senior Associate Dean asked me to come up with a message board for the Scalia Law School’s class of 2020. Since the graduates won’t get to have a formal graduation ceremony in May, she would like us to create a special page to let faculty, staff, and administrative members post messages to the graduates. My immediate solution was to create a WordPress page with comment section enabled.

On Monday, I spun up a new site from our WordPress multisite and activated the Scalia Law 2019 theme, which is based off WordPress’s Twenty Nineteen official theme. When Twenty Nineteen first released, I created a child theme to have our own brand, which includes typography, colors, and our logo. For everything else, I depended on the parent theme. Twenty Nineteen is beautiful out of the box and it uses the new Gutenberg block editor.

Within a few hours, I created a page addressing our graduates, “Congratulations Class of 2020! An Extraordinary Class in Extraordinary Times.” I used a big cover image and made the text huge. We launched the site on Tuesday and the messages have been rolling in. I love reading them even though I am not a graduate. The messages are wonderful.

WordPress has been a great asset to my professional career. It has helped me provide many solutions to the needs of the school. Now that the entire network of over 30 sites is hosted on WP Engine, courtesy of the University, I don’t have to worry about the backend. I still have full control the themes, plugins, and full SFTP access. Some IT members at the Law School had criticized me for giving up hosting the server part of the sites, but there’s no way I can run the server as reliable as WP Engine. It would be a huge undertaking and I am not a server administration. If the University offers this huge service at no cost, why not taking advantage of it?

WordPress is great at solving problems that do not required original design. I could get pretty far with some changes to make the templated design suits my brand. Of course, I could create the entire WordPress theme from scratch, but that would required tremendous time and technical investments. For my own personal use, WordPress is far too complex. This blog, for instance, probably uses about five percent of WordPress’s powerful features. This blog has been powered by WordPress since 2003 and it hasn’t changed much over the years. I am still using the classic editor. I still code the theme using HTML and CSS and with only a minimum amount of PHP. I have control of every code I input. Developing a new theme, even from a starter theme, isn’t as simple anymore; therefore, I no longer offer WordPress for freelance clients. The amount of customization is just too much. Of course, I can still do it if I get big projects, but not for my typical clients.

I still love WordPress, but my development has changed. I am now happier to use WordPress as a tool to solve technological solutions instead of trying to offer WordPress as design solutions for client projects. It is a change in perspective that I have to come to term with if I continue to use WordPress. There are other choices out there, but WordPress remains a tough contender in the web space.

My choice is either WordPress or hand code HTML and CSS with some PHP to keep the pages manageable. I am missing the entire trend of static site generators. When web designers and developers moved their sites or blogs to Jekyll, Hugo, or Eleventy, I still keep my blog on WordPress. When they spent countless of hours moving from one static site generator to another or back to WordPress, I am still on WordPress.

A New Site for a Novelist

I am excited to be working on a new website for Kristin Bair whose new novel is coming out in November. Last week, I sent her a test site to review and she tweeted:

My new website for my new novel (out in November!) is being designed by the ever-brilliant @visualgui. He sent me 1st test with questions. I want to look but I’m afraid to look. I am so weird. Not the only weirdo out there, right?

In 2009, Kristin approached me in several months leading up to the release of her debut novel, Thirsty. I don’t know for sure how she found me, but I suspect it had to do with the slideshow “Bonjour Vietnam.” Before I even designed the site for Thirsty, I asked Kristin if I could read the manuscript and she sent it to me. It was a dark yet beautiful and hopeful novel. After reading the entire book via PDF, I knew the site for the book had to take on a life of its own. Even though the site is aged now—it was before responsive design—I still love it.

In 2014, she followed up with The Art of Floating and I designed a simple, responsive landing page to promote it. I also read the second novel and loved her dreamlike writing and erotic storytelling.

For her third novel, Agatha Arch is Afraid of Everything, I am redesigning her website to promote the new book as well as to bring everything together. It will be a simple, elegant website for a wonderful writer. I pre-ordered the new novel and can’t wait to read it.

The first draft for the new site went well. Her feedback was positive. We’re on the right direction. It feels great that she trusts me to bring her visions to life. I love working on a small-scale website like this. It gives me a break from my regualr work. I can’t wait to share it when we launch it.

Get Naked

Joining the CSS Naked Day 2020, I am removing my CSS to reveal the bare-bones HTML of this site. Without the CSS makeups, the site looks plain, but still works well. It also makes me appreciate the power of CSS. I didn’t realize that my SVG logo has no width or height in HTML. As a result, the default logo takes up the entire browser width. I fixed it. Other than the logo, the blog is all text. I hope you enjoy it. This is as naked as I get online.

Visualgui Designs

Visualgui is my personal blog started in 2003. It is a space for me not only to practice my writing but also to hone my web design skills. In addition to blogging vigorously almost everyday, I redesign it several times a year. The process is more iterative than comprehensive. The most drastic changes would be the typography. Whenever I discovered a new text face, I would love to set it here first. I focus mainly on readability.

This blog has been powered by WordPress from the get-go. The theme I had custom remained the same with just three files: index.php, style.css, and screenshot.png. That’s all. Even though WordPress has grown into a powerful content management system, I still use it as a blog system and nothing more. Keeping the theme simple allows me to quickly change the design with CSS. Most of the time, I can visualize the design in my head. I know what typefaces I want to use and how to set them on the page. I also can see how I layout my content. When I begin coding, I just modify my CSS file. The process is inspired by JAY-Z. He never wrote down his lyrics. He just went into the booth and recorded.

Because I make changes all the time, I have wanted to put together a page to showcase all the designs I have done over the years. Here are the screenshots dating back from 2004 for posterity.

Affordable Typefaces

After reading Professional Web Typography, a reader wrote:

Hi,

I contributed to your web book. Thank you for making the web a better place. I have been working on the web (semi-professionally) since around 2004. It’s amazing to me how much things have changed, yet how much they have remained the same. I’m intrigued with webfonts, but am frustrated that professional typefaces are largely cost prohibitive for someone like me that would seek to use them on a small personal website that would generate only dozens of pageviews in a very good month. I would support professional type designers if there were more affordable options.

In any case, thanks again for your web book and for your thoughts on Visualgui.com. I couldn’t stop until I went through all 19 pages in the design category.

Please keep sharing.

Regards,
MF

Here’s my response:

Dear MF,

Thank you for reading my web book. I feel your frustration on the cost for professional typefaces, but I also understand the amount of work goes into creating a professional typeface. For my personal use, I search for typefaces that I can afford.

For example, I am a member of the Font of the Month Club runs by David Jonathan Ross. For $6 a month, I have access to fun, experimental typefaces I can use for my personal projects. I also made a few investments in Future Fonts. I invested early; therefore, the cost was around $15 to $50. As the designers update their fonts, I get the latest releases for free.

If there’s a typeface I wanted but the price is high, I purchase only the fonts I needed. For example, I wanted to use Exchange, by Frere-Jones, for body text; therefore, I only bought two fonts (a regular and an italic) for $75. With just two styles, I know I cannot have bold and bold italic text, but I can turn them off easily with a few lines of CSS—that’s one of the beauties of the web.

If none of those above services work, you can always find a handful of decent typefaces on Google Fonts.

I wish you all the best with your typographic journey.

Thanks once again for contributing to my book.

Regards,

Donny Truong
donnytruong.com