How I stopped blocking GoogleBot and passed the Mobile-Friendly Test

Screenshot of Janepedia failing Google's Mobile-Friendly Test

So I was reading a Food Blogger Pro Podcast transcript about SSL and HTTP vs. HTTPS, and a curiosity brewed in me to check janepedia.com against Google Search. I typed “janepedia” into the address bar, and lo and behold https:// prefixed it, but in red letters beneath the URL read, “Your page is not mobile-friendly.” I had to do some digging, but my robots.txt was blocking GoogleBot…and the issue was not as straightforward a fix as I’d have expected—but it also wasn’t as complex a fix as I’d expected, so that’s awesome. I briefly feared I was going to have to dial 4367443.

Failing the Mobile-Friendly Test

Georgie codes mobile-first, so I knew that could not be an issue at all. Thinking now, I suppose I’d not feel so confident in her code’s mobile-friendliness if she didn’t say, “I code mobile-first,” so many times. So…ha, I guess it’s good to brag about yourself sometimes! That troubleshooting step was easy to tick off.

After some time, I noticed the CSS of my site wasn’t loaded. Perturbed and assumptive, I submitted feedback about this, even detailing the fact that I use responsive images because Georgie rubbed off on me, so the photos rendered by the test should have at least been 300px wide. Because I’m a visual person, here is the image code of/for the image on my about page (which I recently rewrote):

<img src="https://janepedia.com/img/pages/about/me-newt-1000.jpg" srcset="https://janepedia.com/img/pages/about/me-newt-300.jpg 300w, https://janepedia.com/img/pages/about/me-newt-500.jpg 500w, https://janepedia.com/img/pages/about/me-newt-1000.jpg 1000w" alt="Photo of myself with Newt, a miniature jack donkey; in the background are miniature donkeys in another pen" width="1000">

This is also why I find editing and uploading images more tedious than shaving my legs legally blind (because glasses + the shower don’t mix) while I’m trying to avoid scars, blisters, and whatever else might invoke an audible whine (as opposed to my typical inaudible ones paired with mere facial expressions).

If the CSS had been rendered, obviously the width of the image would be max-width:100% regardless of whatever size was rendered, though. I mean, the CSS literally would solve all the issues. So I finally realized, Oh! Duh! and looked up—on Google Search, no less—how to even fix this. I’m not a fan of their community-based support system—there are a lot of snarky, entitled people who feel a need to parent people asking for help (or maybe I’m just different in that respect and try to fight fire with kindness and logic). It’s why I prefer smaller companies.

The partially-loaded page dilemma

From one topic, I discovered there was an option to view loading issues at the top of the scroll frame. Despite it having a yellow caution symbol, I overlooked it in favor of the big red text saying my site wasn’t mobile-friendly.

Screenshot of page loading issues for Janepedia in Mobile-Friendly Test

After, I deduced it was a GoogleBot + robots.txt issue, based on the detailed table.

Resolving the GoogleBot vs. robots.txt problem

I looked it up, ’cause last time I let bots index my files, those ended up in Google Search, and I was getting hacked left and right. I started using iThemes Security, formerly known as Better WP Security, but I stopped because I felt uncomfortable letting a plugin have that much control over my website—and after a major update, it changed every single image to not be visible to anyone. (I use Wordfence Security now, as suggested by Nancy.)

This is where it gets complicated, because I didn’t want to allow every theme (no one needs to know which themes I have on my site!) nor every plugin to be indexed. I think this will be another reason I dislike third-party plugins, because it means I have to keep up with the paths of them to allow on robots.txt.

Using another community forum topic response, I came up with this:

Allow: /wp-content/plugins/fd-footnotes/*.js
Allow: /wp-content/plugins/jetpack/css/*.css
Allow: /wp-content/plugins/recipe-card/css/*.css
Allow: /wp-content/plugins/recipe-card/js/*.js
Allow: /wp-content/themes/janepedia-mercury/*.min.css
Allow: /wp-content/themes/janepedia-mercury/style/*.min.css
Allow: /wp-includes/js/*.js
Allow: /wp-includes/js/jquery/*.js

To test whether this worked, I pressed the circle with an arrow at the far-right end of the URL field on the test to refresh.

Screenshot of Janepedia passing Google's Mobile-Friendly Test
It passed!

Remaining loading issues

  • I removed the social follow plugin and just placed my social media into my footer.
  • The remaining issue thereafter is created by Google Analytics (don’t even get me started on the support topic for that.)

I hope you find this useful if you find yourself having similar issues/dilemmas. Afterward, I pressed “Submit to Google” so I wouldn’t have to wait for unreliable bot timing. If you want to further optimize your robots.txt for search engine optimization (SEO), WPBeginner has a detailed post about that.

Leave a Comment

Comments on this post

Evelina @ AvalinahsBooks’s gravatar

Reading this makes me shudder with anxiety so much I don’t even want to check if mine’s mobile friendly 😀 it’s so cool you can work this out yourself! It’s such an incredible power. Meanwhile I’m still struggling with the smallest things. Like the stupid search console, or the fact that my jetpack and WordPress are still struggling to be friends xD

Reply to this »