Naguel

I'm Nahuel, and these are my work experiences, ideas and thoughts as a web developer working on the eCommerce industry.

Change Googlebot crawl rate

Change Googlebot crawl rate

As weird as it sounds it could happen that Google hits your site at a pace your server won't be able to handle, causing page speed issues and a potential site down related to a high CPU usage and too many MySQL connections.

Google claims to be smart enough to be able to self-regulate the crawl rate at which it hits a site (they said that they have "sophisticated algorithms to determine the optimal crawl speed for a site"), but that's not true all the time.

If you can contact your hosting provider, they might be able to tell you the number of requests the bots are making (where around 20000 in the last six hours could be enough to cause issues) and the logs for those calls.

185.93.229.3 - - [23/Sep/2021:05:32:55 +0000] "GET /about/ HTTP/1.0" 200 20702 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Make sure that the IP you are seeing on the logs are really from Google not being smart about the crawl rate, and not a DDoS attack in disguise.

If you are certain about Googlebot being the one causing the problems with your server, you need to adjust the crawl rate first, and then block Google, temporarily, while you wait for the new crawl rate to take effect.

Adjust the crawl rate using Google Search Console

If you are thinking about controlling the crawl rate using the Crawl-delay directive on your robots.txt file, forget it. Google ignores that.

You need to use what's remaining of the old Google Search Console, not the new fancy one, here https://www.google.com/webmasters/tools/settings.

Selecting the “Limit Google's maximum crawl rate“ option will display a slider for you to configure a lower crawl rate.

How low? I can't tell you exactly, since that would depend on your site and server capacity, but as reference I remember configuring it as follow for a Magento project living on a size XL AWS instance.

1 requests per second
1 seconds between requests

You can try something like that, or even lower, maybe at 0.5 requests per second.

Block Googlebot temporary for a few days

So, here's the catch: the new crawl rate setting is not immediately applied.

After saving the new crawl rate you'll get a message saying that "Your changes were successfully saved and will remain in effect until Sep 17, 2021" but it's not until you open up the confirmation email that you'll read that "within a day or two, Google crawlers will change crawling to the maximum rate that you set".

This means that your site will keep on getting hit at the same pace for a day or two, so the solution is to block Google for that period of time while you wait for the new crawl rate to get into effect.

Remember to lift the ban on Google after 2 days, and monitor for the following days that Google isn’t affecting the site performance again (so confirming the new crawl rate is working, otherwise you might need to bring it down a little more).

Play spot the difference as part of your front end development process

Play spot the difference as part of your front end development process

I know that inside the front end development kingdom there are a lot of different sub species, different front end developers out there working in different stuff: a few more focused on the functionality closer to back end development, and some working on converting designs into code.

If you're a front end developer whose main task is to look at a PSD file and "make it happen" in code, then I can't stress hard enough how important is for the job that you achieve this successfully. And if you're thinking that this is obvious and there's no need for a post, trust me, it's still a common problem among developers... and it's hard to me to understand why.

What's the main task of a fire fighter? Extinguish a fire. What's the main task of a plane pilot? Fly the plane. Main task of a mother?... I have no idea, but my chips are on keeping the baby alive.

Jokes aside, what do you think is the main task of a a front end dev? Get a design, then implemented it as pixel perfect as it can be.

I saw this problem presented in a variety of seniorities, as it doesn't distinguish between Juniors and Seniors. Of course, the problematic is more present in lower seniorities, that's not surprise, but it still cause for rework in more experienced developers.

If "main task" seems like too much, as you probably thinking, just like I am, that a front end developers faces a lot of task during a day and it's hard to decide which one is the main one (again, related to the fact that your front end developer job description it's for sure different than the others), then just let's not get stuck in grammar.

It might not be your main task as a front end developer, but for sure is an obvious task, something that you should be already been achieving without anybody raising any doubt about it when you deliver.

So why aren't you? Or why you still have this problem in your team?

If it's your problem then awesome, today is a great day to start solving that. And if not your problem but something you see that happens within your team or company... then it's still your problem, sort of.

The most common excuses I heard about it is the lack of tools, or people using the wrong tools, to view and manipulate the designs.

"Oh, it looks different because I don't have Photoshop in my computer so I opened the document with other software that converts .psd files if you're using Linux."

That's not a good excuse and you know it.

We're missing a synchronization here between developers and people in higher position within the company when deciding on what tools to use. Are we going for Adobe Photoshop for the designs' team?, great, then let's make sure that we also have Adobe Photoshop in the developers' computers... and not Linux if we're going to deliver .psd files to them.

Here's another one, "We were on a hurry, so I couldn't spend time on making the implementation pixel perfect". I can only assume that the equivalent for that on a fire fighter would be something like "Oh, we were on a hurry, so we only extinguish the fire on the first floor".

That "explanation" is a symptom of a higher problem within the company that might deserves an individual post, but for the moment just stop the presses and evaluate if you're giving to the developer the necessary time to accomplish all the items included in yours definition of done. If not, then you're asking for the impossible.

On the other side, if you're the developer. Come on dev, stand your ground! Do not go for an impossible estimation knowing that you won't be able to deliver successfully. If the time frame won't allow you to go for a pixel perfect implementation, then speak up. If everybody is on board of what the outcome will be, then good, no surprises for nobody.

And finally, the Top 3 seems to be "I tested it on Chromium for Linux Mint and it looked exactly like the designs". Looks like a child of the first excuse, and still a bad one.

On the developer's side, ask for the specifics, ask what browsers you are supporting for that particular project, or ask for metrics from Google Analytics to know exactly what your final customers are using. And test, my friend, test it on more browsers (specially knowing that Chromium for Linux Mint can't have the final word).

If you're the company, or somebody with a technical seniority that have a say on technical decisions, then define what browser do you support and include them on your definition of done. Avoid having technical decision based on the personal believes of each developer on the team.

Spot the difference

Have you ever played that magazine or newspaper game where you have two almost identical images next to each other and the goal is to spot the 10 differences? Why don't you try the same game with your work before tagging it as done?

I've been on meetings where emotions are high, reviewing an implementation that clearly doesn't look like the designs, where the conversation last for less than a minute because all it takes from the person evaluating the job is to put the original design next to the developer implementation and ask "Does it look the same? Does it look like it is finished?".

And of course it doesn't. Just avoid that meeting by playing the game by yourself and prevents others from playing it for you with your work.

On a not much of a side note, I find the post "The Undefinition of "Done"" quite cool, so check it out as it might help with your development process.

A blank Ghost blog theme for development

A blank Ghost blog theme for development

This blogs runs on the Ghost platform built with Node.js, with a homemade theme that uses Handlebars, HTML, CSS and JavaScript, and if you are reading this article chances are that you're already a Ghost blog owner and that you're looking to develop your own theme.

The problem with the Ghost default theme named Casper, while it's extremely helpful to learn how to develop a custom Ghost theme, is that it is too complex or "it has too many things" you probably don't want for your blog.

For example, when creating the theme for this blog, I had to "deconstruct" Casper to start building on top of the ashes because I wanted a more simple theme, that has my own code wrote the way I wanted it.

Introducing the Pale Ghost blank theme

The Pale Ghost theme is a blank theme, developer ready, focused on the markup without styles, that already includes all the Ghost features for you to customize or remove in order to create your own theme.

I already searched for Ghost blank themes, and truth be told there are many out there, but none suits my desires. All the blank themes I encounter has too much JavaScript, or they are too much complex to the point they looses they purpose of being a blank theme.

This blank theme in particular is very simple, so you might be spending time in the styling part with CSS/SASS/LESS rather than working on the markup. And if you want to move stuff around, you'll find it's easy with the Pale Ghost theme as it is well deconstructed into different .hbs files for an easy development.

It also includes everything a Ghost theme can deliver, just as Casper, but with a simple implementation. The goal with this theme is to avoid the part where we as developers smack things down before actually building up.

For a more technical description of what this blank theme includes you can check the Pale Ghost theme's GitHub page.

Downloading...

You can download this blank Ghost theme from the Pale Ghost theme's GitHub page, where you'll find the "Releases" section.