Naguel

I'm Nahuel, and these are my work experiences, ideas and thoughts as a web developer working on the eCommerce industry.

Migrate a Magento project from any server to Cloudways

Migrate a Magento project from any server to Cloudways

While the title up there make this post looks like a promoted and/or paid one, that's not the case (I wish), and instead this is an honest appreciation of Cloudways and how it has made my life easy while dealing with servers, specially considering that I don't think myself as an advanced devOps.

Recently at the company I work for we had the task to move an existing Magento project, already in production and publicly accessible, from a server outside Cloudways to a server managed by Cloudways.

What's next is going to be long, and everything described could take around 3 days (one day for the server creation, another day for testing the new instance, and a third day for the actual site moving), depending on your skills.

So let's get to it.

Preparation

Make sure to tick a few boxes before moving forward by checking if you have the following:

  • Credentials to access the original server over SSH and SFTP, with enough permissions to manipulate files within the web root folder (usually public_html or www), use scp, tar, and cat for server configuration files such as the one for Nginx and Varnish.
  • Server permissions to use mysqldump to create a database backup, or a way to get your hands on a database backup of the project living on the original server.
  • Access or a point of contact for managing DNS changes for the project’s domain.
  • Access to the payment gateway accounts and/or credentials needed to configure the payment methods.

Finally, make sure to identify which third-party services currently connect directly to the server over SFTP or similar, because those will need to be reconfigured to point to the new IP and/or to be given new SFTP credentials.

Create the server in Cloudways

Decide on the server size and create it on Cloudways, making sure to factor in how much space you will need for the database, files, and "maneuver" as you will be moving zipped files from the original server to unzip them later.

Server creation example using a "PHP - Custom App" for a Magento Open Source 2.2

Cloudways offers Magento-ready instances for specific versions. If the project you are moving uses a Magento version already listed in Cloudways you can select this version from the dropdown that appears when creating the server, otherwise make sure to select “PHP - Custom App” instead.

This is important because, for example, if you are moving a Magento 2.2 version to Cloudways you can not select “Magento - Version 2.4.1 (with Elasticsearch)” from the dropdown as this instance will come with Elasticsearch 7 out of the box and you won’t be able to downgrade it to Elasticsearch 6 (which is the required version for Magento 2.2 using ElasticSuite).

If you need to go down the route of selecting “PHP - Custom App”, then you must contact Cloudways Support for them to "flag" this instance as Magento internally, after creating it.

This is because Cloudways offers specific configurations and performance-oriented settings for each type of instance, like an internal logic in place to load the specific Varnish settings a Magento project would require.

Cloudways Support flagging a "PHP - CustomApp" as Magento internally

Whatever the Application type you choose, you still need to contact Cloudways Support to ask them to:

  • Downgrade Composer from version 2 to 1 by running composer self-update --1 (you can't do it because you don't have enough server permissions to do so).
  • Install a few Elasticseach plugins by running bin/elasticsearch-plugin install analysis-phonetic; bin/elasticsearch-plugin install analysis-icu; bin/elasticsearch-plugin install analysis-smartcn; on the /usr/share/elasticsearch folder (another thing you won't be able to do due to permissions).

You still need to configure a few bits here and there using the Cloudways Dashboard to make sure you match your project needs, including but not limited to the following items:

  • PHP version, execution limit, memory limit, error reporting, and timezone.
  • MySQL version and timezone.
  • Nginx WAF module to be Cloudflare, Sucuri or disabled.
  • Elasticsearch version, if needed.
  • Redis, if needed.
  • Varnish, if needed.

I can't tell what you need to put in those fields as that depends on the project.

Configure the Application

Inside the "Domain Management" section, make sure to configure the project's domain and all the additional necessary domains (like the ones used for different stores if your Magento project is configured to use multiple websites and stores).

"Domain Management" configuration inside "Application Management"

Your "SSL Certificate" configuration will depend on what your project needs.

Since you are moving from one server to other, chances are you already have a SSL certificate that you can use, otherwise the Let's Encrypt option is quite good.

If you are using a firewall such as Cloudflare or Sucuri, the certificate might be "living there" already so there's nothing you need to configure here.

Do not add any cron in the "Cron Job Management" section for now.

We'll come back to the "Application Management" configuration later at the end.

Stop crons on the original server

By editing the crontab with crontab -e on the original server, or by contacting the original server hosting provider, disable all the crons by adding # at the beginning of each line.

An example of editing the crons with crontab -e and commenting out every line

This needs to happen before you move any file from the original server to Cloudways, and before the database backup creation.

Technically speaking, you want to avoid Magento to keep on writing any file in, for example, the var folder within the Magento root folder, related to Orders processing or Catalog management (like stock updates), and you want Magento to stop altering tables related also to those entities, which are the usual stuff affected by crons (among many other things).

If, for example, there are pending Order and/or Catalog tasks, they will be taking place in Cloudways after we finish the migration and we enable back the crons on the new server.

Move the files

Make sure to move the pub/media folder from within the Magento root folder, and any other custom folder and file that doesn't live in the repository.

Usually, there are some custom folders and files inside the pub and var folders that are not created automatically.

If you happen to be using Capistrano on the original server you would have a shared folder in the web root. In that case, just move that folder from the original server to Cloudways.

The way I do it is by creating a .tar.gz file with everything in it, but any cache folder...

[user_original]:~$ tar cvzfh shared.tar.gz shared/ --exclude='pub/media/catalog/product/cache';
tar cvzfh filename.tar.gz folder-to-zip/ --exclude='path/to/exclude';

...and then I transfer it from the original server to Cloudways using scp...

[user_original]:~$ scp shared.tar.gz user_cloudways@18.133.133.133:/mnt/data/home/master/applications/example/public_html
scp filename.tar.gz dest_username@123.dest.ip.123:/path/where/to/transfer

Make sure to push this file into the public_html folder in the Cloudways server, so you just need to unzip it there with tar -xvf shared.tar.gz; and then delete it.

Something I would like to do, just in case, after this big move, is to fix any potential permissions issue with the files and folders I just unzipped, by reseting those permissions using the Cloudways Dashboard.

Application Management setting to reset files/folder permissions

Enable Maintenance Mode

Put the site living on the original server into Maintenance Mode by running bin/magento maintenance:enable before proceeding.

For what it's worth, the downtime starts now.

Move the database

Using mysqldump create a database backup on the original server.

[user_original]:~$ mysqldump -uroot -p database_name > database_export.sql;

For security reasons, make sure this happens on a folder not accessible over the browser, like on the private_html folder or anywhere outside the web root.

Once done, push this .sql file to the Cloudways server using scp again.

[user_original]:~$ scp database_export.sql user_cloudways@18.133.133.133:/mnt/data/home/master/applications/example/private_html
scp filename.sql dest_username@123.dest.ip.123:/path/where/to/transfer

For the same security reasons, make sure to push this file to the private_html folder in the Cloudways server.

Once the transfer is completed, and before anything else, you need to remove the DEFINER from the .sql backup to avoid any "Access denied; you need (at least one of) the SUPER privilege(s) for this operation" issue while trying to import it.

Basically, run the following command on the private_html folder in Cloudways (or anywhere you pushed the .sql file before).

[user_cloudways]:~$ sed 's/\sDEFINER=`[^`]*`@`[^`]*`//g' -i database_export.sql;

Now the backup is ready to be imported into the actual database in the new server.

For this to happen you need to contact Cloudways Support with the request.

Cloudways Support handling the database import

After the import, make sure to delete the .sql file.

Configure Magento

By accessing directly to the recently imported database in Cloudways, and not through the Magento Admin Panel, make sure to have the correct details for Varnish and Elasticsearch, keeping in mind that Cloudways uses port 8081 for Varnish, and 127.0.0.1:9200 for Elasticsearch.

Make sure there's no reference in the core_config_data table to the IP for the old server that you might need to change for the IP of the new one.

Go back to the files, and edit the app/etc/env.php file you transferred from the original server to fix the credentials for connecting to the new database living in Cloudways now.

In that same file, correct the details for Redis (if used) keeping in mind that Cloudways uses 127.0.0.1 and 6379 for the host and port, respectively.

Once done, clear cache and reindex to make sure everything is working.

Configure the Application again

Go back to the "Application Management" in the Cloudways Dashboard and configure, in the new server, all the crons you commented out on the original server, making sure to use the right paths for the new server.

Inside the "Application Settings", configure the "WEBROOT" to point to the pub folder from within the Magento root folder.

Point the domain to the new server

First thing first, make sure Magento is still on Maintenance Mode on the new server, and whitelist your IP to be able to access the site (using the bin/magento maintenance:allow-ips command).

Now change the DNS settings to point the project's domain to the new server.

The whole idea is to open the site only for you in order to test it before allowing real customers. So, while the DNS change propagates, some customers will hit the site on the old server (that will show the maintenance page forever and ever), and others will hit the site on the new Cloudways server (that it will also show the maintenance page while you test it).

Now is when you test the site and make sure everything is okay.

Disable Maintenance Mode

Once you are done with the testing open the site, now in Cloudways, to the public, by disabling the Maintenance Mode (with the bin/magento maintenance:disable command).

The downtime ends now.

Prepare your self-evaluation before your performance review

Prepare your self-evaluation before your performance review

Don't we love having a good performance review, from our bosses to ourselves, every now and then? Of course we do! Is that moment were we get to be told how great of an employee we have been (finger cross that's the case) and to potentially get a salary raise (and I don't have to ask how much we love that).

Thing is that our performance review, appraisal, catch up, or whatever you would like to call, it is about having a conversation where you should also bring something to the table.

Why?

This is an opportunity to not only get feedback from your company, but also to help shape your role into it, to define a roadmap for your own career for the next cycle up to, let's say, your next performance review.

When you are not part of this conversation, when you are just a listener, you leave your future in the company up to whatever other decides for you. If you are "sort of lucky", the company you work for might already has defined a plan for you, and if you are "very lucky" you will like that plan. You would be betting, twice.

But, again, that's oddly the case when we don't have a say for our career.

How?

What I like to do for basically all meetings I have (and my performance review is no exception) is come prepare with notes and bullet points to discuss.

For this particular scenario I recommend doing a self-evaluation, ahead of the meeting, to know were we stand and were we would like to go from there.

Depending on our role in the company we can also cover where the company stands, where we would like it to go, and how we think we can get there.

Average is officially over, so don’t be that at work
There was a time when you would learn a skill and that would be enough to succeed, but nowadays what you learn has an expire date as much as the milk in your refrigerator.

I divide my self-evaluation, self-assessment, auto-evaluation, or, again, whatever you would like to call it, into different sections starting with "What I like to do and what I don't like to do".

This is a reflection time opportunity for me to know, up to this moment, after everything I went through from the last catch up, of all the things I've been doing so far what I actually like to do, and if I think I would like to continue doing them.

For example, let's say I've been doing a lot of documentation since the last time we had a catch up. Did I like that? Or was it more of an emergency role I had to take on that I would like to dodge from now on? Can I think of any available role I might like to fulfill with this skill that I think I'm good at and also like to perform?

The "what I don't like to do" part is to identify if the tasks I've been performing were those sort of a one-time-only tasks due to an emergency, or if they are in fact part of my job description.

If the stuff you don't like to do outnumber the things you do like, then you have a bigger problem here. But that's just another good idea for doing a self-evaluation: you can get yourself aware of how happy (or not) you are at your job.

The second section is called "Things I should put more attention at", which basically means "what I've been doing wrong" (but let's keep a positive tone here and not sound that radical). Now is when I need to be truly honest with myself in order to be able to identify where I have room to improve at work.

It's not only about telling where I could be better, but also to discover why I wasn't doing my best so far, and what I think could increase my performance on these areas.

I consider important to demonstrate that I'm able to tell when I'm not at my best before somebody points that out to me, and that I'm also capable of being part of "fixing" this situation with ideas and suggestions of my own.

Third section is about "Things we should keep an eye on", which involves the company itself and it's my chance to talk about how I see the place I work in.

You are, supposedly, at a different position than your superiors, therefore you might be able to provide another vision at how things are: what do you think the company is doing well, what things would be good to avoid losing, and what things do you think the company should put more effort in.

Clearly, this would be attached to your bosses' willingness to get some feedback on a meeting designed mainly to provide it to you. But, if you have the chance to give it a shot, just go for it.

Since my role is more technical I do have a fourth-ish section which is more like a twist for the last one, and I call it "Things we should keep an eye on... from a technical point of view" (the title speaks for itself).

This is my opportunity to say where I would like things to go, for the company, on the technical side, which is allegedly my area of expertise.

Following on the example of me doing a lot of documentation, I can bring ideas on how we can make this behaviour part of the company's culture.

Up there you can see a real self-evaluation I did on March 2019 before a performance review I had, and two more on the sidebar.

Keeping them works as a log to retrace my steps when I feel overwhelmed to assess if I'm on track regarding my latest self-evaluation of my performance or if I need to adjust something.

Basically, my idea is to avoid working blindly from one catch up to the other.

Magento and FORBIDDEN/12/index read-only / allow delete (api)

Magento and FORBIDDEN/12/index read-only / allow delete (api)

If you are unlucky enough to be reading this article it means that you are seeing that error when trying to reindex Magento.

ElasticSuite Category Indexing index process unknown error:
{"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}],"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"},"status":403}
ElasticSuite Thesaurus Indexing index process unknown error:
{"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}],"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"},"status":403}

This happens because you are probably running out of disk space and Elasticsearch turns itself into read only mode because, well, there's no more space to do stuff.

In order to get out of this situation, you need to first disable that Elasticsearch feature of switching to read only mode (otherwise it will happen again) by running the following...

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_cluster/settings -d '{"persistent":{"cluster.routing.allocation.disk.threshold_enabled":false}}'

...and then get Elasticsearch out of the read only mode with the next command:

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

Besides this, you'll need to do some disk clean up to recover some space.