Naguel

I'm Nahuel, and these are my work experiences, ideas and thoughts as a web developer working on the eCommerce industry.

Magento and a deadlock found when trying to get lock

Magento and a deadlock found when trying to get lock

"What now?" issues every now and then with Magento are the norm, and I got this one a few days ago:

SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction, query was: INSERT INTO `catalog_product_index_eav_temp` SELECT DISTINCT [...]

Apparently the server went down for a second, or MySQL went down... or something happened and a reindex process was interrupted (in my case the one for the "Product EAV" index), and there was no way to make it work again.

Even the usual bin/magento indexer:reset and then a manual reindex with bin/magento indexer:reindex didn't do the trick and the only difference was that instead of the previous error I was getting the classic one:

Product EAV index is locked by another reindex process. Skipping.

Lucky me, I find this tweet by @willemwigman on Twitter that basically indicates that, when you have no way to "unlock" an index and after you already tried the reset and manual reindex, spot the lock node within your app/etc/env.php file and change the prefix within the inner config node.

<?php
return [
    [...]
    'lock' => [
        'provider' => 'db',
        'config' => [
            'prefix' => 'new-prefix-here'
        ]
    ]
];

Reset the indexes and try to reindex again.

That alone should do the trick.

Create and apply a patch in Magento

Create and apply a patch in Magento

Magento usually has bugs (thank you, Captain Obvious) after the release of a version, that are later corrected on subsequent releases (known issues).

Using patches to change core files in Magento is not a bad practice when the goal behind it is to bring core fixes existing on newer versions to the one we have running, avoiding so a full upgrade.

How to apply a patch

I'm starting the other way around as applying a patch might be more common than actually creating one, specially because Magento itself releases patches for its own platform, third-party vendors release patches for their extensions, and the community also makes patches for others to use.

There's a well known Composer package named cweagans/composer-patches that handles the patches and applies them to the original Magento modules every time you run composer install and/or composer update.

Technically speaking, this tool installs all Composer packages required, then removes the Magento modules affected by the patches we are including, and re-install those modules with the code changes applied.

➜  ~ composer install
Gathering patches from patch file.
Removing package magento/module-customer so that it can be re-installed and re-patched.
  - Removing magento/module-customer (102.0.5)
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Package operations: 1 install, 0 updates, 0 removals
Gathering patches from patch file.
Gathering patches for dependencies. This might take a minute.
  - Installing magento/module-customer (102.0.5): Loading from cache
  - Applying patches for magento/module-customer
    patches/composer/magento/module-customer/22952.patch (#22952 - Delegated account creation fails with custom attributes present in customer address)

For making this happen, first you need to add your .patch file in the patches/composer/magento/[module-name]/ folder (relative to the Magento root, at the same level your composer.json file lives).

The name of the .patch file doesn't really matter, but I personally like to use the number of the GitHub issue where the known issue was discussed, if that even exists. Otherwise, any descriptive name would do.

For the [module-name] folder name use the Composer package name containing the files you are patching.

As an example, the Composer package name for the Magento_Customer module is module-customer. This information is on the composer.json file of each package, inside the vendor folder.

Finally, on the root composer.json file, inside the extra node create the patches node and define the patch to apply as the following example.

"extra": {
    "composer-exit-on-patch-failure": true,
    "patches": {
        "magento/module-customer": {
            "#22952 - Delegated account creation fails with custom attributes present in customer address": "patches/composer/magento/module-customer/22952.patch"
        }
    }
}

See that I'm adding a description to what the patch does, and the GitHub issue number, followed by the relative path to the .patch file itself.

How to create a patch

Identify the core file you would like to change using a patch.

For the sake of an example, I'm going to patch the vendor/magento/module-customer/Model/Address.php file to apply a fix for the "Delegated account creation fails with custom attributes present in customer address" issue reported in the magento/magento2 repo itself.

Create an exact copy of the file, and make all the necessary changes.

If you are using PhpStorm, you can create a "scratch file" to hold any code temporary, like a draft, by pressing CMD + Shift + N.

Comparison between the original file and the one with the changes

Output the changes into a .patch file using the diff command on the Terminal while standing in the root of your Magento.

➜  ~ diff -Naur path/to/old/file.php path/to/new/file.php > example.patch

Following my example, I need to execute the following...

~ diff -Naur vendor/magento/module-customer/Model/Address.php /Users/nahuelsanchez/Library/Application\ Support/JetBrains/PhpStorm2021.1/scratches/scratch_113.php > example.patch

...in order to get my example.patch file.

My example.patch file generated before any manual intervention

Unfortunately, here's when we need to perform some manual intervention on the generated example.patch file.

First, remove the date and time appearing the end of the first two lines.

Second, you can see that the first path is the real path to the original file living in the vendor folder. Add a a/ prefix to it.

Third, replace the second path with the first path, but using the b/ prefix instead.

Fourth, and finally, add a new line on top of everything with diff --git a/vendor/core/file.php b/vendor/core/file.php.

Before and after the manual intervention on my example.patch file

Do not touch anything else there, not even the empty lines at the end (if any).

The resulting file ready to be used as a patch should look as the following example:

diff --git a/vendor/magento/module-customer/Model/Address.php b/vendor/magento/module-customer/Model/Address.php
--- a/vendor/magento/module-customer/Model/Address.php
+++ b/vendor/magento/module-customer/Model/Address.php
@@ -159,7 +159,9 @@
         $customAttributes = $address->getCustomAttributes();
         if ($customAttributes !== null) {
             foreach ($customAttributes as $attribute) {
-                $this->setData($attribute->getAttributeCode(), $attribute->getValue());
+                if (isset($attribute)) {
+                    $this->setData($attribute->getAttributeCode(), $attribute->getValue());
+                }
             }
         }

Install Elasticsearch for Magento 2 on macOS

Install Elasticsearch for Magento 2 on macOS

Depending on the Magento version you have, or the Elasticsearch version you want, the process of getting this work locally can be easy or a nightmare I'll try to simplify here in this post.

If you already know what version to install, skip the following and jump straight to the installation instructions.

What Elasticsearch version do you need?

If we dive into the official Magento documentation and take a quick look at the different changes made to it across time, we can learn a few things.

Magento 2.3.4 documentation

Available at the magento/devdocs' GitHub repo, tag 2.3.4.

Magento 2.3.1 adds support for Elasticsearch 6.x and it is enabled by default. Magento still provides connectivity for Elasticsearch 2.x and 5.x but these must be enabled in order to use these versions.

Magento 2.3.5 documentation

Available at the magento/devdocs' GitHub repo, tag 2.3.5.

Magento 2.3.5 adds support for Elasticsearch 7.x.x (default) and 6.8.x. Both ES 2.x and 5.x are End of Life and are no longer supported in Magento.

Magento 2.4 documentation

Available at the magento/devdocs' GitHub repo, branch 2.4.1-develop.

You must install and configure Elasticsearch 7.6.x before upgrading to Magento 2.4.0.
Magento does not support Elasticsearch 2.x, 5.x, and 6.x.

Or, to summarize:

  • For Magento 2.3.1 to Magento 2.3.4 you need to install Elasticsearch 6.x.
  • For Magento 2.3.5 to Magento 2.3.n (anything before 2.4.0) you'll need to install Elasticsearch 7.x.x. It works also with 6.8.x, but version 7.x.x is easy to install.
  • For Magento 2.4.0 and up you need to install Elasticsearch 7.6.x. No support for previous versions.

If you are working with multiple Magento projects on different versions, I would suggest you to install Elasticsearch 6.x to support from Magento 2.3.1 to Magento 2.3.5, and everything before Magento 2.4.

If you have all projects on Magento 2.3.5 and up then install Elasticsearch 7.6.x.

Install Elasticsearch 6.x on macOS

You need Homebrew first, so please install it by doing the following:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

You can check the official Homebrew page for more information about it and alternative installations options.

Then, we need to add a third-party repo to Homebrew and a few packages, including Elasticsearch itself.

brew tap elastic/tap; brew cask install homebrew/cask-versions/adoptopenjdk8; brew install elasticsearch@6.8;

When you do that you'll get a lot of output on your terminal, but you need to only care for the installation path of the last elasticsearch@6.8 package.

➜  brew install elasticsearch@6.8;
==> Downloading https://homebrew.bintray.com/bottles/elasticsearch%406.8-6.8.8.catalina.bottle.tar.gz
Already downloaded: /Users/nahuelsanchez/Library/Caches/Homebrew/downloads/01d0782011cbecdb3b1125469ee2ed60ef07926a8f36752c64d1b8c2b763736d--elasticsearch@6.8-6.8.8.catalina.bottle.tar.gz
==> Pouring elasticsearch@6.8-6.8.8.catalina.bottle.tar.gz
==> /usr/local/Cellar/elasticsearch@6.8/6.8.8/bin/elasticsearch-keystore create
==> Caveats
Data: /usr/local/var/lib/elasticsearch/
Logs: /usr/local/var/log/elasticsearch/elasticsearch_nahuelsanchez.log
Plugins: /usr/local/var/elasticsearch/plugins/
Config: /usr/local/etc/elasticsearch/

elasticsearch@6.8 is keg-only, which means it was not symlinked into /usr/local,
because this is an alternate version of another formula.

If you need to have elasticsearch@6.8 first in your PATH run:
  echo 'export PATH="/usr/local/opt/elasticsearch@6.8/bin:$PATH"' >> ~/.zshrc
  
To have launchd start elasticsearch@6.8 now and restart at login:
  brew services start elasticsearch@6.8
Or, if you don't want/need a background service you can just run:
  elasticsearch
==> Summary
🍺  /usr/local/Cellar/elasticsearch@6.8/6.8.8: 136 files, 103.4MB

The path below "Summary" is the one we need to mind, where mine is /usr/local/Cellar/elasticsearch@6.8/6.8.8.

Go to that folder and install a few Elasticsearch plugins.

bin/elasticsearch-plugin install analysis-phonetic bin/elasticsearch-plugin install analysis-icu; bin/elasticsearch-plugin install analysis-smartcn;

Now, because only God knows why, we need to fix something on this freshly installed Elasticsearch package, otherwise it won't start.

Standing again into that installation path, cd into the libexec/config folders, and edit the jvm.options file.

Comment the line it says 8:-Xloggc:logs/gc.log by adding a # at the beginning of it, and below add 8:-Xloggc:/tmp/logs_gc.log.

This is how the file might ended up looking

You are basically done now.

Start Elasticsearch by doing brew services start elasticsearch@6.8.

➜  brew services start elasticsearch@6.8
==> Successfully started `elasticsearch@6.8` (label: homebrew.mxcl.elasticsearch@6.8)

To check if everything is good, you can do a cURL to the Elasticsearch instance running.

➜  curl "http://localhost:9200/_nodes/settings?pretty=true"
{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "elasticsearch_brew",
  "nodes" : {
    "ArBWW_c3QRunhfQn460sxQ" : {
      "name" : "ArBWW_c",
      "transport_address" : "127.0.0.1:9300",
      "host" : "127.0.0.1",
      "ip" : "127.0.0.1",
      "version" : "6.8.8",
      "build_flavor" : "oss",
      "build_type" : "tar",
      "build_hash" : "2f4c224",
      "roles" : [
        "master",
        "data",
        "ingest"
      ],
      "settings" : {
        "client" : {
          "type" : "node"
        },
        "cluster" : {
          "name" : "elasticsearch_brew"
        },
        "http" : {
          "type" : {
            "default" : "netty4"
          }
        },
        "node" : {
          "name" : "ArBWW_c"
        },
        "path" : {
          "data" : [
            "/usr/local/var/lib/elasticsearch"
          ],
          "logs" : "/usr/local/var/log/elasticsearch",
          "home" : "/usr/local/Cellar/elasticsearch@6.8/6.8.8/libexec"
        },
        "transport" : {
          "type" : {
            "default" : "netty4"
          }
        }
      }
    }
  }
}

That host and port on that cURL command (which by default is localhost:9200) is the host and port you need to use for configuring Elasticsearch in Magento. If it doesn't work, try the same cURL call but with 127.0.0.1:9200.

Remember to brew services stop elasticsearch@6.8 when you are done working so it's not running forever in your computer.

Install Elasticsearch 7.x on macOS

You need Homebrew first, so please install it by doing the following:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

You can check the official Homebrew page for more information about it and alternative installations options.

Then, we need to add a third-party repo to Homebrew and the Elasticsearch package itself.

brew tap elastic/tap; brew install elastic/tap/elasticsearch-full;

When you do that you'll get a lot of output on your terminal, but you need to only care for the installation path of the elastic/tap/elasticsearch-full package, which is located just at the end below "Summary".

For example, mine says /usr/local/Cellar/elasticsearch-full/7.8.0.

Go to that folder and install a few Elasticsearch plugins.

bin/elasticsearch-plugin install analysis-phonetic bin/elasticsearch-plugin install analysis-icu; bin/elasticsearch-plugin install analysis-smartcn;

That's all, basically.

Start Elasticsearch by doing brew services start elastic/tap/elasticsearch-full.

To check if everything is good, you can do a cURL to the Elasticsearch instance by running curl "http://localhost:9200/_nodes/settings?pretty=true" which will output a big JSON.

That host and port on that cURL command (which by default is localhost:9200) is the host and port you need to use for configuring Elasticsearch in Magento. If it doesn't work, try the same cURL call but with 127.0.0.1:9200.

Remember to brew services stop elastic/tap/elasticsearch-full when you are done working so it's not running forever in your computer.

Can I have both installed at the same time?

No... well, I didn't try it but I think you can't. I would avoid such experiment.

What you can do for sure is uninstall one and install the other on demand.

If you do that you need to uninstall the Elasticsearch plugins you install it and install them again so you ended up with the right plugins version compatible with the Elasticsearch version currently installed.

Anything else?

Reindex Magento before trying to load the store.