Magento eCommerce – Fix your Google Search Console Coverage Server Errors(5xx)

This article covers:

  1. How to fix Google Search Console Coverage Server Error (5xx) for Magento stores
  2. How to create a large number of 301 redirects with Google Sheet

Audiences of this article:

  • Magento stores experiencing indexing/coverage issues, which is likely to be caused by changing product categories
  • This could also apply to other types of eCommerce platforms in need of creating a large number of 301 redirects

Why there are so many server errors when Google crawls your Magento store?

Magento has its built-in URL rewrites mechanism which automatically creates redirects for product URL updates. However, in some cases, a manual redirect is needed when below admin activities take place:

  • Some previously active store categories have been deleted
  • Admin -> Configuration -> Catalog -> Search Engine Optimisation _> Use Categories Path for Product URLs changed from Yes to No.

The above admin activity will result in previously indexing product URLs get 404 not found, which could be hundreds of thousands. If not taken care of, your store’s search performance will likely to take a toll.

The Solution – an effective way to create a large number of 301 redirects

Thousands of broken URL may sound horrible and tedious to fix, yet there are some pleasing factors about this issue, such as:

  • Google search console has allowed us to download a list of 404 URLs (with a maximum number of 1000 at a time)
  • In most cases, to fix a product 404 page caused by changing category URLs, all we need to do is redirect the broken URL to the product page without all the categories, see below example:
Redirect 301 https://mystore.com/category1/category2/product.phtml https://mystore.com/product.phtml

Sounds promising right? Let’s tackle this issue step by step.

Step 1 Download a list of 404 URLs

Go to your Google Search Console: Coverage -> Server error (5xx) and click on the Export data button to download as Google Sheet, if you have more than 1000 affected pages, after completing your 301 redirects and submitting for reviews, repeat these steps.

Two Google Sheet Functions for 301 redirects

  1. REGEXEXTRACT to extract the part of URL after last forward slash
  2. CONCATENATE to join “Redirect 301 “, “reditrct_url” and “destination_url” together, which can be pasted into your .htaccess file to complete the URL redirection.

Step 2 Extract last part of URL after last forward slash

The actual formula we used is:

 =REGEXEXTRACT(A1,"(?:.*/)(.*)$") 

Here is an example:

Above will help us get a list of product name URL key which is usually configured as the Canonical product URL. In other words, they are most likely working but we need to test the results before install 301 redirects on the server. Remember: always test your destination URLs!

Step 3 Test your destination URLs

When dealing with thousands of 301 redirects, we have to make sure each destination URL is working. We achieve this by running a custom script against the destination URL column. Open your Google Sheet -> Data -> Script Editor and add the function below:

function getStatusCode(url_key){
     // add your store domain name for testing
     var product_url = "https://eharvest.com.au/" + url_key
     var response = UrlFetchApp.fetch(product_url);
     return response.getResponseCode();
}

The above function will return 200 is the destination URL is working, otherwise it will return [#ERROR!], which we’ll have to take care of them manually. It could be products which are no longer available, so redirect them to similar products. See examples below:

Step 4 Form the final 301 Redirect Code by using CONCATENATE function

The actual formula we used here is:

 =CONCATENATE("Redirect 301 ", A1, " ", "/", B1) 

See example below:

If you’re familiar with Javascript, you can also write your own Google Script the achieve the same results as CONCATENATE function.

Now copy and paste the column C to the top of your .htaccess file to install 301 redirections, and don’t forget to test it. Once done you can go back to your Google Search Console and start the validation process.

If you have more than a thousand broken URLs, you will need to monitor your Search Console regularly to check your validation process. Once new errors appear in your example listing, download them and start over until all the errors are fixed.

To many, a business is a lifetime commitment. It's easy to start one yet difficult to make it successful. Attitude, skills, experiences and dedication help hone the craft along the way, but it's often the great vision and resilience to remain focused wins the game. Read more about me here

More from Our Blog

See all posts