How to Avoid a Disastrous Website Migration: A Comprehensive Guide for Marketing Directors

Use Screaming Frog or Google Search Console to generate a full list of your website’s URLs.

The first thing you should do is crawl your old (live) website. Use a tool like Screaming Frog SEO Spider or Google Search Console to generate a list of all the URLs on your current website. The crawl will also reveal any 404 errors or 301 redirects that currently exist on your live site. It’s not a bad idea to clean these up before compiling your full list of URLs: you want to avoid creating too many redirect chains on your new website, which could eat up your crawl budget or overload the server with too many requests.

Don’t Forget to Check for Orphaned Pages

It’s important to keep in mind that a crawl might not reveal every single page on your website. A site crawler like Screaming Frog will only show pages that are linked internally on your website, which means that if you have any orphan pages that are not linked to internally from any other page, the crawler will not pick those up. These could include important landing pages used for ad campaigns that users cannot normally navigate to from your website’s linking structure.

You can use your CMS or your database to find these orphan pages, or you can find them through your Google Analytics data or Google Search Console. If you do have orphan pages, make sure to include these URLs in your list so that we can map them to your new URLs.

Create a mapping document that associates old URLs with new ones. This is essential for implementing redirects correctly.

The site structure and URLs on your new website have probably changed. Since your old website is already indexed in the search engines, you need to ensure those pages are pointing to the new version of the page (also called a redirect).

Since there are likely multiple external sources linking to your website, you don’t want visitors following those links to be met with a 404 page after the migration. To avoid this, you need to map all of your old URLs to the new URLs on the new website.

This can easily be done in a spreadsheet with a column for the old URL and a column for the new URL. Once you’ve mapped out the URLs, your developer can set up an .htaccess file containing all of the redirects.

Using your URL mapping document from the previous step, create a list of redirect rules to be included in your new site’s .htaccess file.

An .htaccess file is a configuration file used on Apache web servers that lets you configure certain rules and settings at the directory level, before a page is loaded. It’s in this file that we will insert rules to redirect your old URLs to your new URLs. 

Here is the standard 301 redirect structure for an .htaccess file: 

Redirect 301 /the-original-url https://example.com/the-new-url

It’s important to note here that the source URL (the URL from the old site) does not contain the top-level domain name, only the “slug”, or everything after the first slash. However, the target URL (the new URL) must contain the full URL, including the protocol (https) and the domain name. 

You can also use regex expressions to bulk redirect certain URLs and reduce load on your server. For example, if you need to bulk redirect an entire subdirectory from your old site to a single page on your new site, you could do the following:  

RedirectMatch 301 ^/a-sub-folder/(.*)$ https://www.example.com/the-new-url

This would redirect every single child URL under /a-sub-folder/ to the new URL. Here’s a useful resource for common .htaccess redirects that can help you with setting up your .htaccess file.

Note: Redirects using .htaccess files only apply to servers running Apache. For servers running Nginx (pronounced “Engine X”) or other web software like Squarespace, you will need a different approach, since it will not support the .htaccess file. For more information, contact your hosting provider on how you can implement redirects on your server. 

Update canonical tags to reflect the new site’s URLs, ensuring that search engines understand which page is the preferred version.

If you’re changing domains, you must ensure that the canonical links on the new site are referencing the new domain, and not the old one. Failure to do so could cause several issues and could potentially prevent your new site from being indexed. If you’re using WordPress as your CMS, this normally gets taken care of automatically based on the base URL you have set in the wp-config.php file, but it’s best to double-check and make sure the canonicalization is correct to avoid any problems. 

Here’s what a canonical tag looks like for our home page:  

<link rel=”canonical” href=”https://www.ontarioseo.ca/” />

Also, if your site is accessible by both the www and the non-www version, you should make sure there is a canonical set on both versions pointing to the same version, either www or non-www. Otherwise, these will be seen as duplicate pages, since they are essentially two separate URLs showing the exact same content. However, it is best practice to simply redirect one version to the other, to avoid this issue altogether. 

Replace the internal links on your new website with the new links to eliminate 301 redirects.

If you’ve copied content over from your old site into your new site, whether manually or using an automatic importer, chances are that your content contains hardcoded internal links that are referencing the old URLs. Since we’ve already implemented 301 redirects in previous steps, these internal links should be redirecting to the new URLs, but it’s good practice to update your internal linking to reference the new URLs to reduce load on the server. 

Ensure your new site is crawl-able and that there are no broken links or redirection errors.

In the first step, we crawled the old website to fetch a list of URLs. Now, it’s time to crawl the new website and double-check that there are no issues, 404 errors, broken images, or any remaining internal 301 redirects. If there are, these should be fixed before launching the site. This ensures a positive user experience, helps with SEO, improves indexing efficiency, and contributes to accurate analytics and reporting.

Create a custom 404 error page with helpful links and information to guide users who land on non-existent pages.

Creating a custom 404 page is often overlooked by developers but is an important step to ensure you don’t lose any potential site visitors that land on a broken page, or an old URL that was missed in the redirects from the previous steps. 

A good 404 page will let the user know that the page they are looking for is no longer available, and then guide the user towards relevant content like a blog article or a main service page. It can also prompt the user to perform a search on the website for specific content they might be looking for. 

Your server must also be configured to recognize this page as an actual 404 page and return the 404-error code in the browser. This can be achieved by defining your custom 404 page in the .htaccess file, like this:  

ErrorDocument 404 /this-is-my-404-page.php

Doing so will redirect any traffic to a non-existent page to the custom 404 page defined on this line and will return the “404 error” response code, letting the search engines know that this page should not be indexed. Failing to properly define your 404 page could result in Search Console issues. 

A content management system like WordPress or Drupal will usually handle the 404 page automatically, but if you’re building a custom website with PHP or React, it is imperative that this step does not get missed.

This is crucial for tracking user behaviour and maintaining data continuity during the migration.

A simple and easy task to forget, launching your new site without your Google Analytics or Google Tag Manager scripts could cause loss of visitor data, especially during those crucial first days after the migration. Don’t forget to include these scripts on your site!

Ensure you have a complete backup of your old site in case anything goes wrong during the migration.

This should go without saying, but please make a copy of your old site’s files and database before launching or migrating. You could also keep a password-protected copy of this site on a staging or development server to be accessed at any time. This will allow you to reference the old website and compare site structure and data, if needed. If your new site is a complete rebuild with brand new content and a reorganized site structure, being able to refer to the old website can be especially helpful since you’ll be able to easily retrieve content, data, or code snippets as needed. 

Keep in mind that if you do choose to keep a copy of your website behind a password-protected page, this will use up additional resources on the server and may incur additional costs. There are also some potential security concerns if your old website contains sensitive information, or old web technology that may become vulnerable. In this case, additional security measures could be put in place to properly secure the website.  

Test the redirects you implemented to ensure they are functioning correctly.

Now that your new website is launched, you should test all of your old URLs and make sure they are redirecting. If your old website only had several pages, you could go ahead and test these manually, but if you have hundreds of URLs, it’s best to do this with a tool like Screaming Frog. 

In Screaming Frog, you can run a crawl from a predefined list of URLs. Simply change the mode from “Spider” to “List”, then click the Upload button and select your list of URLs from a file or enter them manually. Using the URL mapping spreadsheet you created before launch, you can grab the list of old URLs and just replace the domain name with your new domain (If your domain name hasn’t changed, you can leave it as is). Then, run this list of URLs through Screaming Frog to crawl them and check whether they are redirecting properly. If all is well, the crawl should return a list of your URLs with the “200” response code (if the URL on your new site hasn’t changed), or “301” (if the URL is redirecting to the new one). If you see any 404 errors, that means some redirects were missed. Find all the 404s, and create new redirect rules in your .htaccess file to redirect these pages to the appropriate page.

Ensure your robots.txt file is properly configured to allow search engines to crawl the new site.

The robots.txt file is a text file that lives at the root directory of your website. Its purpose is to indicate to crawl bots which files and directories they are allowed to crawl.   

When your website is in the development and staging phases, developers will sometimes add a line in the robots.txt file instructing the search engine robots not to crawl the website, to ensure the search engines don’t crawl a website in development. Here’s what it looks like:

User-agent: * 
Disallow: /

Here, “User-agent” is used to define the specific robots we are targeting for the disallow rule below. In this case, the “*” is a wildcard, meaning we are targeting all robots. If you wanted to target only the Google bot, you would enter “User-agent: Googlebot”. The “Disallow” line below it instructs the robots not to crawl anything starting from “/”, which is the root folder of the website.  

When the staging site gets deployed to the live production server ahead of the launch, depending on the development workflow, there is a chance that this robots.txt file will get carried over to the live site with these instructions still in place. That would mean that your new, live site would be blocked from being accessed by all search engine robots. To fix this, open the robots.txt file that sits in the root directory of your website, and make sure to remove the slash in the disallow line, like this:   

User-agent: * 
Disallow: 

This tells the robots that nothing is disallowed, and that they are “allowed” to freely crawl the entire website. You can test the indexability of your website by running it through a crawling tool like Screaming Frog. These tools will usually indicate if a website cannot be crawled due to a disallow rule in the robots.txt file. 

If you’re running pay-per-click (PPC) campaigns, update the ad URLs to reflect the new site structure.

After you’ve launched your new website, you should update your PPC campaigns so that they point to the new site. Even though we’ve implemented redirects (which will redirect users who click on the ad to the correct page), if your PPC campaigns are pointing to the old site, attribution will be lost in Analytics because of the redirect.

Check for duplicate content issues that might arise after the migration. Resolve them to avoid SEO penalties.

If the “Ensure Your Canonical Links Are Pointing To the New Site” step was done correctly in the pre-launch checklist, there should be no duplicate content issues on your website. If there are any duplicate content issues, it’s likely due to repeating content on separate pages that could be consolidated at a later time. For now, let’s just make sure that there are no canonical issues. The www version of your site (https://www.example.com) should be redirecting to the non-www version of your site (https://example.com), or vice versa. If both these versions are accessible without a redirect, and the canonicals aren’t correctly set, Google will flag your site as having duplicate content, making this step an important one not to miss.

Additionally, if you’ve purchased any other domains that you’d like to point to your new site, you must correctly implement redirects for these domains to point to your main domain. For instance, if your main domain is example.com, but you’ve also purchased example.net and example.org, you should redirect both example.net and example.org to example.com. In some cases, some servers will simply point your add-on domains like example.net and example.org to the website folder, which would make your entire site completely accessible via these add-on domains. The page example.org/about would be visible, instead of redirecting to example.com/about. Google would treat these as 2 separate websites and flag it as duplicate content, penalizing your rankings and hurting your SEO. This is why it’s imperative that you handle your additional domains correctly and implement redirects to your main domain. 

Create and submit a new sitemap to search engines to help them discover and index your new site efficiently.

After you’ve checked everything and cleaned up any issues resulting from the site migration, like internal 301 redirects and 404 errors, you should submit the new sitemap to Search Console to signal to Google to crawl your entire website again, so it can index your new links and site structure.

If you’re using WordPress, we highly recommend using the Yoast SEO plugin. Yoast is free and comes with a built-in dynamic XML sitemap, which will generate and update automatically when you make changes to your pages or create new ones. To access your Yoast XML sitemap, go to www.example.com/sitemap_index.xml (replace example.com with your domain name). Yoast breaks up the sitemap into separate individual sitemaps for each post type on your website (pages, posts, etc.). This will differ based on the post types created in your theme. A sitemap will also be created for your post type categories and tags, as well as author pages. To remove these pages from the sitemap, go to Yoast Settings -> Search Appearance. The Content Types, Taxonomies, and Archives tabs will allow you to remove specific types from the sitemap. To do so, set “Show in search results” to off.