How to recover a site’s old URLs before migrating it
Why the URL inventory is the step that saves the most traffic in a migration, which sources to build it from (Search Console, server logs, the Wayback Machine), and how to turn it into a redirect map.
By Javier Tenjo, Full stack developer
On this page
Nearly every migration that loses traffic loses it for the same reason: nobody made the list. The new site is designed, built and launched with care, but the old site’s addresses, the ones Google had indexed and other sites linked to, are left with nowhere to go. Every one of them starts returning a 404.
The list of old URLs is tedious to put together and invisible in the finished product. It is also the step that saves the most traffic.
Why inventory before migrating
Because afterwards you cannot. While the old site is online, its URLs can be crawled, exported and checked. As soon as the server is switched off or the CMS replaced, that information goes with it, and rebuilding it from the errors that trickle in is slow and incomplete.
Every old URL carries something of value:
- Rankings in Google, sometimes earned over years.
- Links from other sites, which you cannot edit and which will keep pointing at the old address.
- Direct visits: bookmarks, links in emails, printed documents, QR codes on a leaflet.
A well-placed 301 redirect preserves nearly all of that. To put it in place, you first have to know the URL exists.
Where the list comes from
No single source has everything. Each one sees part of the site, and a good list comes from cross-checking several:
| Source | What it provides | What it misses |
|---|---|---|
| The CMS or database | Everything published today | What was deleted long ago but still gets links |
| A crawl of the current site | What is linked internally | Orphan pages and old URLs already retired |
| The sitemap | What the site declares | Often out of date or incomplete |
| Search Console | URLs with impressions and clicks (up to 16 months), indexed pages | Whatever Google has not shown in that period |
| Analytics | Pages with real visits | What gets links but no traffic |
| Server logs | Every URL anyone or anything requested | Only as far back as they are kept |
| Backlink tools | URLs with links from other sites | Coverage depends on each tool |
| Wayback Machine | Years of the domain’s history | Gaps on small or poorly linked sites |
Search Console is the most valuable source for prioritising, because it tells you which URLs bring in traffic from Google right now. Its performance report keeps up to 16 months of data, and it is worth exporting before you touch anything.
Server logs are the most complete source for the present: they record every request, including Googlebot’s and those of visitors arriving through old links. If your hosting keeps them, ask for a copy covering the last few months.
What the Wayback Machine adds
Internet Archive’s Wayback Machine has been saving copies of web pages since the late nineties. As well as the captures you can browse, it keeps an index of every URL it has seen on each domain, with the date of each capture and the status code the server returned at the time.
For a migration, that makes it especially useful in two situations:
- Sites with a long history, where earlier redesigns and migrations left URLs that still receive links even though nobody remembers them.
- Sites whose old CMS no longer exists, or that nobody can access. Sometimes it is the only list available.
It has limitations worth knowing. It only stores what its crawlers found or what someone asked it to save, so a small site may have gaps spanning years. And it picks up a lot of noise: URLs with session parameters, pagination, filters, stray files. It needs filtering before use.
Turning the list into a redirect map
With the sources gathered, the work is sorting and deciding:
- Merge and clean. Combine all the lists, normalise the URLs (same scheme, same domain, same trailing slash) and remove duplicates. Drop parameters that do not change the content, such as campaign ones.
- Keep what matters. HTML pages, not every image or stray asset. If a URL was already returning 404 on the old site, there is no need to inherit it.
- Sort by value. At the top go URLs with Search Console traffic, inbound links or plenty of visits in the logs. Those are the ones that cannot fail.
- Give each one a destination. It is a three-column table: old URL, new URL and status code. The new URL should be the closest equivalent, not the home page.
- Decide what is retired. If a page has no equivalent, the honest answer is a 410 (gone for good) or a 404. Redirecting everything left over to the home page is what Google treats as a soft 404.
There are rules worth respecting when building the map: a single hop from each old URL to its final destination, even if there were earlier migrations, with no chains or loops. And redirects are kept for at least a year, ideally longer. Why each one matters is explained in 301 vs 302 redirects.
On large sites, many URLs follow patterns (/product/<name> becomes /shop/<name>/) and are handled with one rule per pattern rather than one line per URL. Whatever fits no pattern is resolved by hand.
The map on launch day
The map is not finished once it is written. On launch day and in the weeks that follow:
- Test a sample of the most valuable URLs and check that each one reaches its destination in a single hop, with a 301.
- Watch for 404 errors in Search Console and the server logs. URLs that show up and were not in the map are the ones that slipped through the inventory: add them.
- Review the internal links on the new site so they point straight at the new URLs, without going through redirects. This is covered in broken links and SEO.
WordPress migration without losing SEO covers the full migration process, of which this inventory is the first step.
Recovering yours
The old URLs finder queries Internet Archive’s index and returns every address it has seen on your domain, with its last status code and the date of its last capture. You can keep only the ones that returned 200, filter to HTML pages, group them by folder to see where the bulk of the site lived, and download them as a CSV or as a redirect map skeleton ready for you to fill in the destinations. It also lets you ask Internet Archive to archive the current pages before they change.
Use it as one source among several, alongside Search Console and the server logs. Once the map is in place, the redirect checker helps you verify that each old URL lands where it should.
A well-run migration barely shows: the site changes and the traffic stays. Behind that there is almost always a long, unglamorous spreadsheet listing every old URL and its destination. It is worth putting the time into it before switching anything off.