Why Your Site Needs a Crawlability and Indexation Audit (And How to Do One)

You have probably spent time on page titles, meta descriptions, and maybe even link building. Those are important. But if Googlebot cannot crawl your pages or chooses not to index them, none of that effort matters. A proper crawlability and indexation audit is the foundation that every other SEO activity rests on. Without it, you are flying blind.

I have worked with enough sites over the years to know that the biggest traffic leaks are almost never about keywords. They are about technical barriers that stop search engines from even seeing the content. A site with beautiful writing and zero indexable pages is just a private blog. So let us talk about what a crawlability and indexation audit really involves, why it matters for a business like a Brisbane-based consultancy, and how you can run one yourself without getting lost in the data.

What We Actually Mean by Crawlability and Indexation

Crawlability is the ability of Googlebot (and other search engine bots) to reach your pages by following links. Indexation is what happens after crawling: Google decides whether to store that page in its index so it can appear in search results. The two are connected but not the same. A page can be crawlable but not indexed, or indexed but not crawlable (which is rare but possible with certain server configurations).

When I run a crawlability and indexation audit, I am checking both sides. First, can Googlebot actually walk the path from the homepage to every important page? Second, is Google choosing to index those pages once they are crawled? The answers often surprise site owners.

Common Problems That Show Up in These Audits

Here is a short list of issues I see regularly. They are not exotic. They are just easy to miss when you are focused on content or design.

Blocked resources in robots.txt: A single disallow line can hide your entire site from Googlebot. I have seen robots.txt files that accidentally block CSS and JavaScript files, which hurts rendering and can prevent indexation.
Broken links and redirect chains: Googlebot has a crawl budget. Every 404 or multi-hop redirect wastes that budget. Over time, important pages get crawled less often.
Noindex tags on key pages: Sometimes a developer adds a noindex tag during a site migration and forgets to remove it. The page stays live but invisible to search.
Duplicate content without canonical tags: When the same content lives at multiple URLs, Google may index the wrong one or decide not to index any of them.
Poor site architecture: If your deepest pages are five clicks from the homepage, they might never get crawled at all.

Each of these is easy to fix once you know it exists. The hard part is finding them. That is where the audit tools come in.

The Tools You Need

You do not need an expensive enterprise platform to do this work. Most of the heavy lifting can be done with a combination of free and low-cost tools. Here are the ones I rely on.

Google Search Console is the starting point. The index coverage report shows which pages Google has indexed, which it could not crawl, and why. It also surfaces manual actions and security issues. If you only use one tool, use this one.

Screaming Frog is my go-to SEO spider for a deep crawl. It mimics how Googlebot moves through a site. It finds broken links, redirect chains, missing meta tags, duplicate content, and hundreds of other issues. The free version crawls up to 500 URLs, which covers most small to medium sites. For larger sites, the paid version is worth every dollar.

Ahrefs has a Site Audit feature that runs from their servers. It shows crawlability problems, page load speed metrics, and structured data errors. It also gives you a health score that is useful for tracking progress over time.

Lighthouse audit (built into Chrome) and core web vitals data from Search Console help with the performance side. Page load speed is not strictly a crawlability issue, but slow pages get crawled less often. Googlebot has a crawl budget, and slow servers eat into it.

How I Run a Crawlability and Indexation Audit

The process has four phases. Each phase answers a specific question.

Phase 1: Check the Gatekeepers

Start with robots.txt and the XML sitemap. Open robots.txt in a browser and look for disallow directives that might block important sections. Then open the XML sitemap and confirm it contains only canonical URLs that you want indexed. Remove any URLs that return 3xx, 4xx, or 5xx status codes. Google Search Console lets you test a URL against your robots.txt to see if it is blocked. Use that.

Phase 2: Crawl the Site

Run Screaming Frog or another SEO spider from your own machine. Configure it to follow all links, respect robots.txt, and render JavaScript if your site relies on JS for navigation. The spider will produce a list of every URL it finds, along with status codes, response times, content types, and meta directives. Export that list and sort by status code. Fix every 4xx and 5xx. Then look at the redirect chains. Any chain longer than two hops should be shortened to a single 301 redirect.

Phase 3: Compare Crawl Data with Index Data

Open the index coverage report in Google Search Console. Look at the "Excluded" tab. You will see pages marked as "Crawled - currently not indexed" or "Discovered - currently not indexed." Those are pages Googlebot found but chose not to put in the index. Common reasons: thin content, duplicate content, noindex tags, or low perceived value. For each excluded page, decide whether it should be indexed. If yes, improve the content and remove any blocking directives. If no, consider adding a noindex tag or removing the page entirely.

crawlability and indexation audit

Cross-reference the excluded pages with your crawl data. Sometimes a page is excluded because Googlebot found a canonical tag pointing elsewhere, or because the page load speed was too slow for mobile-first indexing. The audit reveals the root cause.

Phase 4: Fix the Architecture

Site architecture is often the silent killer. If your most valuable content lives deep in a folder structure with no internal links, Googlebot might never find it. Map out your site’s link graph. Make sure every important page is reachable within three clicks from the homepage. Use descriptive anchor text. Add internal links where they are missing.

Also check your use of canonical tags. Every page should either be self-referencing canonical or point to the canonical version. Mismatched canonicals confuse Google and can lead to the wrong URL being indexed.

Structured Data and Indexation

One area that often gets overlooked in a crawlability and indexation audit is structured data. Adding schema markup does not directly help crawling, but it can influence how Google displays your page in search results. Rich snippets can increase click-through rates, which signals to Google that your page is valuable. That can lead to better indexation over time. Use Google’s Rich Results Test to validate your markup. If you have errors, fix them.

Structured data also helps Google understand what a page is about. For an ecommerce site, product schema with price, availability, and reviews can make the difference between a plain snippet and a rich result that drives traffic.

Mobile-First Indexing and Crawl Budget

Google now indexes the mobile version of your site first. If your mobile site is slow, has broken resources, or hides content behind tabs, your indexation will suffer. Run a mobile-friendly test in Search Console. Check that your mobile pages have the same content as desktop. Also check that Googlebot can access all the resources (CSS, JavaScript, images) on mobile. Blocking those in robots.txt or via noindex tags can prevent proper rendering.

Crawl budget is the number of URLs Googlebot will crawl on your site within a given period. For small sites, budget is rarely a bottleneck. But for large sites with thousands of URLs, every wasted crawl matters. Removing broken links, consolidating redirect chains, and improving page load speed all conserve crawl budget for the pages that matter.

When to Run This Audit

I recommend a full crawlability and indexation audit at least once per quarter. More often if you are in the middle of a site migration, a redesign, or a content overhaul. The audit should also be part of any pre-launch checklist for a new site. It is much cheaper to catch a noindex tag before launch than after you have published fifty articles.

For a Brisbane-based SEO consultant like myself, this kind of audit is the bread and butter of technical SEO. It is not glamorous. But it is the highest-leverage activity you can do for organic growth. You can write the best content in your industry, but if Google cannot crawl it or will not index it, nobody will see it.

Final Thoughts

A crawlability and indexation audit is not a one-time project. It is a maintenance discipline. Google changes its algorithms. Your site changes its structure. New pages get added, old pages get removed, and somewhere along the way a robots.txt rule gets edited or a canonical tag gets misapplied. The only way to stay on top of it is to run the audit regularly and act on what you find.

If you have never done one, start this week. Open Google Search Console, look at the index coverage report, and pick the top five excluded pages. Figure out why they are excluded and fix them. Then run a crawl with an SEO spider and clean up the redirect chains. You will likely see a traffic increase within a month. That is not a theory. That is what happens when you remove the barriers between your content and the people searching for it.