Crawlability and indexability are the foundation of organic search visibility. If Google cannot crawl a page, it cannot index it. If it cannot index a page, that page will never appear in search results regardless of how good the content is.
Crawlability vs Indexability
Crawlability refers to whether a search engine's bot can access and download the content of a URL. Crawl barriers include robots.txt blocks, server errors, authentication requirements, and network timeouts.
Indexability refers to whether a crawled page is eligible to appear in search results. A page can be crawlable but not indexable due to noindex directives, canonical tags pointing elsewhere, or low content quality.
Crawl Budget
Google allocates a crawl budget to each domain based on two factors: crawl rate limit (how fast it can crawl without overloading your server) and crawl demand (how valuable Google considers your pages). Sites under 10,000 pages rarely need to worry about crawl budget. Sites above 100,000 pages must actively manage it.
Optimizing Crawl Budget
- Remove low-value pages from crawl paths - Block faceted navigation, internal search results, and duplicate parameter URLs via robots.txt.
- Fix server errors - Persistent 5xx responses reduce Google's crawl rate for your domain.
- Improve response times - Faster servers allow more pages to be crawled in the same time window. Keep server response times under 200ms.
- Flatten site architecture - Important pages should be reachable within 3 clicks from the homepage.
Common Crawl Barriers
robots.txt Blocks
The most common accidental crawl barrier. Verify your robots.txt is not blocking critical page sections, CSS files, or JavaScript files that Google needs for rendering.
Server Errors (5xx)
If Googlebot receives 5xx errors repeatedly, it reduces crawl frequency for the entire domain. Monitor server uptime and fix intermittent errors promptly.
Redirect Chains
Chains longer than 5 hops cause Googlebot to abandon the crawl for that URL. Flatten all chains to single-hop redirects.
Orphan Pages
Pages with no internal links pointing to them are unlikely to be discovered by crawlers. Ensure every important page is linked from at least one other page on your site.
Indexability Signals
noindex Directive
The noindex meta tag or X-Robots-Tag HTTP header prevents a crawled page from appearing in search results:
<meta name="robots" content="noindex, follow" />
Use this for utility pages (login, cart, thank-you pages) that should not rank.
Canonical Tags
A canonical tag pointing to a different URL tells Google to index that other URL instead. Check that canonicals are intentional and not created by CMS defaults.
Content Quality
Google may choose not to index a page if it determines the content is too thin, too similar to other indexed pages, or of insufficient quality. Pages with fewer than 200 words of unique text are at higher risk of exclusion.
Diagnosing Index Issues
Google Search Console > Page Indexing is the primary diagnostic tool. It categorizes every known URL by its indexing status:
- Crawled - currently not indexed - Google crawled the page but decided not to include it. Usually a content quality issue.
- Discovered - currently not indexed - Google knows about the URL but has not crawled it yet. This indicates crawl budget constraints.
- Blocked by robots.txt - Verify this is intentional.
- Excluded by noindex tag - Verify this is intentional.
URL Inspection Tool
Test any individual URL to see Google's cached version, check for rendering issues, and request indexing. Use this to verify fixes after resolving crawl or index barriers.
JavaScript Rendering
Google renders JavaScript but with a delay. Pages that rely entirely on client-side rendering may experience slower indexing compared to server-rendered content. For critical SEO pages, use server-side rendering (SSR) or static site generation (SSG) to ensure content is present in the initial HTML response.
Test JavaScript rendering with the URL Inspection tool's "View Rendered Page" option to confirm Google sees the same content users see.