Crawling

How search engines discover pages by following links across the web.

1 min readLast updated Apr 2026

Reviewed by Golden Digital·Operator-reviewed ecommerce reference

Quick Reference

CategorySEO for Ecommerce

Related Terms1

How search engines discover pages by following links across the web.

Why It Matters

If Google can't crawl your pages, they can't rank. Crawl budget is limited—large sites especially need to ensure important pages are discoverable while unimportant pages don't waste crawl resources. Poor crawlability is a silent SEO killer.

Practical Example

Scenario

An ecommerce site with 50,000 product pages analyzes their crawl stats in Google Search Console.

Calculation

Crawl budget: 10,000 pages/day. Wasted on: filter pages (3,000), out-of-stock (2,000), pagination (2,500). Only 2,500/day on actual products.

Result

By blocking filter URLs and consolidating pagination, they redirect crawl budget to products—improving average crawl frequency from weekly to daily on key pages.

Pro Tips

1Submit an XML sitemap with only indexable, important pages
2Use internal linking strategically—pages with more links get crawled more
3Block unimportant pages (filters, sorts, admin) in robots.txt to save crawl budget
4Monitor Google Search Console 'Crawl Stats' regularly for issues

Common Mistakes to Avoid

Creating infinite URL variations via filters, parameters, and sorts

Burying important pages deep in site architecture (4+ clicks from homepage)

Blocking pages you actually want indexed via robots.txt or noindex

Frequently asked questions

Crawl budget is how many pages Google will crawl on your site daily. For sites under 10,000 pages, it's rarely an issue. For larger sites (100K+ pages), it's critical—you want Google spending budget on important pages, not duplicates or filters.

Google Search Console → Settings → Crawl Stats shows pages crawled per day and by type. Also check Coverage report for crawl errors. Individual URL inspection shows when a page was last crawled and any issues discovered.

Put it to work

Related Terms

Indexing

Search engines storing and organizing discovered pages in their database.

ONE FREE AUDIT

Run your free Shopify audit.

50+ factors. Real fixes. Same framework we run for $10M brands.

Run a Free Audit Tell us about your brand

Free · No credit card required

You'll talk directly to the people doing the work.

7.7x blended return at scale — seven figures attributed, across five straight quarters · Modern Prairie · Apr 2025–Jun 2026 Read the Modern Prairie case study