Lucent Owl logo Lucent Owl

SEO Diagrams

Last updated:

1 related guide
diagrams crawling core-web-vitals robots-txt canonicalization

From Crawl to Results

Flow

A page's path from crawl queue to results, and the control acting at each step.

How a search engine crawls, renders, indexes, and serves a page URLs from links on known pages and from sitemaps enter the crawl queue. The crawler, Googlebot, fetches the raw HTML. Pages that return 200 wait in the render queue, sometimes for seconds or longer, until a headless Chromium renderer runs their JavaScript. The indexer groups duplicate URLs and keeps one canonical URL per group, and serving ranks canonical URLs for each query. Links found in the raw HTML and after rendering go back to the crawl queue. Controls act at each stage: robots.txt disallow at the crawl queue, the HTTP status code and an initial noindex at the crawler, blocked CSS or JavaScript at the renderer, noindex, redirects, and rel=canonical at the index, and snippet controls at serving. links on known pages, sitemaps Crawl queue URLs waiting to be fetched robots.txt disallow URL isn't fetched. If other pages link to it, it can still be indexed without its content. Crawler (Googlebot) fetches the raw HTML HTTP status code, initial noindex 4xx removes the URL, and 5xx or 429 slows crawling. A noindex here may mean the page is never rendered. Render queue waits seconds, or longer Renderer headless Chromium runs the JS Blocked CSS or JavaScript The page renders incompletely, so content and links on it can be missed. Index groups duplicates, keeps canonicals noindex, redirects, rel=canonical noindex keeps the page out of results. Redirects and canonical tags steer which URL represents it. Serving ranks canonical URLs per query nosnippet, max-snippet, data-nosnippet Limit what the result, including AI features, can show from the page. 200 OK new links found in the HTML and after rendering page moves to the next stage discovered links re-enter the queue control acting at that stage

LCP Across One Page's Visits

Chart

LCP across one page's visits, with the median and 75th percentile.

Distribution of Largest Contentful Paint across one page's visits A histogram of LCP times for an illustrative page, in quarter-second bins from 0 to 7 seconds. 70 percent of visits are within the 2.5 second good threshold and 30 percent are slower. Visits peak just under 1 second, with half finishing by 1.6 seconds, and a long tail of slower visits stretches to the right, with about 4 percent beyond 7 seconds. The good threshold is 2.5 seconds. The 75th percentile falls at 2.8 seconds, past the threshold, so the page fails LCP even though most visits are fast. 0 s 1 s 2 s 3 s 4 s 5 s 6 s 7 s 0% 2% 4% 6% 8% 10% 12% median 1.6 s good threshold 2.5 s 75th percentile 2.8 s: fails slowest quarter of visits lies to the right 4% of visits beyond 7 s within 2.5 s (70% of visits) slower than 2.5 s (30% of visits) LCP of each visit, in seconds (illustrative page) share of visits

Found this useful? Share it:

Share on LinkedIn