Exploring all the pages of a website requires choosing the right method from a dozen available approaches. XML Sitemap, search operators, SEO crawlers, log analysis: each technique covers a different scope and leaves distinct blind spots. The challenge is not to know them all, but to understand which one detects what, and especially what it misses.
JavaScript Rendering and Crawling: What Each Bot Actually Sees
The ability of a tool to explore the pages of a site directly depends on its handling of JavaScript. Googlebot has been rendering JavaScript since 2019 and thus accesses client-side generated content. In contrast, generative AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Meta-ExternalAgent, Bytespider) do not render any JavaScript: they download JS files without executing them.
For a site built as a SPA (single-page application) or heavily reliant on client-side rendering, this difference changes everything. Dynamically generated internal links remain invisible to these bots. A crawl with a tool that does not render JS will therefore produce an incomplete list of URLs, sometimes significantly so.
A complete exploration project must also take into account the HTML size limit of Googlebot, set at 2 MB per URL. Beyond that, links located at the bottom of the page are simply not read. Third-party tools that mimic Googlebot’s pipeline inherit this constraint.
For those looking for a concrete entry point, it is possible to explore the pages of the Marcelllin site via its sitemap, which illustrates how a structured XML index works.
Comparison of Methods to Find All Pages of a Site
Each URL discovery method covers a specific scope. The table below summarizes the actual capabilities of six common approaches.
| Method | Indexed Pages | Orphan Pages | Blocked Pages (robots.txt) | JS Rendering |
|---|---|---|---|---|
| Site Operator: Google | Yes | No | No | Yes (via Google) |
| XML Sitemap | Yes (if declared) | No | Possible | Not relevant |
| SEO Crawler (Screaming Frog, etc.) | Yes | Partially | Configurable | Optional |
| Google Search Console | Yes | No | No | Yes |
| Server Log Analysis | Yes | Yes | Yes | Not relevant |
| AI Crawlers (GPTBot, ClaudeBot) | Partial | No | No | No |
The conclusion is clear: no single method covers all the URLs of a site. Google’s site operator only shows indexed pages, and Google does not guarantee to display all known URLs. The sitemap only lists URLs that the webmaster has voluntarily declared. Only server log analysis detects orphan pages, as it records every HTTP request received, including those from bots or old external links.
XML Sitemap and robots.txt File: Two Complementary Readings
The robots.txt file tells bots what they should not explore. Consulting it before a crawl helps identify directories that are intentionally excluded (administration pages, staging environments, filtering URLs). The XML sitemap, on the other hand, provides the list of URLs that the site wants to see indexed.
Cross-referencing the two reveals frequent inconsistencies: URLs present in the sitemap but blocked by robots.txt, or vice versa. These discrepancies often indicate configuration errors that harm SEO.
SEO Crawler with JavaScript Rendering: The Broadest Scope
An SEO crawler like Screaming Frog or Sitebulb, configured with JavaScript rendering enabled, offers the most extensive coverage among tools accessible without server access. It follows internal links, executes JS, and can integrate the sitemap as a source of starting URLs.
- Configuring the crawl to ingest the XML sitemap as a list of seed URLs allows for the detection of declared but internally unlinked pages
- Enabling JavaScript rendering ensures the discovery of dynamically generated links, which neither the sitemap nor the site: operator does
- Exporting the list of crawled URLs and comparing it to server logs over a given period reveals orphan pages visited only by bots or via external links
This combination (JS crawler + sitemap + logs) constitutes the most reliable protocol for creating a complete inventory of a site’s URLs.
Orphan Pages and Dead-End Pages: The Blind Spots of Internal Linking
An orphan page has no internal links pointing to it. It exists on the server, can be indexed if an external link leads to it, but remains invisible to a crawler starting from the homepage. Server logs are the only way to detect it without knowing its URL in advance.
Dead-end pages pose a symmetrical problem: they exist in the structure but contain no outgoing links to other pages on the site. A visitor landing on one can only leave the site. For search engines, these pages dilute the crawl budget without redistributing authority through internal links.
- Identify orphan pages by cross-referencing logs/crawl, then link them to the internal structure or remove them
- Spot dead-end pages by the absence of outgoing links in the crawler export
- Ensure that each strategic page is accessible within three clicks from the homepage
A site with several thousand pages naturally accumulates these anomalies over time due to redesigns and content additions. A quarterly audit of the internal linking structure limits technical debt.
Site Structure and Crawl Budget: What Search Engines Prioritize
Search engines allocate a limited number of requests per crawl session to each site. This crawl budget depends on the size of the site, its response speed, and update frequency. A flat structure, where each page is accessible in few levels of depth, facilitates the quick discovery of all URLs.
Parameterized URLs (filters, sorting, pagination) consume crawl budget without providing unique content. Blocking them in robots.txt or managing them via canonical tags reduces waste. Robots.txt controls crawling, the canonical tag controls indexing: both act at different levels and complement each other.
The architecture of a site also determines the maximum depth reached by crawlers. Beyond five levels of depth, the probability that a page is crawled during a session decreases significantly. Reducing the average depth remains the most direct structural lever to improve crawl coverage.