Crawling and indexing
An XML sitemap helps discovery—it does not guarantee indexing
A sitemap is a useful list of preferred URLs and metadata for crawlers. It is not an instruction to rank or even index every submitted page. The rest of the site still has to send consistent signals about which pages matter and why.
What submission really means
Submitting a sitemap helps a search engine discover URLs and gives you reporting in Search Console. Google still decides when to crawl and whether a page belongs in its index based on access, canonicalization, rendering, duplication, quality, and wider site signals.
1. Publish only the URLs you want indexed
A sitemap should contain absolute, canonical, indexable URLs that return successful responses. Exclude redirects, errors, alternate parameter versions, internal search results, private pages, and URLs blocked from indexing. Splitting very large or operationally distinct sets into child sitemaps can make monitoring clearer.
- Use one consistent hostname, protocol, case, and trailing-slash policy across links, canonicals, redirects, and sitemap locations.
- Set lastmod only to the date of a meaningful page change; do not refresh every timestamp on each sitemap generation.
- Keep every important URL reachable through ordinary crawlable links even when it also appears in the sitemap.
2. Map languages as equivalent documents
Each translated page should identify its own canonical URL and list the full reciprocal hreflang set, including itself. The HTML head or XML sitemap can carry these mappings; using both consistently makes them easier to test. An x-default URL can serve visitors whose language does not match a specific version.
- Translate URLs when that helps people, but keep a stable internal content identity so alternates cannot drift apart.
- Do not point every language canonical to the English page; each genuine translation normally canonicalizes to itself.
- Redirect duplicate locale roots and path variants to one final URL before advertising that URL in alternates or the sitemap.
3. Diagnose indexing outside the XML file
When a URL is discovered but not indexed, inspect it in Search Console and test the live page. Confirm the final response, robots controls, rendered content, canonical selected by Google, internal links, and similarity to other pages. A technically valid page may still be omitted when its purpose or unique value is unclear.
- Compare the declared canonical with Google’s selected canonical and remove conflicting signals before requesting another crawl.
- Check whether useful content and internal links are present in rendered HTML without requiring fragile interaction or delayed scripts.
- Consolidate thin or overlapping pages instead of generating more URLs to make the sitemap appear larger.
4. Measure coverage and search value separately
Track submitted and indexed pages by template and language, but do not treat maximum coverage as the business goal. Search impressions, qualified visits, conversions, and assisted journeys show whether indexed content is useful. Review sitemap errors and meaningful content releases on a regular cadence.
- Submit the sitemap in the correct Search Console property and keep the URL referenced in robots.txt.
- Use small cohorts after a release so you can distinguish a technical change from a broad demand or algorithm shift.
- Retire obsolete pages deliberately with redirects or appropriate status codes and remove them from links and sitemaps.
A sitemap and indexing checklist
- 01 List only absolute, preferred URLs that return 200 and are allowed to be indexed.
- 02 Align internal links, redirects, canonicals, trailing slashes, and sitemap locations.
- 03 Provide complete reciprocal hreflang groups for every genuine translated document.
- 04 Use truthful lastmod values and omit optional metadata that you cannot maintain accurately.
- 05 Submit and monitor the sitemap in Search Console, then inspect representative excluded URLs individually.
- 06 Improve internal linking and distinct page value before assuming that repeated submission will create visibility.
Discovery is the beginning, not the finish line
A clean sitemap removes ambiguity and makes monitoring easier. Indexing and visibility come from the complete system: accessible responses, consistent canonical and language signals, crawlable internal links, distinctive useful content, and evidence that visitors find the result valuable.
Primary references
Continue with a related guide
These checks influence one another. Follow the next guide to verify the surrounding signals, not only one tag or score.