Sitemap: Complete Definition and Guide

What is a Sitemap?

A sitemap is an XML file that provides search engines with a complete map of all accessible pages on a website. It contains the list of URLs with associated metadata: last modification date, estimated update frequency, and relative priority of each page. This file, accessible at /sitemap.xml, allows indexing robots (crawlers) from Google, Bing, and other search engines to efficiently discover and index all of a site's content.

The sitemap is a fundamental element of any SEO strategy. Without it, search engines must discover pages by following internal links, which can be slow and incomplete, especially for large sites or poorly linked pages. At KERN-IT, all our Wagtail projects include an automatically generated and optimized XML sitemap, using the TranslatedSitemap class that correctly handles multilingual versions (French and English) of each page.

Why Sitemaps Matter

The sitemap plays a crucial role in a website's visibility on search engines, and its absence can have significant consequences for organic search rankings.

• Exhaustive discovery: the sitemap ensures that all site pages are known to search engines, including those that would be difficult to access through standard crawling (deep pages, recently published content).
• Accelerated indexing: when a new page is published, the updated sitemap allows search engines to discover and index it more quickly, without waiting for an internal link to be crawled.
• Multilingual management: for multilingual sites like those developed by KERN-IT, the sitemap with hreflang tags indicates to search engines the alternative language versions of each page, avoiding duplicate content issues.
• Quality signal: the last modification date in the sitemap informs search engines of fresh content, encouraging them to recrawl updated pages.
• SEO diagnostics: the sitemap serves as a diagnostic tool in Google Search Console. It allows comparing submitted pages with those actually indexed and identifying indexing issues.

How It Works

An XML sitemap follows a standardized format defined by the sitemaps.org protocol. Each entry is encapsulated in a <url> tag containing at minimum the <loc> tag (the page URL) and optionally <lastmod> (last modification date), <changefreq> (update frequency), and <priority> (relative priority).

In the Wagtail ecosystem used by KERN-IT, the sitemap is dynamically generated by the framework. Wagtail queries the PostgreSQL database to retrieve all published pages (live=True) and generates the corresponding XML. Our custom TranslatedSitemap implementation enriches this process by adding hreflang annotations for each language pair and filtering pages based on the referenced field, allowing certain pages to be excluded from indexing.

The robots.txt file, also served by our Wagtail projects, includes a directive pointing to the sitemap: Sitemap: https://example.com/sitemap.xml. This allows crawlers to automatically locate the sitemap without manual configuration in webmaster tools.

For large sites (over 50,000 URLs), the protocol provides a sitemap index that references multiple sub-sitemaps. Django and Wagtail handle this functionality natively with SitemapIndex.

Concrete Example

KERN-IT's corporate website, built with Wagtail, generates a sitemap that includes all pages published in French and English: homepage, service pages, blog posts, project pages, glossary definitions, product pages, and job listing pages. Each entry includes the last modification date automatically extracted from Wagtail and hreflang annotations linking the FR and EN versions.

When a new blog post is published through the Wagtail admin interface, the sitemap is automatically updated on the next request. Google Search Console, configured with this sitemap, detects the new page and typically indexes it within hours, ensuring rapid visibility for new content.

Implementation

1. Configuration in Django/Wagtail: enable the Django sitemap framework (django.contrib.sitemaps) and configure the /sitemap.xml URL in the urls.py file. Wagtail automatically adds its pages to the sitemap.
2. Page filtering: use the referenced field from KERN-IT's BasePage models to control the inclusion or exclusion of specific pages from the sitemap (temporary pages, thank-you pages, etc.).
3. Multilingual management: implement hreflang annotations in the sitemap for bilingual sites. Our TranslatedSitemap automatically handles this feature for French and English locales.
4. Search engine submission: submit the sitemap in Google Search Console and Bing Webmaster Tools. Regularly check the indexing report to detect errors.
5. Robots.txt: include the Sitemap: directive in the robots.txt file so crawlers automatically find the sitemap.
6. Monitoring: monitor the indexing rate via Google Search Console. A significant gap between submitted and indexed pages indicates a problem to investigate (duplicate content, low-quality pages, technical errors).

Associated Technologies and Tools

• Wagtail: Django CMS that automatically generates the sitemap for all published pages, with native support for multilingual sites.
• Django Sitemaps: framework built into Django that provides base classes for XML sitemap generation.
• Google Search Console: Google tool for submitting the sitemap, tracking indexing, and diagnosing SEO issues.
• Schema.org: structured data vocabulary complementary to the sitemap for enriching information transmitted to search engines.
• Screaming Frog: SEO audit tool that verifies consistency between the sitemap and the actual site structure.
• Internal linking: strategy complementary to the sitemap for reinforcing page discoverability through internal links.

Conclusion

The XML sitemap is a simple but essential tool for organic search optimization. It serves as the direct communication channel between your website and search engines, ensuring that your content is discovered, indexed, and updated efficiently. At KERN-IT, sitemap integration is automated in all our Wagtail projects, with native multilingual management and fine-grained indexing control via the referenced field. Combined with solid internal linking and Schema.org structured data, the sitemap forms the technical foundation of a performant and sustainable SEO strategy.