GitHub - apache/sling-org-apache-sling-sitemap: Apache Sling Sitemap (original) (raw)

Apache Sling Sitemap

The Apache Sling Sitemap module is an extension for Apache Sling that helps to serve XML Sitemaps to Search Engines. It was designed to cover various uses cases from small sites serving sitemaps on-demand, large sites generating them in the background to even sites that collect 3rd party data to include dynamically rendered pages.

Highlights

A simple, builder-like API to create Sitemaps, that hides all the XML specifics from the implementation
Support for on-demand and background generation w/ continuation after job interruption
Support for nested sitemaps, that are automatically collected into a sitemap indexes
Support for auto-balancing sitemaps into multiple files for background generation

Getting Started

The Sling Sitemap module is rather abstract, as it highly depends on the content structure of the application it is used in. To get started a few things must be done:

Implement at least one SitemapGenerator. The abstract ResourceTreeSitemapGenerator may be a good starting point for any generator walking the resource tree.
Configure the SitemapServlet to register the resource type(s) which may be a sitemap root resource.
Configure a service user mapping for org.apache.sling.sitemap:sitemap-reader granting read access to the content.
Configure another service user mapping one for org.apache.sling.sitemap:sitemap-writer granting read access to the content and write access to the storage path for background generation (per default /var/sitemaps).
Finally, either configure a SitemapScheduler to create a job for background generation, or implement theSitemapGenerator to serve the sitemaps on-demand.

Implementation Details

Content Model

In order to serve a sitemap a resource must be marked as sitemap root resource. This is done by adding a sling:sitemapRoot = true property either to the resource or its jcr:content child.

When multiple resources in a resource tree are marked as sitemap roots, the one closest to the repository root is considered to top level sitemap root and serves a sitemap-index additionally to the sitemap.

/content/site/ch/
 + de-ch/
  - sitemapRoot = true
  + faqs/
  + news/
  + products/
   - sitemapRoot = true
 + fr-ch/
  - sitemapRoot = true 
  ...

In the example above the paths de-ch/ and fr-ch/ are both top level sitemap roots and de-ch/products/ is a nested sitemap root. The sitemaps will be served as following, assuming the appropriate mappings being in place:

https://site.ch/de-ch.sitemap-index.xml
https://site.ch/de-ch.sitemap.xml
https://site.ch/de-ch.sitemap.products-sitemap.xml
https://site.ch/fr-ch.sitemap-index.xml
https://site.ch/fr-ch.sitemap.xml

Sitemap Generation

The module does not ship a specific SitemapGenerator implementation. Products/Projects using the Apache Sling Sitemap module must implement an appropriate SitemapGenerator that fits their content model. An abstractResourcceTreeSitemapGenator implementation is available to cover the most common use cases.

Each SitemapGenerator may produce multiple sitemaps for a given sitemap root. For example, a default sitemap and a news specific sitemap that contains only up to 1000 urls that were changed in the past 2 days. Or as another example, consider an eCommerce site, that generates a product sitemap for each top level category of a catalog. To enable that, a SitemapGenerator can return 0..n names for a given resource, each name representing a single sitemap at the given resource.

SitemapGenerator implementations need to be registered as OSGI services. In case of an overlap, it depends on the service.ranking which SitemapGenerators are used for a particular sitemap root. For example, it may be that twoSitemapGenerators can generate a default sitemap for a sitemap root, but the second one can also generate a news sitemap. The first, higher-ranked SitemapGenerator will be used for the default sitemap, but the second one will still be taken into account for the news sitemap.

Background Generation

A configurable scheduler can be configured to trigger background generation. Multiple schedules can be created for different sitemap names or generators as described before. This enables the background generation to align on the cadence in which particular content may be updated, for example product catalogues that sync on a regular basis.

For each sitemap root in the repository and for each sitemap name returned for them, a job with the topicorg/apache/sling/sitemap/build will be queued. The SitemapGeneratorExecutor implementation consumes those jobs and calls the corresponding SitemapGeneator. It is recommended to create an unordered queue for those jobs so that they can be distributed across multiple instances within a cluster.

The SitemapGeneratorExecutor provides an execution context to the SitemapGenerator which it may use to keep track of progress. The implementation on the other hand will persist this state along with the already written sitemap after a configurable amount of urls has been added. This allows to resume jobs after an instance gets restarted or discarded in a dynamic cluster. Per default the SitemapGeneratorExecutor is configured with a chunk size of Integer.MAX_VALUE, which effectively means that no checkpoints will be written. When using this feature make sure to find a good balance between write overhead and performance gain for those particular cases.

Background generation supports auto-balancing according to configurable limits for size (in bytes), and the number of urls in a single sitemap file. This is transparently handled by the SitemapGeneratorExecutor, providing a Sitemapinstance which pipes added urls to multiple files when needed. As a consequence returning sitemap files from storage for a given name and sitemap root may result in multiple return values.

On-demand Generation

For smaller sites, calculating sitemaps in the background may not be necessary and serving sitemaps when they get requested may even result in higher accuracy. On the other hand serving a sitemap on-demand within the timeout of different crawlers highly depends on the amount of content and the SitemapGenerator implementation(s) used. Because of that, serving sitemaps on-demand must be explicitly enabled.

To enable serving sitemaps on-demand, a SitemapGenerator must indicate that a particular sitemap name should be served on demand. Alternatively the SitemapGeneratorManagerImpl can be configured to force all sitemaps to be served on-demand. In both cases the SitemapServlet changes its behaviour slightly:

When serving a sitemap-index, it queries for all sitemap roots and adds the sitemaps of those, that should be served on-demand. Additionally, all sitemaps form the top level sitemap root's storage location are added if not already present.
For serving sitemap files it first checks if the sitemap should be served on-demand, if not it falls back to the sitemap file from the top level sitemap root's storage location.

On-demand generation does NOT support auto-balancing. The configured limits will be ignored.

Sitemap Extensions

The builder-like API supports adding sitemap extensions (image, video, news, alternate languages ...) by implementing anExtensionProvider. This provider has to be registered specifying namespace, prefix, and localName of the xml element the extension adds to an url object.

In order to hide the implementation detail from the consumer API, the ExtensionProvider works with an abstractAbstractExtension class, and the consumer API only with an Extension marker interface. To implement an Extension:

Extend the marker interface with methods to capture the extension specific data
Implement this extension interface by extending the AbstractExtension class
Provide an instance of the extension implementation by implementing an ExtensionProvider
Make sure to register the ExtensionProvider with the extension.interface set to the fqn of the extension interface

An example extension implementation can be found with the AlternateLanguageExtension.

For the following implemented extensions, refer to the respective interfaces in o.a.s.sitemap.builder.extensions: