Rustdoc internals - Rust Compiler Development Guide (original) (raw)

Rust Compiler Development Guide

Rustdoc Internals

This page describes rustdoc's passes and modes. For an overview of rustdoc, see the "Rustdoc overview" chapter.

From Crate to Clean

In core.rs are two central items: the rustdoc::core::DocContext struct, and the rustdoc::core::run_global_ctxt function. The latter is where rustdoc calls out to rustc to compile a crate to the point whererustdoc can take over. The former is a state container used when crawling through a crate to gather its documentation.

The main process of crate crawling is done in clean/mod.rs through several functions with names that start with clean_. Each function accepts an hiror ty data structure, and outputs a clean structure used by rustdoc. For example, this function for converting lifetimes:

fn clean_lifetime<'tcx>(lifetime: &hir::Lifetime, cx: &mut DocContext<'tcx>) -> Lifetime {
    if let Some(
        rbv::ResolvedArg::EarlyBound(did)
        | rbv::ResolvedArg::LateBound(_, _, did)
        | rbv::ResolvedArg::Free(_, did),
    ) = cx.tcx.named_bound_var(lifetime.hir_id)
        && let Some(lt) = cx.args.get(&did).and_then(|arg| arg.as_lt())
    {
        return lt.clone();
    }
    Lifetime(lifetime.ident.name)
}

Also, clean/mod.rs defines the types for the "cleaned" Abstract Syntax Tree (AST) used later to render documentation pages. Each usually accompanies aclean_* function that takes some AST or High-Level Intermediate Representation (HIR) type from rustc and converts it into the appropriate "cleaned" type. "Big" items like modules or associated items may have some extra processing in its clean function, but for the most part theseimpls are straightforward conversions. The "entry point" to this module isclean::utils::krate, which is called by run_global_ctxt.

The first step in clean::utils::krate is to invokevisit_ast::RustdocVisitor to process the module tree into an intermediatevisit_ast::Module. This is the step that actually crawls therustc_hir::Crate, normalizing various aspects of name resolution, such as:

After this step, clean::krate invokes clean_doc_module, which actually converts the HIR items to the cleaned AST. This is also the step where cross- crate inlining is performed, which requires converting rustc_middle data structures into the cleaned AST.

The other major thing that happens in clean/mod.rs is the collection of doc comments and #[doc=""] attributes into a separate field of the Attributes struct, present on anything that gets hand-written documentation. This makes it easier to collect this documentation later in the process.

The primary output of this process is a clean::types::Crate with a tree of Items which describe the publicly-documentable items in the target crate.

Passes Anything But a Gas Station (or: Hot Potato)

Before moving on to the next major step, a few important "passes" occur over the cleaned AST. Several of these passes are lints and reports, but some of them mutate or generate new items.

These are all implemented in the librustdoc/passes directory, one file per pass. By default, all of these passes are run on a crate, but the ones regarding dropping private/hidden items can be bypassed by passing--document-private-items to rustdoc. Note that unlike the previous set of ASTtransformations, the passes are run on the cleaned crate.

Here is the list of passes as of March 2023:

There is also a stripper module in librustdoc/passes, but it is a collection of utility functions for the strip-* passes and is not a pass itself.

From Clean To HTML

This is where the "second phase" in rustdoc begins. This phase primarily lives in the librustdoc/formats and librustdoc/html folders, and it all starts withformats::renderer::run_format. This code is responsible for setting up a type thatimpl FormatRenderer, which for HTML is Context.

This structure contains methods that get called by run_format to drive the doc rendering, which includes:

In item, the "page rendering" occurs, via a mixture of Askama templates and manual write!() calls, starting in html/layout.rs. The parts that have not been converted to templates occur within a series of std::fmt::Displayimplementations and functions that pass around a &mut std::fmt::Formatter.

The parts that actually generate HTML from the items and documentation start with print_item defined in html/render/print_item.rs, which switches out to one of several item_* functions based on kind of Item being rendered.

Depending on what kind of rendering code you're looking for, you'll probably find it either in html/render/mod.rs for major items like "what sections should I print for a struct page" or html/format.rs for smaller component pieces like "how should I print a where clause as part of some other item".

Whenever rustdoc comes across an item that should print hand-written documentation alongside, it calls out to html/markdown.rs which interfaces with the Markdown parser. This is exposed as a series of types that wrap a string of Markdown, and implement fmt::Display to emit HTML text. It takes special care to enable certain features like footnotes and tables and add syntax highlighting to Rust code blocks (via html/highlight.rs) before running the Markdown parser. There's also a function find_codes which is called by find_testable_codes that specifically scans for Rust code blocks so the test-runner code can find all the doctests in the crate.

From Soup to Nuts (or: "An Unbroken Thread Stretches From Those First Cells To Us")

It's important to note that rustdoc can ask the compiler for type information directly, even during HTML generation. This didn't used to be the case, and a lot of rustdoc's architecture was designed around not doing that, but aTyCtxt is now passed to formats::renderer::run_format, which is used to run generation for both HTML and the (unstable as of March 2023) JSON format.

This change has allowed other changes to remove data from the "clean" ASTthat can be easily derived from TyCtxt queries, and we'll usually accept PRs that remove fields from "clean" (it's been soft-deprecated), but this is complicated from two other constraints that rustdoc runs under:

The "clean" AST acts as a common output format for both input formats. There is also some data in clean that doesn't correspond directly to HIR, such as synthetic impls for auto traits and blanket impls generated by thecollect-trait-impls pass.

Some additional data is stored inhtml::render::context::{Context, SharedContext}. These two types serve as ways to segregate rustdoc's data for an eventual future with multithreaded doc generation, as well as just keeping things organized:

Other Tricks Up Its Sleeve

All this describes the process for generating HTML documentation from a Rust crate, but there are couple other major modes that rustdoc runs in. It can also be run on a standalone Markdown file, or it can run doctests on Rust code or standalone Markdown files. For the former, it shortcuts straight tohtml/markdown.rs, optionally including a mode which inserts a Table of Contents to the output HTML.

For the latter, rustdoc runs a similar partial-compilation to get relevant documentation in test.rs, but instead of going through the full clean and render process, it runs a much simpler crate walk to grab just the hand-written documentation. Combined with the aforementioned "find_testable_code" in html/markdown.rs, it builds up a collection of tests to run before handing them off to the test runner. One notable location in test.rs is the function make_test, which is where hand-writtendoctests get transformed into something that can be executed.

Some extra reading about make_test can be foundhere.

Testing Locally

Some features of the generated HTML documentation might require local storage to be used across pages, which doesn't work well without an HTTPserver. To test these features locally, you can run a local HTTP server, like this:

$ ./x doc library
# The documentation has been generated into `build/[YOUR ARCH]/doc`.
$ python3 -m http.server -d build/[YOUR ARCH]/doc

Now you can browse your documentation just like you would if it was hosted on the internet. For example, the url for std will be rust/std/.

See Also