rules - Factor Documentation (original) (raw)

rules

Vocabulary
robots

Definition

IN: robots

TUPLE: rules
user-agents allows disallows visit-time request-rate
crawl-delay unknowns ;