QA Framework: Operational Guidelines (original) (raw)

1. Introduction

1.1. Document overview

The principal parts of this specification are:

• This introduction provides the context for understanding the requirements in section 2, including some use cases.
• Section 2 presents eight guidelines, each consisting of a number of checkpoints that contain lists of conformance requirements.
• Section 3 briefly summarizes the relationship of W3C's Working Groups to the QA Activity
• Section 4 defines conformance to this specification and explains how to make conformance claims.
• Two checklists in external appendices present: all checkpoints in document order; and, all checkpoints sorted by priority (the ICS for this specification).

Integrated with this specification on a checkpoint-by-checkpoint basis, although published separately, is:

• Operational Examples & Techniques, which provides suggestions and examples of how each checkpoint might be satisfied, as well as templates to aid the implementation of the checkpoints' requirements.

1.2. Scope and goals

The scope of this specification is a set of verifiable requirements for the process and operational aspects of the quality practices of W3C Working Groups. The primary goal is to help the W3C Working Groups (WGs) with the planning, development, deployment, and maintenance of conformance test materials (TM).

For this guidelines document, the term conformance test materials includes:

• conformance test suites,
• validation tools,
• conformance checklists,
• any other materials that are used to check or indicate conformance of an implementation to a specification.

1.3. Class of product and audience

The class of product that is the target of the requirements in this specification is: all process and operational aspects of Working Group quality practices. This includes

• the WG's planning and commitment for QA,
• staffing to meet commitments,
• organization and logistics of the quality practices,
• publication and maintenance,
• synchronization of quality processes with other WG activities.

The intended audience for these guidelines is all W3C Working Group members, as well as the actual developers of conformance materials for W3C specifications.

Early adoption of sound quality practices by the Working Groups (WGs) is optimal. Nevertheless, these guidelines are written for newly chartered WGs and existing WGs alike. Working Groups who may already be doing some of these activities should review the document and incorporate the principles and guidelines into their quality practices as much as possible.

1.4. Motivation and expected benefits

As the complexity of W3C specifications and their interdependencies increase, quality assurance becomes even more important to ensuring acceptance and deployment in the market.

These guidelines aim to capture the experiences, good practices, activities, and lessons-learned of the Working Groups, and to present them in a comprehensive, cohesive set of documents for all to use and benefit from. They thereby aim to:

• standardize the best of current practice,
• allow the WG's toreuse what works rather than having to reinvent,
• which should facilitate and expedite the work of theWGs,
• and should also promote consistency across the variousWG quality activities and deliverables.

Read more in the introductory sections of QA Framework: Introduction.

1.5. Relationship to other specifications

This document is part of a family of seven QA Framework documents, whose aim is to help the WGs improve all aspects of their quality practices by solidifying and extending current quality practices found within the W3C.

The QA Framework family includes:

• an introduction, QA Framework: Introduction [QAF-INTRO],
• a set of three guidelines documents,
• a companion set of three examples & techniques documents.

The Introduction should be read. It is the key to understanding the whole QA Framework family, and especially the pairwise relationship between the guidelines documents and their examples & techniques counterparts.

The QA Framework documents are independently implementable, but still are interrelated and complementary. For example, there is a close relationship between the WG processes for dealing with versions and errata (in this Operational Guidelines), and the test framework architecture for handling multiple versions and errata levels (in Test Guidelines).

1.6. Understanding this document

This specification uses a model of organizing guidelines and implementable, verifiable checkpoints. Each guideline and its checkpoints are organized and structured as follows:

Guideline [_Number_]. [_Statement of the guideline_]

[_some discussion of the guideline and its rationale_]

Checkpoints:

Checkpoint [_Number_]. [_the statement of the checkpoint (its "title")_][Priority�[_Level_]]

Conformance requirements:

[_a statement of the conformance requirements of the checkpoint, i.e., how to fulfill the checkpoint_]

Discussion [_, Notes, Rationale_]

[_optional sections of informative notes, rationale, clarifying examples, and cross references to related guidelines or checkpoints_]

Examples & Techniques for checkpoint.

Notes:

• the checkpoint priority has value 1, 2 or 3;
• "Examples & Techniques for checkpoint" is a link to the corresponding section in QA Framework: Operational Examples & Techniques; [OPS-EXTECH]
• the "Conformance requirements" section contains one or more individual conformance requirements that use the RFC2119 keywords (the Conformance clause discusses this more).

1.7. Checklists to assess conformance

Two checklists facilitate the use of this guidelines specification. The checklists are in two separate appendices:

• a single table containing all guidelines and checkpoints sorted in their original order [OPS-CHECKLIST];
• all checkpoints sorted by their priorities into three tables, respectively containing the checkpoints of priority 1, 2, 3 (see next section) [OPS-ICS].

The first checklist provides a topical outline of the "Guidelines" chapter of this specification. The second checklist is an Implementation Conformance Statement (ICS) pro forma for this specification.

Use the ICS checklist to record a conformance assessment of a Working Group to this specification. For each checkpoint, consider the individual conformance requirements of the checkpoint and record in the ICS checklist whether the checkpoint is satisfied, or not satisfied, or not applicable.

1.8. Checkpoint priorities

Some checkpoints are more critical than others for the timely production of high-quality, highly usable test materials. Therefore each checkpoint has a priority level -- Priority 1 (highest), Priority 2, or Priority 3 -- based on the checkpoint's importance. See the priority definitionsin the Conformance clause.

1.9. A chronological view of the guidelines

"Earlier is better" is a general truism for QA planning and execution. It is also true that later (than ideal timing) is better than nothing. Recognizing the diverse realities of W3C's many existing working groups, theOperational Guidelines specifically avoided binding the guidelines to any ideal timeline.

This table presents an example of an idealized chronological correlation between the individual guidelines and the stages of aWG's life and work. The ideal example assumes test materials (TM) development within the WG, starting at early working drafts, proceeding in parallel with specification development, and culminating with TM publication at the start of CR.

Correlation of Guidelines with phases in Working Group (WG) life

Notes:

• Abbreviations used for Recommendation-track phases: WD -- (published) Working Draft;LC -- Last CallWD; CR -- Candidate Recommendation; PR -- Proposed Recommendation; Rec -- Recommendation
• The "applies" sub-row is about the stage(s) of WG life and work to which the topic of the guideline applies
• The "address" sub-row is about the stage(s) at which the WG ideally addresses the requirements of the guideline.

1.10. Usage of terminology in this document

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" are used as defined in RFC 2119 [RFC2119]. When used with the normative RFC2119 meanings, they are all upper case. Occurrences of these words in lower case comprise normal prose usage, with no normative implications.

Unusual terms in this framework document are defined when first used. Generally useful QA-specific terms will usually be put into the QA Glossary[QA-GLOSSARY] eventually. When used in this specification, terms have the meanings assigned herein, or in the QA Glossary if they are not defined herein.

1.11. Use cases

The following scenarios illustrate typical uses of Operational Guidelines:

• A new Working Group is just drafting a Charter. The Charter-drafting team, including prospective WG members, consults the QA Framework: Introduction. Appreciating the payback of early quality commitment and integration, the team refers to Operational Guidelines. Guided by the checkpoints of the charter-phase guidelines, and using the provided Charter Template, the WG writes its quality level and test materials commitments into its draft Charter.
• An existing Working Group has just finished writing its Requirements and Use Cases documents, and is beginning to draft its specification. Concurrently, it is starting to discuss test suite (TS) plans. Consulting Operational Guidelines, it jump starts the TS project by appointing a QA Moderator and a part-time TS team, and quickly produces a first version of its QA Process Document (QAPD) using the provided QAPD template.
• A Working Group has been re-chartered to finish a Second Edition (maintenance release) of its specification, and to develop the next functional version, 2.0. The group did not develop a test suite (TS) during its first charter, but a collaboration of outside organizations and an industry consortium has developed one. The WG and developers have agreed in concept to transfer it to a stable home in W3C.Operational Guidelines is consulted for a detailing of preliminary steps, requirements, and specific activities involved in making a smooth and trouble-free transfer.
• A Working Group is almost ready to request Candidate Recommendation (CR), and has met its goal of having a comprehensive test suite (TS) ready for the trial implementation period of CR. Looking at making the TS publicly available, it uncovers numerous unanticipated procedural issues aboutTS publication. The issue of the right TS distribution licenses is proving particularly hard. Consulting Operational Guidelines, the WG finds discussion of the pros and cons of the W3C Software License and the Document License, as well as guidance on resolving an optimal licensing strategy .
• A Working Group has finished building a basic test suite (TS) for its specification, and is nearly finished with plans for the first release of its basic TS. After the release, it wants to take advantage of the large test collections of several early implementors, and start accepting contributions. Rather than figure out the issues and write a Test Contribution & Review process from scratch, it takes guidance from Operational Guidelines, and uses the examples and templates of Operational Examples &Techniques to significantly shorten the development of its contribution process.