QA Handbook draft (original) (raw)

Abstract

The QA Handbook (QAH) is a non-normative handbook about the process and operational aspects of the quality practices of W3C's Working Groups (WG). It is intended for Working Group chairs and team contacts, to help them to avoid known pitfalls and to benefit from experiences gathered from the W3C Working Groups themselves. It provides techniques, tools, and templates that should facilitate and accelerate the work of the WGs. This document is one of the QA Framework family of documents of the Quality Assurance (QA) Activity, which includes the other existing or in-progress specifications:Specification Guidelines; and, Test Guidelines.

Status of this document

Warning ... this is an Editor's draft of QAH. It has had some QAWG review & discussion. Lots in it (but not everything yet) reflects QAWG consensus. There are open issues. There are many incomplete (tbd) bits.

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports indexat http://www.w3.org/TR/.

This document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.

This is the first published Working Draft of this document. This draft accurately reflects the redesign of the QA Framework (QAF) resolved by QA Working Group (QAWG) at its 2004 Tech Plenary face-to-face. The QA Handbook (QAH) replaces and incorporates the best features of the former QA Framework: Introduction and QA Framework: Operational Guidelines.

Although accurate in overall content, this draft is incomplete in some details, especially in the examples -- lots more examples are intended. Also, some links to intended references are yet to be done (generally indicated by "@@"). Because this is a significant departure from the last-published QAF, QAWG wants public review and comment on the overall direction as soon as possible. Details will be fleshed out in future publications.

The QA Handbook will be coordinated with the other parts of the QA Framework, Specification Guidelines and Test Guidelines. Synchronization at uniform level of maturity will occur no later than Last Call. The first publications of the various parts will be somewhat more independent.

The QAWG does not expect this document to become a Recommendation. Rather, after further development, review and refinement, it will finally be published and maintained as a WG Note.

Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

The QA Working Group Patent Disclosure page contains details on known patents related to this specification, in conformance with W3C policyrequirements.

Comments on this document may be emailed to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Commenters please note that your comments will be publicly archived and available, do not send information that should not be distributed, such as private data.

Table of contents

[Ed note. As before, it will be auto-generated. Manual skeleton looks like:]

  1. Introduction & roadmap
    1. [...several subsections...]
  2. Early planning and commitment
    1. General modus operandi
    2. Test development framework and processes
    3. Life after WG -- maintenance
  3. Day-to-day QA operations
  4. Licensing & branding
  5. Acquiring test materials
  6. Appendix -- Orientation to QA Framework

Two separate external appendices to this document ... QAPD template and Charter template.

Key to highlighted styles in this draft

Introduction & roadmap

Five simple stories

Here are five use cases for The QA Handbook (QAH). Told as stories, they show when and how QAH could be helpful to chairs and staff contacts. They cover five different situations that might typically arise over the life of the WG.

The QA Handbook could be useful to chairs and staff contacts at Charter time...

Story 1: Think about quality early
The Charter-drafting team of a new Working Group, led by staff contacts and co-Chairs, looks through The QA Handbook. On the advice of theearly-commitment Good Practice bits, the team debates how much it can realistically commit to at this early stage. After deciding on some test materials deliverables, milestones, and spec synchronization that it considers to be safe commitments, it uses the provided Charter Template to write it into the draft Charter.

Or it could be a big help to a chair who is trying to jump-start a test suite project...

Story 2: Jump-starting a testing effort
An existing Working Group has just finished writing its Requirements and Use Cases documents, and has begun to draft its specification. At the same time, it is taking a first look at its test suite (TS) plans. As recommended in the The QA Handbook, the Chair jump starts the TS project by appointing a QA Moderator and part-time TS team, whose first output is a quick QA Process Document (QAPD) using the provided QAPD template.

Or maybe the chairs/staff contacts are pondering the steps to transfer a test suite from an external entity...

Story 3: Test-suite transfer
A Working Group has re-chartered to finish a Second Edition of its specification, and to develop the next functional version, 2.0. The group did not develop a test suite (TS) during its first charter, but a collaboration of outside organizations and an industry consortium has developed one. The Chair and staff contacts have negotiated agreement in principle to transfer it to a stable home in W3C. InThe QA Handbook, they find a handy checklist of preliminary steps, requirements, and specific activities for a smooth transfer.

Or maybe they need to take pre-emptive action due to looming possible license-issue hassles...

Story 4: Test suite licensing
A Working Group is almost ready to request Candidate Recommendation (CR), and has gotten a comprehensive test suite (TS) together for CR's trial implementation period. As the Chair starts to arrange for publication of the TS, she finds the WG split on issues around TS distribution licenses to use. ConsultingThe QA Handbook, she finds discussion of the pros and cons of the W3C licenses (Software License and Document License), and advice on resolving an optimal licensing strategy.

Or maybe they can borrow the experience of other W3C WGs for various useful and common test suite processes...

Story 5: Test development processes
A Working Group has built and released a basic test suite (TS) for its specification. A staff contact has been given the Action Item to plan its expansion to a more comprehensive TS, by leveraging and integrating the large test collections of several early implementors. Rather than figure out the issues and write a Test Contribution & Review process from scratch, he looks at the summary advice in The QA Handbook. QAH points him both to some useful templates, and to more detailed stuff in QA Framework: Test Guidelines, significantly shortening completion of his action item.

Why QAH?

WG Chairs and staff contacts are overworked. They share a lot of the same concerns and pressures as specification editors.

Common problems are:

A lot of groups have faced these issues, and there is a growing body of experience in W3C about what works and what doesn't. QAH aims to pull this together.

Audience and purpose

The QA Handbook (QAH) is intended for: chairs and staff contacts of W3C Working Groups. These include the process-savvy and novices alike.

QAH is a non-normative handbook about the process and operational aspects of the quality practices of WG life -- mostly test suite stuff, but also some specification quality assurance. QAH aims to flag avoidable pitfalls, and to provide techniques, tools, and templates that will facilitate and accelerate the work of the WGs.

Because there is no normative content in QAH, conformance to QAH is not an issue. QAH takes this approach for several reasons, chiefly:

Advice is presented in the imperative voice in Principle and_Good Practice_ sections. The aim is to present helpful guidance gathered from the experience of W3C WGs. If it looks helpful, follow it. If not, don't -- there's no consequence.

QAH scope

Therefore, the QAH deals with things that impact these aspects of WGs' test activities and other quality practices:

Other QA Framework resources

Editors, test builders, and general WG membership will find advice specific to their roles in the other parts of the QA Framework:

Chairs and staff contacts will likely also get involved with these eventually. A brief orientation to the rest of the QA Frameworkis provided in an appendix to this document.

Roadmap to the rest of QAH

After this orientation section, QAH contains four topical modules:

Each presents advice and guidance about some part of the WG's test and quality activities. These tend to have a chronological correspondence to phases (rec-track stages) of the WG's activities. Earlier is better, but later is better than never. Here's a graphic depiction of when, in a WG's life, the various QAH topics are ideally dealt with and applicable:

[TBD. here is planned to be embedded a chopped down version of the OpsGLchronology diagram]

Early planning and commitment

Why care? It's not always easy to anticipate Working Group (WG) needs during the rush and pressure of Chartering, or in the busyness of early WG life. But the earlier the WG can accurately identify and commit to its test and other quality related deliverables, the less likely that the WG get big surprises later on, or run into problems that will delay its progress towards Recommendation.

Additional reason -- binding decisions about WG membership (e.g., numbers per company) often are made at Charter time.

What does this mean? In practice, this means attention to a handful of points, which we enumerate as Good Practices.

Good Practice: Decide ASAP -- will the Working Group build test materials or acquire them?

Clearly, it is going to make a big difference in a Working Group's (WG) staffing requirements -- building test materials tends to be labor intensive (extremely so for some types of specifications). Even if the WG imports them, some staff resources will have to be applied (see last module). The particular test-related activities and milestones in the WG's programme of work will in general be completely different for develop versus acquire options.

Good Practice: Think about and enumerate the QA activities & deliverables that might help the Working Group through the Recommendation track. Minimally, commit to assuring the timely existence of test materials.

Different kinds of QA deliverables might include:

Test materials are by far the most common QA deliverable, and likely will be a key to quick and painless passage through Candidate Recommendation (CR).

Good Practice: Synchronize QA deliverables with specification milestones, and for the bigger QA deliverables, define and schedule intermediate milestones if possible.

This advice echoes that in Tips for Getting to Recommendation Faster, for example see item #6 in section 2.

Good Practice: Consider whether the Working Group should tie any quality criteriato Rec-track advancement.

For example, finishing test materials deliverables before requesting Candidate Recommendation (CR) is common, in order to facilitate CR's Implementation Assessment. (Counter-example: as in the above Story, if the WG is still building them during CR, it is likely to uncover things that will oblige it to cycle in CR.)

This good practice takes the synchronization of the previous one a step further -- the specification cannot advance unless the committed test or other quality criteria are met.

How to Organize a Recommendation Track Transition talks some about the role of test suites in advancement decisions.

Good Practice: Put some thought into how to staff the WG's QA plans.

The earlier this is done, the more options will be available. Some options include:

The third option isn't really different from the first. It's just a way of doing it. But notice that it's an option that is only available by looking into these questions at Charter time.

By the way, there has been confusion (@@find the chairs archive reference@@) about "W3C Process only allows each company to have two members on the WG". In fact, that is not from W3C Process. W3C Process gives considerable freedom for this to be tailored to WG needs -- W3C Process says it may be specified in the Charter. So, for example: two per company in technical discussion, issue resolution, voting, etc; and, additional dedicated test suite builders allowed.

By the way, this is another good reason to put some thought into test suite plans and other quality-related deliverables as early as possible.

Tips for Getting to Recommendation Faster (section 3) also talks some about the value of (early) staffing decisions.

Examples:

How can I do it?

[Editor's overview of what went into this module: implemented QAWG's general consensus (see f2f minutes) thatold-CP1.4was the important one -- it becomes the Principle -- and the other CPs of GL1 & 2 are details (become Good Practices). CP1.5(Rec-track criteria), CP2.1(internal or external?), CP2.3(request alloc of people for QA) are worth mentioning and survived as details -- "Good practice" or "Recommended practice" or ..." CP2.2(staff commitments) survives. Dropped CP1.1(A/AA/AAA to Ops, Spec, Test). CP1.2(commit to TM) and CP1.3(commit to complete TM) seem like details -- should become minimal bottom line of some good practice or whatever. See also checklist.]

3. Day-to-day QA operations

Why care? Over and over we hear the message from successful test suite projects -- define the work process early: test dev't framework, issues list, email list, faq, contribution & review process, etc. [Examples TBD: DOM, Xquery, maybe SVG...] Conversely, there are examples of disorganized collections of test cases with no one apparently in charge, no way to find their status, correctness, [@@Example: what was KD's example of "can't find anyone who's responsible for amorphous undocumented heap of "tests"?]

The W3C Process Document clearly organizes the way that a WG's specs are progressed through the Rec track. But there is no similar template for the other aspects of a WG's life, particularly aspects related to all-important test and quality processes -- logistical setup, communications with the outside, licensing and branding policy, test development process, etc.

Good Practice: Put all of the Working Group's important test and QA-related information in one place in a QA Process Document (QAPD).

How can I do it? Simple: Use the QAPD template. It will guide the user through everything needed, and then some. It is not only a template, but also a checklist of sorts, for the sort of things that the Working Group (WG) should consider having in its QA Process Document. The QAPD template has been assembled as the union of good practices seen in real QAPDs of W3C Working Groups.

In the past, before the construction of the QAPD template, test efforts would often copy-edit a QAPD from another effort. There isn't really any reason to do this anymore. Here are some examples from which the QAPD template has been assembled:

What does it mean? In practice, it means (easiest solution) producing a QAPD recording at least those details of the WG's test- and QA-related work processes that are outlined in the QAPD template and briefly discussed in the following subsections.

3.1 General modus operandi

[TBD. This section should have lots of examples]

See the corresponding section in QAPD template.

Good Practice: Identify a Working Group point-of-contact about test materials or other QA-related business [was CP4.1]

This can be a special appointed "QA Moderator". Or it might be the WG Chair (or a co-chair), or staff contact. The important part is that there be a single identified point-of-contact to whom others can address questions, contributions, etc, and who will be accountable for responding.

See @@QAPD subsection@@.

Here are a couple @@Example(s)@@.

Good Practice: Specify an archived email list to use for QA-related communications (test suite questions, bug reports, contributions, etc). [was CP4.4]

It can be a dedicated "Test" list. Or it can be a public IG list. Or it can be a public comment list separate from the WG list, in the case that there isn't an IG. Because of the variety of public/private mixes amongst W3C's WGs, "one size fits all" does not work well here.

Here are a couple @@Example(s)@@.

See @@QAPD subsection@@.

Good Practice: Identify Web page(s) for test suites, announcements, and other QA deliverables. [was CP4.4]

Obviously, this needs to be publicly accessible. Doing it all in the public portion of the WGs Web space is one way to achieve that. In particular, that provides a good, secure repository location for test materials [was CPx.y].

Some WGs [@@Ex: SVG] have two locations actually -- a private CVS repository for the in-development test materials, and a simplified public repository for periodic public releases of test materials.

Here are a couple @@Example(s)@@.

See @@QAPD subsection@@.

The Working Group's QAPD is also a handy central location to record its licensing policies and (any) branding policies. These are sufficiently important that they have their own QAH module, which addresses:

3.2 Test development framework and processes

If the Working Group (WG) is developing test materials itself, then there are several topics associated with TM development process are going to impact its operations and management. These are described, along with_Principle_ and Good Practice guidance, in some detail in@@QA Framework: Test Guidelines@@. These topics, which might conveniently be documented in the WG's QAPD, include definitions of:

3.3 Life after WG -- maintenance

Finally, as a part of planning about "life after WG", the Working Group (WG) will need to decide what happens to both its test materials and associated processes. These maintenance-related topics are described in some detail in @@TestLite@@, but again the information might conveniently be located in the WG's QAPD:

Success criteria:

[Overview of what appears in this module:old-CP4.1(qa moderator); dropped CP4.2(appoint a task force); *** CP4.3(QAPD) is central principle***; CP4.4(email & web); CP5.4,CP6.2(license stuff) and CP4.5(branding) are recommended here to be documented in QAPD, but otherwise there is a forward reference to IANAL/license module for details; CP6.1(repository) rolled into Web site stuff as a detail; CP5.1,CP5.3,CP5.5(TM-related) are recommended here to be documented in QAPD, but otherwise @@TestLite@@ is referenced for details; CP6.3,CP6.5(TM-publication-related) are recommended here to be documented in QAPD, but otherwise @@TestLite@@ is referenced for details. CP8.1,CP8.2,CP8.3are about TM maintenance -- are recommended here to be documented in QAPD, but otherwise @@TestLite@@ is referenced for details. For reference, see the old OpsGL checklist.]

Licensing & branding

**Cartoon?**[Anyone have any good lawyer cartoons/jokes?]

[TBD. in CR? (find out from MB who this is -- e.g., Xquery? -- and what the details are] .

Why care? Get it right early, or it may stall the Working Group's Rec-track progress indefinitely. While this might seem to be a routine piece of Day-to-day operations, it has proved to be sufficiently troublesome within W3C that it deserves to be a Principle by itself.

What does this mean? There are two kinds of licensing issues: submission license, and publication license. Both of them can be problematic and can interrupt the WG's progress on Rec-track if not early addressed and carefully handled. Test materials license and other IPR issues are the subject of ongoing debate and discussion within W3C, but there are some tactics to minimize potential problems.

Good Practice: As early as possible, get WG consensus and define acceptable license terms for submission of test materials'. [wasCP5.4]

W3C currently has a more or less routine submission license, Contribution of Software or Test Materials to W3C. By early attention, it should become clear whether any potential sources require custom terms, before possible disputes can impact the WG's deliverables and schedules. Cases are known where potential submitters would not accept the standard terms, and custom terms had to be negotiated.

Good Practice: As soon as the nature of the Working Group's test materials becomes clear, get consensus and define license terms for publication of the test materials. [wasCP6.2]

Any W3C-hosted materials must have approved license and use terms. Experience has shown that there is no single license that is appropriate for all test materials, so the WG needs to address this after it has come to an understanding of the structure, content, and intended use of its test materials.

Currently approved W3C licenses that may be applied to test materials are the @@Document License@@ and the @@Software License@@. The Document license has two characteristics that are attractive for test materials:

On the other hand, there are situations in which the Document License is inappropriate, because (for example) some Test Materials require modification or completion in order to apply them.

In some cases, one license may seem appropriate for some parts of Test Materials, but the other license better for different parts. Test Materials might contain some mixture of these components: test software, test documentation, and test cases (see @@TestLite?@@). A careful look at contents and use cases may reveal that different licenses to different components is the best solution.

If the WG considers that neither the Document nor the Software License is applicable, not even piecewise as just described, it should consult with W3C Legal.

Good Practice: Consider whether to have brands, logos, or conformance icons associated with the Working Group's test materials. Define associated policies. [was CP4.5]

[TBD] Points to develop about logos policy: terms for valid application, conformance claims, challenge procedures, policing & enforcement policy, etc. Some of these start to border on significant legal questions.

[TBD] Reference to and discussion about W3C Logo and Icon Usage, and maybe also SpecLite material about Conformance Claims.

Examples:

[TBD. Should be easy to come up with a handful of good examples. Especially for brands/logos.]

[Overview of what went into this module: all the useful stuff associated with CP5.4(submission licenses) and CP6.2(publication licenses); CP4.5(branding policy) and references to W3C branding policy. See also checklist.]

Acquiring test materials

The big-three problems are:

The IPR and staff issues are similar in concept to what the Working Group faces if builds build the TM, but probably lesser in degree. A pre-transfer quality assessment might seem unique to the acquire option, but the actual steps will probably look similar to a test case contribution/review process in a build-your-own scenario (see @@QA Framework: Test Guidelines@@).

Good Practice: Do a quality assessment of candidate test materials before going any further.

Basic things that a good quality assessment might cover would for example include: clarity of scope, traceability, atomicity, user documentation, etc.

A more comprehensive list of things that an assessment process could cover might include: correctness, traceability, atomicity, user documentation, maintainer documentation, declaration of scope, completeness (vis a' vis declared scope), harnesses or interfaces for application of the TM, reconfigurability, results assessment, results recording & reporting, automation features, versioning/errata support, declaration of publication licenses, integrated submission procedures, etc."

QA Framework: Test Guidelines deals with this topic in much more detail, including (planned) templates and assessment aids.

Examples:

Good Practice: Ensure there are adequate staff to support the transferred test materials.

A Test Materials (TM) manager is still needed, but total staff resources ought to be considerably less than build-your-own scenarios. With luck, the original TM manager of the external TM source might become a WG member after the test materials are transferred.

Good Practice: Sort out IPR issues with the external party that produced the test materials.

How can I do it?

This is a virtualization of an actual transfer scenario that QA helped to moderate. It could serve as a checklist of steps to consider for Working Group's taking the acquire route.

Legend: EG the external group or entity;QAWG the QA Working Group; TM the test materials; WG the Working Group acquiring the test materials.

  1. Before the transfer, WG with the help of QAWG:
    • performs an assessment of the quality of candidate TM (by WG, QAWG)
    • identifies and commits to a set of test-related deliverables from the candidate TM. These could be: releases, appeal/challenge processes, maintenance plan, submission/review process, Web site, mail list, etc. (by WG)
    • identifies sufficient staffing, including at least a TM manager.Recommendation: recruit the TM manager from the EG (if one exists) to become a WG member after the transfer. (by WG)
    • makes the decision to seek/accept the transfer. (by WG)
    • (potentially) initiates Charter amendment (by WG, QAWG may consult), if the TM acquisition doesn't fit within the current Charter.
  2. During the transfer:
    • EG and W3C reach agreement to transfer the TM (by WG, QAWG, EG)
    • WG and EG perform the actual transfer of the TM, WG creates an initial repository (by WG, EG)
  3. After transfer, initial test development/framework process setup:
    • WG appoints a TM manager.
    • The TM manager creates a QA Process Document for WG (by WG, TM manager, QAWG may consult)
    • set up the TM home page, a TM issues mailing list (by WG, TM manager)
    • determine the appropriate W3C IPR license (by WG, QAWG)
  4. First W3C public release of the new TM:
    • make any needed enhancements prior to the first public release: fix known/reported errors, produce documentation (by WG), W3C license labelling, etc.
    • announce the first public release of TM (by TM moderator, Communications Group)
    • joint W3C/EG press release (by WG, QAWG, Communications Group, EG)
  5. After the first public release, the TM enter the maintenance phase (see@@QA Framework: Test Guidelines@@).

[Overview of what went into this module:CP7.1,CP7.2,CP7.3(transferring a suite). See alsochecklist. [tbd] Flesh it out more? per the OpsGL & OpsET content for those checkpoints.]

Appendix -- Orientation toQA Framework

Audiences for the QA Framework parts

The following list identifies the parts of the The QA Framework (QAF) that might interest those who fill the various roles within a Working Group (WG). While each part of QAF is targets itself at a specific principal audience, the various parts might have somewhat broader interest and applicability.

all WG participants

For any (potential) WG participant, the early planning and commitment parts of The QA Handbook might provide helpful context for understanding what the WG has committed to deliver. Familiarity with the Specification Guidelines will be helpful to any participant who actively participates in the advancement of the WG's specifications to Recommendation.

WG spec editors and authors

As for all WG participants,The QA Handbook might be interesting, for shedding some light on the context in which the WG is operating. A good working understanding the Principles and Good Practices of Specification Guidelines, together with its collected examples, tools, and templates, should be a valuable resource in choosing document structure, formats, and techniques that will facilitate production of a high-quality specification.

WG chair

As the person ultimately responsible for all aspects of the WG's work, a familiarity with the guidance for operations and process ofThe QA Handbook (QAH) should be helpful -- Chairs and staff contacts are the principal intended audience of QAH. Some familiarity with the advice and guidance of Specification Guidelines and Test Guidelines should be helpful as well, as the Chairs ultimately oversees both the advancement of theWG's specifications and theWG's test materials projects.

WG-TS participant

Those who are active in building the test materials of the WG should benefit from reading the guidance for test materials in Test Guidelines, and from its associated examples, techniques, and tools. Because of the close dependency of test materials on the functional specifications, a familiarity with the Specification Guidelines could be useful as well.

WG-TS moderator

The person who manages the WG's QA projects should have working understandings of guidance and techniques for specifications ofSpecification Guidelines, as well as the test materials guidelines, techniques and examples of Test Guidelines. In addition, the test-related process and logistical advice of Test Guidelines should prove useful.

non-WG spec reviewers

Whether from other WGs, or the public at large, the Specification Guidelines will be helpful to those who review a WG's specifications, by providing some objective metrics by which to measure the specifications.

non-WG TM reviewers

Whether from other WGs, or the public at large, reviewers of a WG's test materials of a WG would benefit from a familiarity with Test Guidelines. Its guidance and examples should facilitate a critical review of a WG's test materials.

Reviewers of Activity proposals & charters

For those W3C Members who will be reviewing Activity proposals and proposed WG charters, and helping to form their Advisory Committee Representative's positions, the early planning and commitment parts of The QA Handbook might be helpful in evaluating whether or not the WG's attention test and quality deliverables is appropriate and consistent with the WG's overall programme of work.

QA Activity participants

Participants in the QAWG are an expert resource for the W3C Working Groups, and accordingly should be expert on all parts of the QA Framework; participants in the QA Interest Group (QAIG) need familiarity with all parts as well, in order to effectively render some of the QAIG's chartered deliverables.

QA Framework Primer

Issue: we decided in 20040414 telecon to transfer this from QAF-Intro to an appendix. Its editing is in-progress. Simplified & edited to appropriately relate to QAF-Lite, do we reaffirm that this should be here? 20040426 -- tentatively yes. Maybe simplify to a checklist of sorts?

This chapter, in the style of a primer, introduces the use of the QA Framework document family. It progresses through significant milestones in aWG's life, from writing a Charter through publishing Recommendations, and looks at associated test suite and other quality practice activities.

First Step -- QA Commitment

Because QA is properly an integral part of the activities of each WG, each WG has to consider and commit to a set of QA deliverables appropriate to its work. A spectrum of possibilities are discussed and illustrated in the early planning and commitment module of The QA Handbook.

If a WG is being newly formed, and if the WG is able to anticipate and agree at Charter time on deliverables like test suites, then it should consider documenting those QA deliverables in its Charter, just as it does all other WG deliverables. Again, see the early planning and commitment section of The QA Handbook. A WG being re-chartered is a similar case to a newly formed WG, although the scope and direction of its work might be clearer.

For an already-chartered Working Group undertaking new QA projects, if these deliverables are not documented in the Charter already, then there are a couple of options. The W3C Process describes how to amend a Charter to accommodate significant new deliverables, if the WG wishes to take this route.

Set Up Processes and Logistics

Once the Working Group (WG) is off and running, and assuming that it has planned on some test- or other quality-related deliverables, the next step is to chose and document the processes and logistics that it will use for its QA activities. These include such typical details as:

The sections within the Day-to-Day operationsmodule of The QA Handbook give good-practice advice about how to do this, plus examples and a handy template for writing a QA Process Document.

Planning and Writing the Specification

There is a tight bond between how the specification (Recommendation) is written on the one hand, and on the other hand its testability and its suitability as a basis for interoperable implementations.

New specification. QA Framework: Specification Guidelinesshould be applied from very beginning. Among the key topics that it addresses are:

Consider the advice of QA Framework: Specification Guidelineseven at the stage of planning the structure and presentation style of the spec. Along with W3C "pubrules" and W3C Manual of Style, spec authors and editors should refer to the spec guidelines throughout their work, on topics like testable language, clarity, conciseness.

New Edition of specification. A new Edition of the same functional level of a specification is typically used for incorporation of errata (e.g.,XML 1.0 Second Edition). Normally, this should not be considered a good time to align a specification to QA Framework: Specification Guidelines -- the changes associated with such alignment could significantly disrupt and restructure the specification.

New Version of specification. A new Version of the specification refers to a significant functional change and enhancement. This presents a good opportunity to improve the testability and implementability of the specification, as just described for new specifications.

Reviewing and Progressing the Specification

This stage in the specifications life has two significant aspects:

When the specification is published in TR space, then non-WG W3C Members and the general public begin to review and comment. Such reviewers should consult and understand the material in Specification Guidelines [QAF-SPEC], in order to have an informed set of evaluation criteria for the conformance, testability, and interoperability aspects of the specification.

Working Group (WG) participants and especially WG-TSparticipants should refer to the sections (checkpoints) of the operational guidelines, regarding specification exit criteria ([QAF-OPS], checkpoint 1.5) and synchronization of the test materials with specification progression ([QAF-OPS], Guideline 3). The project enters The Matrix [MATRIX] at Last Call Working Draft (if not sooner). Additionally, a de-facto process convention is emerging, that there should be significant conformance test materials when exiting Candidate Recommendation and commencing Proposed Recommendation. This is the same timing as the explicit process requirement of two interoperable implementations.

Designing and Building Test Materials

There are several scenarios for how the Working Group (WG) "builds" its conformance test materials:

Intra-WG build. Before starting the development, the WG-TS participants should be thoroughly familiar with the material in Test Guidelines [QAF-TEST]. There is useful information for both high-level planning -- e.g., breadth-first Basic Effectivity versus fully detailed suite? -- as well as specific detail for the building of individual test cases. Another aspect of building test materials is an acceptance procedure for the individual bits, as they are built. This is addressed in the review-procedures requirements ([QAF-OPS], checkpoint 5.4) of the operational guidelines.

Import completed test materials. Several high-quality test suites have been developed outside of the relevant W3C WG, and then transferred to the WG. WGs which are considering such a transfer should refer to the test materials transfer guideline ([QAF-OPS], Guideline 7) of the operational guidelines. Clearly, the quality of the candidate test materials should be carefully assessed, and for this the Test Guidelines [QAF-TEST] can provide useful assessment criteria.

Assemble contributions. Some test suites have been built by implementing processes to assemble significant chunks of material from outside (or internal member) contributions. Operational Guidelines [QAF-OPS] addresses the steps needed to complete such a transfer -- they are the same as the preceding paragraph about transferring completed test materials. In addition, there should be careful quality assessment of contributions, for whichTest Guidelines [QAF-TEST] can be useful. Finally, there must be procedures for submission, review, and integration of contributions ([QAF-OPS], checkpoints 5.2, 5.4), which are described in several related checkpoints of the operational guidelines.

Publication of Test Materials

Typically, a Working Group TS group will want to publish releases of test materials, particularly as the specification advances through its final maturity levels (e.g., Proposed Recommendation) towards Recommendation. Test material publication is addressed in TM publication checkpoint ([QAF-OPS], checkpoint 6.3) of the operational guidelines.

As soon as the test materials become public, then there is definite need for a procedure to process challenges to correctness, make determinations, and appeal decisions. This is addressed in a test appeal checkpoint ([QAF-OPS], checkpoint 8.3) of the operational guidelines.

Publication of test materials often comprises an implicit (or explicit) invitation for contributions. The considerations described in "Assemble Contributions" are equally applicable here.

Specification Publication and Beyond

When the specification reaches Recommendation, there is typically a concurrent publication of the test materials. This might be considered a "final" publication, or ongoing development may still be planned according to one of the mechanisms discussed above. In any case, a maintenance procedure must be in place for the test materials. Firstly, there are tie-ins between approved specification errata and validity or applicability of particular tests -- mechanisms for this are discussed in Test Guidelines [QAF-TEST], and the general process overview is discussed in the errata update checkpoint ([QAF-OPS], checkpoint 8.2) of the operational guidelines. Secondly, there is the above-discussed need for both challenge/review/appeal processes. Finally, even if Working Group TS active development of test materials ceases, it may be desired to continue to consider submissions, and review and integrate them per the requirements of the test-contribution checkpoint ([QAF-OPS], checkpoint 8.1) of the operational guidelines.

Life after WG

It is possible that the Working Group (WG) and WG-TS may disband after its charter expires. This situation, and what to do about test materials, is discussed in a "secure repository" checkpoint ([QAF-OPS], checkpoint 6.1) of the operational guidelines.

Acknowledgments

[...tbd...]

References

[...tbd...]

History

[...tbd...]