The JDK Test Framework: Tag Language Specification (original) (raw)

This is a specification document, not a tutorial. For more basic information please consult the jtreg FAQ at http://openjdk.java.net/jtreg.

Contents: Leading comments and defining files ·Tag syntax · Informational tags · Declarative tags · Action tags (action options, class names as action arguments, class name wildcards,action types) · Order of tags · TestNG· Directory context ·Test groups · Source-directory structure and test suite configuration files · Defaults· Shorthands · Examples · Supported names for @requires · Test-specific system properties and environment variables · System environment variables

A particular test may involve several files of various types, but the test is specified in just one of them. Let this be called the defining file of the test.

The defining file of a test may be one of several types of files. Test tags must be enclosed in a comment at the head of the file, optionally preceded by comments which do not contain test tags. The comment in the defining file which includes the test tags is called the leading comment. The following types of defining files are recognized:

A Java source file, with file extension ".java". Only the "/*" to "*/" comment syntax is recognized. On each comment line leading asterisks, if any, are ignored.
A Bourne shell file, with file extension ".sh". The leading comment is the first contiguous set of lines each of whose first character is '#'. The comment is terminated by the first line whose first character is not '#'. Note that shell tests should not contain "#!/bin/sh" on the first line; they are intended to be run only via explicit invocation of a shell.
An HTML file, with file extension ".html". The leading comment is enclosed within the usual "" HTML tag.

The contents of a leading comment are parsed into a sequence of tokens. A token is a maximal contiguous sequence of non-whitespace characters. Whitespace characters, which include newline and carriage-return characters, are significant only insofar as they separate tokens.

Some files within the source directory structure may be configured to implicitly be defining files. For all other files, the first token in the leading comment in the defining file of a test must be "@test". The harness discovers defining files by looking for such tokens: a file without comments is not a defining file; a file whose leading comment does not start with "@test" is not a defining file.

TAG SYNTAX

Within a leading comment, a tag consists of a tag token followed by some number of arguments. A tag token is any token whose first character is '@'. There is at present no syntax for quoting a leading '@' character. Case is significant when comparing tag tokens; all tag tokens defined here are in lower case. It is an error for the leading comment of a defining file to contain any tag token not defined here.

As a special case, the token "@(#)" is not treated as a tag token. For historical reasons, this token may appear in the arguments to an @test tag as part of an SCCS identification string.

The arguments of a tag are simply the tokens that follow it. The argument tokens of a tag extend from the first token after the tag token to the end of the comment, the end of the file, or the next tag token, whichever comes first. Argument tokens are separated by whitespace; if commas are used, they will be considered to be part of the tokens to which they are adjacent.

The arguments of a tag may contain references to various properties, using the syntax${<name>}. The set of names is the same as the set that may be used in an @requiresexpression, and is defined in Appendix 1. Note: For compatibility reasons, this feature is only enabled by default when the test suite declares arequired version of jtreg 4.2 b14 and higher. The feature can be explicitly enabled or disabled by setting [allowSmartActionArgs](#allowSmartActionArgs) in theconfiguration files.

Informational tags document the test. They do not affect the operation of the harness, but may be used by the harness in generating reports.

@test *

Defining-file identifier; in general, *is a comment and is ignored.

As a special case, if the file contains multiple comments beginning with @test, if the tokens that follow@test are id=identifier, the identifier will be used to construct a unique name for the test. If there is no such identifier, a default one will be used.

@bug +

Java bug system numbers

@summary +

Textual summary

@author +

Original author. Use of this tag is now generally discouraged; information about the author and other contributions is generally better left to the source code management system for the test suite.

@comment *

A comment; ignored

Any particular informational tag, except the@comment tag, may occur at most once in a given test. The @comment tag may be used multiple times.

Declarative tags govern the execution of the tags that follow them.

@library <path|jar>+

Add one or more pathnames or jar filenames to the library path list. Each argument must use forward slash to separate file name components, and may use ".." to denote parent directories.

If an argument is relative (that is, it does not begin with '/'), it will be evaluated relative to the directory containing the test. It is an error if the resulting path is outside the test suite.
If an argument begins with '/', it will first be evaluated relative to the root directory of the test suite. It is an error if the resulting path is outside the test suite. If the result does not identify an existing directory, it will be further evaluated against each entry of a search path in turn, until an existing directory is found. The search path is specified by theexternal.lib.roots entry in the test suite configuration files.
An argument may begin with ${java.home}/ to refer to libraries bundled with the JDK being tested. An argument may begin with ${jtreg.home}/ to refer to libraries bundled with the jtreg test harness.

It is an error if any of the entries in the library path list does not identify an existing directory or jar file.

Each entry in the library path list is categorized according to its content.

packages: the contents are one or more source files arranged in a standard package hierarchy.
user modules: the contents are one or more directories, each containing the complete source code for a user module. Each directory must be named for the module it contains, and the directory must directly contain a module declaration (module-info.java).
system module patch: the contents are one or more directories, each containing source files to be compiled and used to patch modules in the system image being tested. Each directory must be named for the module to be patched, and the directory must_not_ contain a module declaration (module-info.java or_module-info.class_). The source files in each directory must be organized in a standard package hierarchy.
precompiled jar file: the entry is a jar file containing precompiled class files.

It is an error if any entry in the library path list cannot be categorized. In particular, any one entry in the library path cannot contain more than one type of content. It is highly recommended that a library directory should only contain content directly related to the library; it should not contain subdirectories containing other content, such as tests, or other separate libraries.

In general, classes in library directories are not automatically compiled as part of a compilation command explicitly naming the source files containing those classes. A test that relies upon library classes should contain appropriate @builddirectives to ensure that the classes will be compiled. It is strongly recommended that tests do not rely on the use of implicit compilation by the Java compiler. Such an approach is generally fragile, and may lead to incomplete recompilation when a test or library code has been modified.

If a library directory is used by a directory of TestNG tests, all the classes in a library directory will be automatically compiled by means of implicit@build tags.

When the harness compiles a test, the source code for each entry (if any) will be placed on the appropriate source path, and any previously compiled classes for each entry will be placed on the class path, application module path or module patch path as appropriate. When the harness runs a test, the compiled classes for each entry will be placed on the class path, application module path or module patch path as appropriate.

The @library tag may be used more than once. It may only be used before the first @run tag (either explicit or implicit.) Specifically, it must appear before any use of @run, @build, @compile, and @ignore.

The @library tag and the library path list have nothing to do with the search path (i.e., the PATH environment variable) defined in shell actions.

@key +

Label this test with the given keyword(s). Some harnesses allow tests to be selected via boolean expressions on keywords. The list of acceptable keywords may be specified in the TEST.ROOT file (see below). The @keytag may be used at most once in a given test.

@modules[/[:]]+

Express a dependence on modules in the system being tested, and optionally, on selected internal packages in some or all of those modules.

If no dependencies are specified explicitly, a default set of dependencies will be read from the modules entry in test suite configuration files.

The dependencies will be ignored if the version of JDK used to run the tests does not support the use of modules. Otherwise, a test will not be run if the system being tested does not contain all the specified modules. All the specified modules will be present in the configuration when the test is run.

A flag may be specified to indicate the desired access to a specified package.

Flag	Access
none	The package should be exported, so that its public types will be available for use by the test at compile time and run time.
open	The package should be opened, so that its types will be available for use by the test at run time using deep reflection.
+open	The package should be both exported and opened, so that its public types will be available for use by the test at compile time and run time, and other types will be available at run time using deep reflection.

@requires

Express a dependence on characteristics of the system being tested. Some harnesses allow tests to be selected according to the characteristics of the system being tested. The expression may be composed of the following elements:

Named characteristics. See Appendix 1 for a complete list of the standard supported names, and details of the test-suite configuration filesfor information on how to extend the set with additional names specific to the test-suite.
Numeric values. Values must be decimal integers, optionally followed a suffix 'K', 'M' or 'G', which is a multiplier of 1024, 1024*1024 and 1024*1024*1024 respectively. The resulting value must fit in along variable.
String values, enclosed in double quotes '"'.
Boolean values 'true' and 'false'.
Operators: ( ) * / % + - == ~= < <= > >= & | !

The @requires tag may be used multiple times in a given test. If it is used more than once, the expressions in the individual instances will be enclosed in parentheses and combined with '&'.

@enablePreview [true|false]

Declares whether the test code uses "preview features". Code that uses preview features must be compiled and executed with additional compile-time and run-time options. If a test declares that it uses preview features, these additional options will be provided automatically, for all @run main and@compile actions, including implicit@compile actions generated by @buildactions.

If the tag is present but no argument given, a default value oftrue will be assumed. If the tag is not present, and a default value has been configured with an[enablePreview](#enablePreview) entry in an enclosing TEST.properties file, then that default value will be used. If the tag is not present and no default value has been configured, a value of false will be assumed and no additional compile-time and run-time options will be provided.

There is no requirement to use @enablePreview. A test can still explicitly provide the necessary options instead.

Action tags tell the harness how to perform the test. They are executed in the order in which they are given. Each action either passes or fails. If an action fails, no following actions, if any, are performed. A test passes if, and only if, all of its actions pass.

Action tags begin with the tag token "@run", and have the following syntax:

@run

The describes the basic type of the action; the

The first token after the "@run" token contains both the action type and options, if any. To parse the type and the options, this token is broken down into subtokens. A subtoken is either the character '=', the character '/', or a maximal contiguous sequence of characters not containing either of these characters.

The type of an action tag is named by the first subtoken of the tag's first token. The remaining subtokens of the first token form the options. Options have the syntax

/<name> or /<name>=<value>

where and are a single subtokens. The may be enclosed in double quotes to prevent the usual interpretation of '=', '/', and whitespace.

ACTION OPTIONS

Not all action types support all options.

/fail

Negate the result of the action. If the action fails it is treated as though it passed, and vice versa.

/timeout=

Specify the timeout value. The default timeout is two minutes. If an action does not finish before the timeout expires, it fails. The timeout period applies to the entire action, not to particular steps of the action. A value of zero means there is no timeout. The /timeout option may not be given in conjunction with the/manual option.

/ref=

Capture the standard output and error of the action and compare it to the content of , which is in the same directory as the test. The action succeeds only if the output matches the content of the file. Even if /fail is specified, the output must match in order for the action to pass.

/othervm

Some test harnesses reuse a VM between tests. This option forces the action to be run in a fresh VM subprocess. Use this option if you need to specify VM options, or if the action might crash the VM.

/manual[=(yesno|done)]

Indicates that this is a manual action requiring user interaction. If the harness has been instructed to run only automatic actions, then this action will be skipped and will be considered to have passed. The /manual option may not be given in conjunction with the /timeout option.

If no option value is given, then the harness assumes that the test itself will handle whatever user interaction is necessary. If "yesno" is given, then the harness will ask the user whether the action is to pass or fail. If "done" is given, then the harness will wait until the user indicates that the test has completed. When the harness queries the user it does so in a manner appropriate to the action type; e.g., for applet actions it will either display "pass" or "fail" buttons or a single "done" button. In the failure case, some harnesses may provide a way for the user to submit text describing the failure.

/policy=

Define the Java system property "java.security.policy" to have the value "=${TESTSRC}/", where TESTSRC is the name of the directory containing the defining file of the test. This definition has the effect of making the given file the sole source of security policies to be used by the security manager. In particular, the system security policy and the user's security policy will be ignored. If the /secureoption is not used then the default security manager will be installed.

Property expansion will be performed in the policy file as usual; in particular, the "test.src" and "test.classes" properties will name the source and class directories of the test being run (see DIRECTORY CONTEXT below).

This option may only be used in conjunction with the/othervm option.

/secure=

Specify a subclass of java.lang.SecurityManager to be installed as the security manager. An appropriate@build tag should be provided to ensure that the class is compiled. If the /secure option is used without the/policy option then the system's built-in policy, equivalent to the original sandbox policy, will be assumed.

This option may only be used in conjunction with the/othervm option.

/native

The test uses native code, which must be provided separately, in a precompiled library, when the test is run.

CLASS NAMES AS ACTION ARGUMENTS

Some actions take one or more class names as arguments. Each such argument is the name of a class, not the name of a class file, so the ".class" suffix should not be given.

A class name may be specified in one of the following ways:

a simple class name: e.g., "Foo"
a package name followed by a simple name: e.g., "bar.Foo", "baz.bar.Foo"
a module name followed by either module-info or a fully qualified class name: e.g., "myModule/module-info", "myModule/baz.bar.Foo"

If a class in a non-Java package is specified, then the source file for that class must be in the corresponding directory relative to the directory containing the defining file of the test.

CLASS-NAME WILDCARDS

Some actions take either class names or class-name wildcards as arguments. A class-name wildcard can be specified in one of the following ways:

"*", meaning all classes in the default package
".*", meaning all classes in the named package
"/.*", meaning all classes in the named package in the named module
"/*", meaning all classes in the named module

ACTION TYPES

build +

Build classes on demand. Each argument may be a class name, as described above, or a class-name wildcard. To locate a source file for a named class, the harness takes the first matching file found by looking in the test-source directory and then in each appropriate directory of the library path list. To locate the source files denoted by a wildcard, the harness evaluates the wildcard relative to test-source directory and relative to each directory of the library path list, taking all matches. Each discovered class will be compiled if its corresponding class file does not exist or is older than its source file. Intended primarily for use before main and applet actions that require more than one class to be compiled. Passes only if the compiler finishes without error, or if none of the specified classes need to be compiled.

It is an error for a build action to be the last action in a test description.

clean +

Remove the class files for the named classes, if they exist. Each argument may be a class name or a class-name wildcard. Intended primarily for compiler tests. Always passes.

compile[/fail][/ref=][/timeout=][/process][/module=]+

Invoke the compiler on the given s, which may include any compiler option other than the "-d" option. Equivalent to "javac +", therefore to specify source files the ".java" suffix must be included. Source file names should be relative pathnames using forward slashes and ".." to denote parent directories; they are normally interpreted relative to the directory containing the defining file of the test. Standard output and standard error are concatenated (not interleaved) so that /ref may be used. Intended primarily for compiler tests; non-compiler tests should generally use build actions. The timeout period, if specified, applies to the entire compile action, not to the compilation of each individual source file. Passes only if the compiler finishes without error.

The /process option can be given to indicate that the compilation includes the use of an annotation processor and that no source files are being compiled. If /processis not given, at least one source file must be specified.

The /module=<module-name> option is used to compile a patch for a module in the system image. In this one situation, the source files must be in a subdirectory named for the module, relative to the directory containing the defining file of the test. Neither the module directory nor the list of source files may contain a module-info.javafile. Internally, this option corresponds to using the javac--patch-module=<module-name>=<path>option.

User modules can be compiled by placing the code for the modules in subdirectories named for the modules. Each directory should have a corresponding module-info.java file.No other options are required: in particular,/module=<module-name> should_not_ be used; that option is just used for patches to system modules. To avoid confusion, it is highly recommended that tests containing user modules should be placed in their own directory, and should not contain any other directories apart from the directories used to contain module source code.

The use of the -classpath and-sourcepath options as arguments to the compile action is discouraged; the @library tag is often more appropriate.

main[/fail][/manual][/othervm][/policy=][/secure=] [/timeout=] * *

Invoke the main method of the specified class, passing any arguments after the class name. Roughly equivalent to "java* *". All initial argument tokens beginning with a dash are considered to be VM options; the first argument not beginning with a dash is the to be invoked, and any remaining arguments are passed to the main method of that class. Passes only if the main method returns normally and does not cause an exception to be thrown, from the main thread or from any other thread. A "main" action is considered to be finished when the main method returns; if a test involves multiple threads, some synchronization may be necessary to ensure that the other threads finish their work before the thread running the main method returns.

The named will be compiled on demand, just as though an "@run build" action had been inserted before this action. If this action requires classes other than to be up to date, insert an appropriate build action before this action. If any s are given then /othervm should be specified. If the/manual option is specified then the harness will skip this action when instructed to run only automatic tests; no arguments to the /manual option are supported.

The use of the -classpath option as an argument to a main/othervm action is discouraged; the@library tag is often more appropriate. The setting of the system properties "java.security.manager" and "java.security.policy" in main/othervmactions is also discouraged; the /policy and/secure options should be used instead.

driver[/fail][/timeout=] *

Invoke the main method of the specified class, passing any arguments after the class name. Although superficially similar to@run main, this is for use when it is desirable to perform additional setup or configuration before running the class containing the actual test code, possibly in a child VM.

junit[/fail][/manual][/othervm][/policy=][/secure=] [/timeout=] *

Invoke the specified class as a JUnit 4.5 test. Roughly equivalent to "@run main org.junit.runner.JUnitCore<class>". All initial argument tokens beginning with a dash are considered to be VM options; the first argument not beginning with a dash is the to be invoked. No additional arguments are allowed. Passes only if the main method of JUnitCore returns normally and does not cause an exception to be thrown, from the main thread or from any other thread. As with a "main" action, a "junit" action is considered to be finished when the main method of JUnitCore returns.

The use of the @library tag, and of the/othervm, /manual, /policy, and /secure options, are the same for a "junit" action as for a "main" action; see above.

In order to use a "junit" action, the JUnit 4.5 jar file may need to be placed manually alongside the jtreg and javatest jar files in the jtreg installation. If junit.jar file is not found, and if no class named org.junit.runner.JUnitCore can be found, any "junit" actions will fail, regardless of whether the/fail option was selected.

This action executes all the tests in a single JUnit test file. All the tests in that one file will be grouped as a single jtreg test for reporting purposes.

testng[/fail][/manual][/othervm][/policy=][/secure=] [/timeout=] *

Invoke the specified class as a TestNG test. Roughly equivalent to "@run main org.testng.TestNG<class>". All initial argument tokens beginning with a dash are considered to be VM options; the first argument not beginning with a dash is the to be invoked. No additional arguments are allowed. Passes only if the main method of TestNG returns normally and does not cause an exception to be thrown, from the main thread or from any other thread. As with a "main" action, a "testng" action is considered to be finished when the main method of TestNG returns.

The use of the @library tag, and of the/othervm, /manual, /policy, and /secure options, are the same for a "testng" action as for a "main" action; see above.

In order to use a "testng" action, the TestNG jar file may need to be placed manually alongside the jtreg and javatest jar files in the jtreg installation. If testng.jar file is not found, and if no class named org.testng.TestNG can be found, any "testng" actions will fail, regardless of whether the/fail option was selected.

This action executes all the tests in a single TestNG test file. All the tests in that one file will be grouped as a single jtreg test for reporting purposes.

Note: this action allows individual tests to be written as TestNG tests, and placed alongside other tests using other combinations of actions. See below for details on how to set up a directory containing only standard TestNG tests.

applet[/fail][/manual[=(yesno|done)]][/othervm][/policy=] [/secure=][/timeout=]

Run the applet described by the first<applet> HTML tag in the given, ignoring any other<applet> tags. The applet action is roughly equivalent to "appletviewer "; the major difference is that the applet is not run in the restricted security environment implemented by the appletviewer. Intended primarily for tests of graphics functionality; tests of the appletviewer itself can be written using shell actions. Passes if the applet does not throw any exceptions from any thread and, if the /manual option is specified with a value of "yesno", the user indicates that the test passes.

The class named in the <applet> tag will be compiled on demand, just as though an "@run build" action had been inserted before this action. If this action requires classes other the class named in the <applet> tag to be up to date, insert an appropriate build action before this action.

If /manual is not specified, then the applet will be run by invoking its init, start, andsetVisible(true) methods (in that order), delaying for a few seconds, and then invoking its stop and destroy methods (in that order).

The /manual option may be specified to indicate that the test requires human interaction. In this case the HTML file itself will be displayed; it should contain any instructions for the user. When displaying the HTML file, a harness may or may not interpret HTML tags other than the <applet>tag; thus the instructions should be written in a way that does not depend upon HTML rendering.

If /manual is specified alone, i.e., without any value argument, then the applet will be run just as in the non-manual case. The harness will skip this action, and treat it as though it had passed, when instructed to run only automatic tests.

If /manual=yesno is specified, then the harness will ask the user whether the test passes or fails, typically by displaying "pass" and "fail" buttons. The applet will be run by invoking its init,start, and setVisible(true) methods (in that order), waiting for the user to click on "pass" or "fail", and then invoking the applet'sstop and destroy methods (in that order).

If /manual=done is specified, then the harness will wait for the user to indicate that the test is complete, typically by displaying a "done" button. The applet will be run by invoking its init, start, andsetVisible(true) methods (in that order), waiting for the user to click on "done", and then invoking the applet's stop and destroy methods (in that order).

The setting of the system properties "java.security.manager" and "java.security.policy" in applet/othervm actions is also discouraged; the /policy and /secureoptions should be used instead.

shell[/fail][/manual][/timeout=]