Reference Manual for Exactly

Exactly version 0.15.0

Table of Contents

Test Cases
Test Suites
Concepts
Configuration parameters
Actors
Types
- Data types
- Types involving logic
Syntax elements
Builtin symbols
- String symbols
- Path symbols
Command line syntax

Test Cases

Specification of test case functionality

Introduction

A test case is written as a plain text file:

[act]

helloworld

[assert]

exit-code == 0

stdout equals <<EOF
Hello, World!
EOF

If the file "helloworld.case" contains this test case, then Exactly can execute it:

> exactly helloworld.case
PASS

"PASS" means that Exactly was able to execute the "helloworld" program, and that all assertions were satisfied.

It also means that the executable "helloworld" file was found in in the same directory as the test case file.

File structure

[act] marks the beginning of the "act" phase.

The "act" phase contains the "action to check" - the thing that is tested by the test case.

It must consist of a single command line, starting with the name of an executable file (by default).
The file must be located in the same directory as the test case file (by default).

[assert] marks the beginning of the "assert" phase.

The "assert" phase contains assertions, such as "exit-code" and "stdout".

The assertions determines the outcome of the test case.
Each assertion either PASS or FAIL. If any assertion FAIL, then the outcome of the test case as a whole is FAIL. Otherwise it is PASS.

Structure

Phases

A test case is a sequence of "phases".
Executing a test case means executing all of its phases.

All phases are optional.

The phases are (in order of execution):

[conf]

Configures the execution of the remaining phases by setting "configuration parameters".

[setup]

Sets up the environment that the "action to check" (the "act" phase) is executed in.

[act]

Contains the "action to check"; executes it and stores the outcome for later inspection.

This is the default phase.

[before-assert]

Prepares for the "assert" phase.

[assert]

Assertions on the outcome of the "action to check" (the "act" phase) that determine the outcome of the test case.

[cleanup]

Cleans up pollution from earlier phases that exist outside of the sandbox (e.g. in a database).

The "act" phase, "action to check" and "actors"

The contents of the "act" phase represents an "action to check" (ATC), and executing the "act" phase means executing this as an OS process.

The "actor" concept makes it possible to have different kind of ATCs. E.g. executable program files, source code files and shell commands.

The actor reads the contents of the "act" phase, and thereby resolves the ATC.

A test case configures which actor to use.

The special "null" actor specifies a no-operations ATC. It is used if the test case does not contain an "act" phase.

Examples

An ATC that is a Python source file, executed with arguments:

[conf]

actor = file % python3

[act]

my-python-program.py argument1 "second argument"

An ATC that is a shell command line:

[act]

$ echo 'This is a' shell command line > &2

Instructions

All phases except [act] is a sequence of instructions, zero or more.

Executing a phase with instructions means executing all of its instructions, in the order they appear in the test case.

Instructions in different phases serve different purposes.

For example, the instruction set of the [assert] phase contains instructions that serve as assertions.

Relation to test suites

Inclusion of phase contents from suites

A test suite can contain test case contents.
When a test case is run as part of a suite, this contents is included in the case.

Part of suite

A test case is considered part of a suite, depending on how it is run.

When run in the following ways, it is part of a suite:

Standalone
- A suite file is given via command line arguments
- A file "exactly.suite" exits in the same directory as the test case file
  The file "exactly.suite" must be a test suite.
  Note that the test case need not be listed in "exactly.suite".
Via test suite
The test case file is listed in the suite.

Execution environment

Directory structure and Current directory

Files and directories used by tests are organized in the "test case directory structure" (TCDS).
It consists of two parts:

Sandbox directory structure (SDS)
A set of temporary directories used by a single execution of a test case. one of them is the initial "current directory".
Home directory structure (HDS)
A set of persistent directories for files used by every execution of a test case.

Sandbox directory structure and Current directory

The SDS is created in a platform dependent location for temporary files.

The current directory is set to one of the directories inside the SDS at the beginning of execution.
This means that files and directories created in the current directory will be removed at the end of the execution.

The current directory is the same for all phases, unless changed by the "cd" instruction.
A change of "current directory" stay in effect for all following instructions and phases.

Home directory structure

The HDS organizes files that exist before the execution and that should probably not be modified.

One of the directories in the HDS is the "act-home" directory. It is the location of the executable program file that is tested - i.e. the location of files referenced from the "act" phase.

All directories in the HDS are initialized to the directory that contains the test case file. They can be changed by the "conf" phase.

File references

Exactly has functionality for referencing files in the TCDS using the "path" type.

The "path" type has syntax for expressing paths relative to any of the TCDS directories.

The directory that a "path" value is relative to is called the "relativity" of the value.

Arguments to instructions that are paths use the "path" type.
Each argument have different accepted "relativities" and different default "relativity".
The defaults are chosen to be the most likely "relativity". The accepted "relativities" are chosen to prevent modification of files in the HDS.

Symbols

A "symbol" is a named constant defined by the "def" instruction.

Once defined, it is available to all following instructions in the phase where it is defined, and in all following phases.

Symbols may be used to define reusable values used by instructions and the "act" phase.

OS process environment

Exactly executes OS processes as part of a test case. Part of their environment is controlled by Exactly.

Current directory

The current directory (PWD) is the same as for instructions.

It is changed by the "cd" instruction.

Environment variables

Exactly maintains two sets of environment variables:

variables of the "action to check" executed by the "act" phase
variables of OS processes executed from phases other than the "act" phase

Environment variables can be manipulated by the "env" instruction.

Timeout

Timeout for all OS processes can be set by the "timeout" instruction.

File syntax

Syntax is line oriented.

Top level elements start at the beginning of a line, and line ends mark the end of elements, although some may span several lines (e.g. expressions inside parentheses).

Phases

"[NAME]" on a single line declares the start of phase NAME.
This line marks the start the "assert" phase, for example:

[assert]

The following lines will belong to this phase.

File contents before the first phase declaration belong to the default phase, which is [act].

The order of the different phases in the test case file is irrelevant. The phases are always executed in the same order, regardless of the order they appear in the test case file.

A phase can be declared more than once.
Contents of multiple declarations are merged, and executed in the order it appears in the file.
Here, "exit-code" is executed before "stderr":

[assert]

exit-code == 0

[act]

helloworld

[assert]

stderr is-empty

Phase contents

All phases except the "act" phase consist of a sequence of "instructions" (see below).

The contents of the "act" phase depends on which "actor" is used.
By default, it is expected to contain a single command line.

Instructions

Instructions start at the beginning of the line with a space separated identifier that is the name of the instruction.

The name may optionally be followed by arguments. Most instructions use a syntax for options, arguments and quoting that resembles the unix shell. The order of options is relevant, though (unlike many unix tools).
The exact syntax depends on the particular instruction.

An instruction may span several lines, as this form of "stdout" does:

stdout equals <<EOF
Hello, World!
EOF

Instruction descriptions

An instruction may optionally be preceded by an "instruction description" - a free text that is displayed together with the instruction source code in error messages.
The purpose of an instruction description is to describe the meaning of the instruction using text that is easier to understand than source code.

An instruction description is a string within backticks (`).

For example, a free text may be easier to understand than a shell command:

[assert]

`PATH should contain /usr/local/bin`

$ tr ':' '\n' < result/stdout | grep '^/usr/local/bin$'

An instruction description may span several lines.

File inclusion

Parts of a test case can be put in an external file, using the "including" directive:

including external-part-of-test-case.xly

Comments and empty lines

Lines beginning with "#" are comments.
Comments may only appear on lines between instructions and phase headers.

Empty lines that are not part of an instruction are ignored.

Empty lines, and lines with comment line syntax, may be part of instructions and the "act" phase, though, as in the "stdout" instruction here:

stdout equals <<EOF
this assertion expects 4 lines of output
# this is the second line of the expected output

the empty line above is part of the expected output
EOF

Processing steps

Test case file "processing" starts after the command line arguments have been parsed, and Exactly has been told to run a test case.

A test case file is processed in a number of steps, where the actual execution of the test is the last step.

The outcome is reported by an exit code and an exit identifier printed as a single line on stdout.

Preprocessing
Transforms the test case file.
This step is applied only if a preprocessor has been given by "--preprocessor", or set in a test suite.
Fails if the preprocessor program cannot be executed, or if it exits with a non-zero exit code.
Exit code 65
Exit identifier PRE_PROCESS_ERROR
Syntax checking and directives processing
Checks the syntax of all elements in the test case file, except for the [act] phase.
Processes directives ("including").
Fails if a syntax error is found, or if a directive fails.
- Syntax error
  Exit code 65
  Exit identifier SYNTAX_ERROR
- File inclusion error
  Exit code 65
  Exit identifier FILE_ACCESS_ERROR
Validation and syntax checking of [act]
Checks references to symbols and external resources (files),
and syntax of the [act] phase (according to the configured actor).
Fails if a reference to a symbol that has not been defined or has invalid type, is found, or if a reference to a non-existing file is found, e.g.
- Invalid syntax of [act]
  Exit code 65
  Exit identifier SYNTAX_ERROR
- Invalid reference to symbol or external resource
  Exit code 65
  Exit identifier VALIDATION_ERROR
Execution
Executes the actual test.
Executes the phases in the predefined order.
Executing a phase with instructions means executing all instructions in the order they appear in the test case file.
Execution halts if an instruction encounters an error, or, in the case of assertion instructions, if the assertion fails.

Outcome

Reporting

Outcome is reported either as an exit code, or as an exit code together with an exit identifier printed as a single line on stdout.

Scenarios

Complete execution

The outcome of a completely executed test case depends on two things:

The "test case status" set by the "conf" phase.
The default is PASS.
The outcome of the "assert" phase.

Together, these determine the outcome of the test case as a whole:

Status	[assert]	Test Case
PASS	PASS	PASS
PASS	FAIL	FAIL
FAIL	PASS	XPASS
FAIL	FAIL	XFAIL
SKIP		SKIPPED

The outcome is reported by an exit code and an exit identifier printed as a single line on stdout.

Test Case	Exit identifier	Exit code
PASS	PASS	0
FAIL	FAIL	32
XPASS	XPASS	33
XFAIL	XFAIL	33
SKIPPED	SKIPPED	0

Error during validation

If validation fails, the test case is not executed.

The outcome is reported by an exit code and an exit identifier printed as a single line on stdout.

Exit code	65
Exit identifier	VALIDATION_ERROR

Error during execution

If an error occur during the execution of a test, then this will halt the test and be reported as an error, and not as a failed test.

Note that an error will be reported even if both the "act" and "assert" phases have been executed successfully.

The outcome is reported by an exit code and an exit identifier printed as a single line on stdout.

The execution may be interrupted by the following causes:

Hard error
An instruction fails to do its job. E.g. an instruction in the "setup" phase fails to create a file.
Exit code 128
Exit identifier HARD_ERROR
Internal error
An error in the implementation of Exactly, or similar, is detected.
Exit code 129
Exit identifier INTERNAL_ERROR

Exit code	128
Exit identifier	HARD_ERROR

Exit code	129
Exit identifier	INTERNAL_ERROR

Other errors

If parsing of command line arguments fails, Exactly halts with exit code 64 (no exit identifier is printed).

Otherwise an exit identifier is printed as a single line on stdout.

The following errors may occur:

File access
Failure of accessing a file referenced via "including".
Exit code 65
Exit identifier FILE_ACCESS_ERROR
Preprocessing
Fails if the preprocessor program cannot be executed, or if it exits with a non-zero exit code.
Exit code 65
Exit identifier PRE_PROCESS_ERROR
Syntax checking
Fails if the test case contains a syntax error.
Also fails if the test case is run as part of a test suite, and the test suite contains a syntax error.
Exit code 65
Exit identifier SYNTAX_ERROR

Summary of exit codes and identifiers

Exit identifier	Exit code
FAIL	32
FILE_ACCESS_ERROR	65
HARD_ERROR	128
INTERNAL_ERROR	129
PASS	0
PRE_PROCESS_ERROR	65
SKIPPED	0
SYNTAX_ERROR	65
VALIDATION_ERROR	65
XFAIL	33
XPASS	33

Phases

[conf]

Configures the execution of the remaining phases by setting "configuration parameters".

Consists of zero or more instructions.

Accepts the "including" directive.

Each instruction sets one of the "configuration parameters".

Phase execution order

This is the first phase.

If the "status" is set to SKIP, then the remaining phases are not executed.
Otherwise, the "setup" phase follows.

Instructions

act-home	Sets the "act-home" directory
actor	Specifies the "actor" that will interpret and execute the "act" phase
home	Sets the "home" directory
status	Sets the "test case status" configuration parameter

[setup]

Sets up the environment that the "action to check" (the "act" phase) is executed in.

E.g.

populating the "sandbox directory structure" with files and directories
setting the contents of stdin
setting environment variables
setting up external resources, such as databases.

Consists of zero or more instructions.

Accepts the "including" directive.

Each instruction should probably have some side effect that affects the "action to check".

Phase execution order

If the "status" is set to SKIP, then this phase is not executed.
Otherwise:

This phase follows the "conf" phase, and is the first phase that is executed with the "sandbox directory structure".

If any of the instructions fail, then execution jumps to the "cleanup" phase, and the test case halts with an error.
Otherwise, the "act" phase follows.

Environment

Current directory
At the beginning of the phase, the "current directory" is the act directory in the "sandbox directory structure".
Environment variables
Default: All OS environment variables set when Exactly is started.
Timeout
Default: 60.

Instructions

$	Executes a command using the current operating system's shell
%	Runs a program installed on the current system (in the OS PATH)
cd	Sets the "current directory"
copy	Copies files and directories into the "sandbox directory structure"
def	Defines a symbol
dir	Creates or modifies a directory
env	Manipulates environment variables
file	Creates or modifies a regular file
run	Runs a "program"
stdin	Sets the contents of stdin for the "action to check"
timeout	Sets the timeout of individual OS processes

[act]

Contains the "action to check"; executes it and stores the outcome for later inspection.

Executes the "action to check" as an OS process.
Its outcome is stored as files in the result directory of the "sandbox directory structure":

exit code	result/exit-code
stdout	result/stdout
stderr	result/stderr

The actor configured for the test case determines what the "act" phase will execute.

Default actor is: "command line" - Executes a command line, in terms of a PROGRAM

If the "act" phase is not specified, or if its contents is empty, then the "null" actor is used.

This is the default phase.

The source of the "action to check".

The syntax depends on which "actor" is used.

The "including" directive cannot be used.

Some escape sequences exist to make it possible to have contents that would otherwise be treated as phase headers.
The escape sequences are only recognized at the first non-space characters of a line.

Source	Translation
\[	[
\\	\

Phase execution order

If the "status" is set to SKIP, then this phase is not executed.
Otherwise:

This phase is executed directly after the "setup" phase.

If the "action to check" cannot be executed, then execution jumps to the "cleanup" phase, and the test case halts with an error.
Otherwise, the "before-assert" phase follows.

Environment

Current directory
The "current directory" is the same as at the end of the previous phase.
(which is the act directory in the "sandbox directory structure", if it has not been changed.)
Environment variables
The environment variables set for the "act" phase.
The "env" instruction in the "setup" phase manipulates the environment variables of the "act" phase/"action to check".
Timeout
Timeout is inherited from the previous phase.
Stdin
The contents set by the "stdin" instruction in the "setup" phase.
If the "action to check" is a PROGRAM that defines stdin, then the stdin set in the "setup" phase is appended to the stdin of the PROGRAM.
If not set in either way, stdin is empty.

[before-assert]

Prepares for the "assert" phase.

E.g. processing files and setting "current directory", in order to make the assertions in the "assert" phase smooth.

Note that such preparations might as well be done as part of the "assert" phase.

Doing the preparations in this phase can make the "assert" phase cleaner, since the "assert" phase then can consist solely of instructions that do "real" assertions (as opposed to preparations for assertions).

The drawback is that if an instruction in this phase fails, then no assertions will have been executed and one will not know if any of the assertions would pass.

Consists of zero or more instructions.

Accepts the "including" directive.

Each instruction should probably have some side effect that supports the instructions in the "assert" phase.

Phase execution order

If the "status" is set to SKIP, then this phase is not executed.
Otherwise:

This phase is executed directly after the "act" phase.

If any of the instructions fail, then execution jumps to the "cleanup" phase, and the test case halts with an error.
Otherwise, the "assert" phase follows.

Environment

Current directory
The "current directory" is the same as at the end of the "setup" phase.
(which is the act directory in the "sandbox directory structure", if it has not been changed.)
Environment variables
Environment variables are inherited from the previous phase.
Timeout
Timeout is inherited from the previous phase.

Instructions

$	Executes a command using the current operating system's shell
%	Runs a program installed on the current system (in the OS PATH)
cd	Sets the "current directory"
copy	Copies files and directories into the "sandbox directory structure"
def	Defines a symbol
dir	Creates or modifies a directory
env	Manipulates environment variables
file	Creates or modifies a regular file
run	Runs a "program"
timeout	Sets the timeout of individual OS processes

[assert]

Assertions on the outcome of the "action to check" (the "act" phase) that determine the outcome of the test case.

If any of the instructions in this phase FAIL, then the outcome of the test case is FAIL, otherwise it is PASS.

An exception to the above is if an instruction encounters an unexpected problem that makes it neither PASS nor FAIL, in which case the outcome of the test case will indicate this unexpected problem instead of PASS or FAIL.

Consists of zero or more instructions.

Accepts the "including" directive.

Every instructions in the "assert" phase is an assertion that report its outcome as either PASS or FAIL.

If an instruction encounter a problem that prevents it from completing its work, it reports this as HARD_ERROR.

In practice, though, some instructions may have more of a purpose of "preparation" for an assertion - they are helpers for the assertions.
Such a preparation might be to sort a file that the "action to check" has produced, so that it's possible to compare it with a constant file containing the expected sorted output, e.g.

This kind of preparation can also be done in the "before-assert" phase, but it might be more suitable to put it in the "assert" phase right before the instruction that does the "true" assertion.

Assertions
Instructions that serve as assertions.
Assertions and helpers
Instructions that may serve as either an assertion or a helper for an assertion.
Helpers
Instructions that prepare things for assertion instructions, and that are not assertions in them selves.
Typically, if these instructions cannot do their job, they report HARD_ERROR, instead of FAIL.
Note that the preparation done by such an instruction can perhaps be done in [before-assert].

Phase execution order

If the "status" is set to SKIP, then this phase is not executed.
Otherwise:

This phase is executed directly after the "before-assert" phase.

This phase is followed by the "cleanup" phase.

If any of the instructions FAIL, remaining instructions in the phase will be skipped and the execution will jump directly to the "cleanup" phase.

Environment

Current directory
The "current directory" is the same as at the end of the previous phase.
(which is the act directory in the "sandbox directory structure", if it has not been changed.)
Environment variables
Environment variables are inherited from the previous phase.
Timeout
Timeout is inherited from the previous phase.

Instructions

Assertions

contents	Tests the contents of a regular file
dir-contents	Tests the contents of a directory
exists	Tests the existence, and optionally properties, of a file
exit-code	Tests the exit code from the "action to check", or from a "program"
stderr	Tests the contents of stderr from the "action to check", or from a "program"
stdout	Tests the contents of stdout from the "action to check", or from a "program"

Assertions and helpers

$	Executes a command using the current operating system's shell, and PASS iff its exit code is 0
%	Runs a program installed on the current system (in the OS PATH), and PASS iff its exit code is 0
run	Runs a "program", and PASS iff its exit code is 0

Helpers

cd	Sets the "current directory"
copy	Copies files and directories into the "sandbox directory structure"
def	Defines a symbol
dir	Creates or modifies a directory
env	Manipulates environment variables
file	Creates or modifies a regular file
timeout	Sets the timeout of individual OS processes

[cleanup]

Cleans up pollution from earlier phases that exist outside of the sandbox (e.g. in a database).

Pollution can come from an earlier phase such as [setup], or the "action to check" of the "act" phase.

Consists of zero or more instructions.

Accepts the "including" directive.

Each instruction probably has some side effect that does some cleanup.

Phase execution order

If the "status" is set to SKIP, then this phase is not executed.
Otherwise:

This phase is always executed.

If an instruction in an earlier phase encountered an unexpected error, then execution has jumped directly from that instruction to this phase.
If no unexpected error has occurred, then this phase follows the "assert" phase.

This is the last phase.

Environment

Current directory
The "current directory" is the same as at the end of the previous phase.
(which is the act directory in the "sandbox directory structure", if it has not been changed.)
Environment variables
Environment variables are inherited from the previous phase.
Timeout
Timeout is inherited from the previous phase.

Instructions

$	Executes a command using the current operating system's shell
%	Runs a program installed on the current system (in the OS PATH)
cd	Sets the "current directory"
copy	Copies files and directories into the "sandbox directory structure"
def	Defines a symbol
dir	Creates or modifies a directory
env	Manipulates environment variables
file	Creates or modifies a regular file
run	Runs a "program"
timeout	Sets the timeout of individual OS processes

Directives

including

Includes test case contents from an external file

SYNOPSIS

including FILE

where

FILE

The absolute or relative path of an existing file.

If FILE is relative, then it's relative to the directory of the current source file.

Paths uses Posix syntax.

DESCRIPTION

The effect of including a file is equivalent to having the contents of the included file in the including file; except that the current phase of the including file cannot be changed by an included file.

The default phase of the included file is the phase from which the file is included.

The included file may contain contents of different phases, by declaring different phases just as in a main test case file.
But the phase of the including file is not changed.

Instructions per phase

[conf]

The instructions in the "conf" phase.

act-home

Sets the "act-home" directory

SYNOPSIS

act-home = DIRECTORY

where

DIRECTORY

The absolute or relative path of an existing directory.

If DIRECTORY is relative, then it's relative to the location of the current source file - the file that contains the instruction.

Paths uses Posix syntax.

actor

Specifies the "actor" that will interpret and execute the "act" phase

SYNOPSIS

actor = command: Specifies that the "command line" "actor" should be used.
actor = file ACT-INTERPRETER: Specifies that the "file interpreter" "actor" should be used, with the given interpreter.
actor = source ACT-INTERPRETER: Specifies that the "source interpreter" "actor" should be used, with the given interpreter.
actor = null: Specifies that the "null" "actor" should be used.

NOTES

The "actor" specified by this instruction has precedence over all other ways to specify the actor.

ACT-INTERPRETER may reference symbols defined in the "setup" phase.

home

Sets the "home" directory

SYNOPSIS

home = DIRECTORY

where

DIRECTORY

The absolute or relative path of an existing directory.

If DIRECTORY is relative, then it's relative to the location of the current source file - the file that contains the instruction.

Paths uses Posix syntax.

status

Sets the "test case status" configuration parameter

SYNOPSIS

status = STATUS

where

STATUS

PASS

The test case is executed and the "assert" phase is expected to PASS.

FAIL

The test case is executed and the "assert" phase is expected to FAIL.

Outcome of the test case is XFAIL, if [assert] FAILs.
If [assert] PASS, the result is XPASS.

SKIP

The test case is not executed.

If stdin must be supplied, or the exit code should be ignored, then use the "run" instruction.

cd

Sets the "current directory"

SYNOPSIS

cd PATH

where

PATH

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

copy

Copies files and directories into the "sandbox directory structure"

SYNOPSIS

copy SOURCE [DESTINATION]

where

SOURCE

The PATH of an existing file or directory.

Accepted relativities (default is "home directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

DESTINATION

The PATH of an existing directory, or a non-existing path.

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

DESCRIPTION

If DESTINATION is not given
SOURCE is installed in the "current directory", as a file/directory with the basename of SOURCE.
If DESTINATION is given, but does not exist
SOURCE is copied as a file/directory with the path of DESTINATION
(the basename of SOURCE is not preserved).
Intermediate directories as created, if required.
If DESTINATION does exist
it must be a directory, and SOURCE is copied into that directory, as a file/directory with the basename of SOURCE.

NOTES

If given, DESTINATION must appear on the same line as SOURCE.

As many attributes as possible of the copied files are preserved (this depends on the Python implementation).

def

Defines a symbol

SYNOPSIS

def TYPE SYMBOL-NAME = VALUE

where

TYPE, VALUE

TYPE	VALUE
string	RICH-STRING
list	LIST
path	PATH
integer-matcher	INTEGER-MATCHER
line-matcher	LINE-MATCHER
file-matcher	FILE-MATCHER
files-matcher	FILES-MATCHER
files-condition	FILES-CONDITION
files-source	FILES-SOURCE
text-source	TEXT-SOURCE
text-matcher	TEXT-MATCHER
text-transformer	TEXT-TRANSFORMER
program	PROGRAM

PATH

Accepted relativities (default is "current directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory
-rel-cd	current directory
-rel SYMBOL-NAME	value of path symbol
-rel-here	location of the current source file

Note: When a PATH value is defined to be relative the "current directory", it means that it is relative the directory that is current when the symbol is REFERENCED,
not when it is defined.

NOTES

SYMBOL-NAME must not have been defined earlier.

The defined symbol is available in all following instructions and phases.

dir

Creates or modifies a directory

SYNOPSIS

dir PATH

Creates an empty directory.

The path must not exist.

dir PATH = FILES-SOURCE

Creates a directory with contents given by FILES-SOURCE.

The path must not exist.

dir PATH += FILES-SOURCE

Adds contents to an existing directory.

Contents is given by FILES-SOURCE.

The path must be an existing directory.

where

PATH

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

Intermediate directories as created, if required.

env

Manipulates environment variables

SYNOPSIS

env [PHASE-SPEC] NAME = VALUE

Sets the environment variable NAME to VALUE.

Elements of the form "${var_name}" in VALUE, will be replaced with the value of the environment variable "var_name", or the empty string, if there is no environment variable with that name (in the specified phase).

env [PHASE-SPEC] unset NAME

Removes the environment variable NAME.

where

PHASE-SPEC

-of act: The manipulation will only affect the environment variables of the "action to check" - the OS process executed by the "act" phase.
-of !act: The manipulation will affect the environment variables of all OS processes in all phases except for the "action to check" in the "act" phase.

NAME

STRING

VALUE

Forms:

TEXT-SOURCE

If TEXT-SOURCE involves a PROGRAM, it will be executed in an environment with the environment variables of the specified phase.

NOTES

If PHASE-SPEC is not given, the manipulation will affect all phases (and the "action to check").

file

Creates or modifies a regular file

SYNOPSIS

file PATH

Creates an empty regular file.

The path must not exist.

file PATH = TEXT-SOURCE

Creates a regular file with contents given by TEXT-SOURCE.

The path must not exist.

file PATH += TEXT-SOURCE

Appends contents to an existing regular file.

Contents is given by TEXT-SOURCE.

The path must be an existing regular file.

where

PATH

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

run

Runs a "program"

SYNOPSIS

run [-ignore-exit-code] PROGRAM

OUTCOME

The result is HARD_ERROR if the exit code is non-zero, unless -ignore-exit-code is given.

NOTES

The program must terminate.

stdin

Sets the contents of stdin for the "action to check"

SYNOPSIS

stdin = TEXT-SOURCE

NOTES

If the "action to check" is a PROGRAM that itself defines stdin, then the stdin set here is appended to the stdin defined by the PROGRAM.
(This requires that the "actor" is "command line".)

timeout

Sets the timeout of individual OS processes

SYNOPSIS

timeout = INTEGER: Sets the timeout to a number of seconds.
timeout = none: Sets no timeout.

$

Executes a command using the current operating system's shell

SYNOPSIS

$ SHELL-COMMAND-LINE

OUTCOME

The result is HARD_ERROR if SHELL-COMMAND-LINE exits with a non-zero exit code.

NOTES

If stdin must be supplied, or the exit code should be ignored, then use the "run" instruction.

%

Runs a program installed on the current system (in the OS PATH)

SYNOPSIS

% STRING [PROGRAM-ARGUMENT]...

OUTCOME

The result is HARD_ERROR if the program exits with a non-zero exit code.

NOTES

If stdin must be supplied, or the exit code should be ignored, then use the "run" instruction.

cd

Sets the "current directory"

SYNOPSIS

cd PATH

where

PATH

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory
-rel-cd	current directory

copy

Copies files and directories into the "sandbox directory structure"

SYNOPSIS

copy SOURCE [DESTINATION]

where

SOURCE

The PATH of an existing file or directory.

Accepted relativities (default is "home directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory
-rel-cd	current directory

DESTINATION

The PATH of an existing directory, or a non-existing path.

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

DESCRIPTION

If DESTINATION is not given
SOURCE is installed in the "current directory", as a file/directory with the basename of SOURCE.
If DESTINATION is given, but does not exist
SOURCE is copied as a file/directory with the path of DESTINATION
(the basename of SOURCE is not preserved).
Intermediate directories as created, if required.
If DESTINATION does exist
it must be a directory, and SOURCE is copied into that directory, as a file/directory with the basename of SOURCE.

NOTES

If given, DESTINATION must appear on the same line as SOURCE.

As many attributes as possible of the copied files are preserved (this depends on the Python implementation).

def

Defines a symbol

SYNOPSIS

def TYPE SYMBOL-NAME = VALUE

where

TYPE, VALUE

TYPE	VALUE
string	RICH-STRING
list	LIST
path	PATH
integer-matcher	INTEGER-MATCHER
line-matcher	LINE-MATCHER
file-matcher	FILE-MATCHER
files-matcher	FILES-MATCHER
files-condition	FILES-CONDITION
files-source	FILES-SOURCE
text-source	TEXT-SOURCE
text-matcher	TEXT-MATCHER
text-transformer	TEXT-TRANSFORMER
program	PROGRAM

PATH

Accepted relativities (default is "current directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory
-rel-cd	current directory
-rel SYMBOL-NAME	value of path symbol
-rel-here	location of the current source file

Note: When a PATH value is defined to be relative the "current directory", it means that it is relative the directory that is current when the symbol is REFERENCED,
not when it is defined.

NOTES

SYMBOL-NAME must not have been defined earlier.

The defined symbol is available in all following instructions and phases.

dir

Creates or modifies a directory

SYNOPSIS

dir PATH

Creates an empty directory.

The path must not exist.

dir PATH = FILES-SOURCE

Creates a directory with contents given by FILES-SOURCE.

The path must not exist.

dir PATH += FILES-SOURCE

Adds contents to an existing directory.

Contents is given by FILES-SOURCE.

The path must be an existing directory.

where

PATH

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

Intermediate directories as created, if required.

env

Manipulates environment variables

SYNOPSIS

env NAME = VALUE

Sets the environment variable NAME to VALUE.

Elements of the form "${var_name}" in VALUE, will be replaced with the value of the environment variable "var_name", or the empty string, if there is no environment variable with that name.

env unset NAME

Removes the environment variable NAME.

where

NAME

STRING

VALUE

TEXT-SOURCE

NOTES

The manipulation affects all following phases.

file

Creates or modifies a regular file

SYNOPSIS

file PATH

Creates an empty regular file.

The path must not exist.

file PATH = TEXT-SOURCE

Creates a regular file with contents given by TEXT-SOURCE.

The path must not exist.

file PATH += TEXT-SOURCE

Appends contents to an existing regular file.

Contents is given by TEXT-SOURCE.

The path must be an existing regular file.

where

PATH

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

run

Runs a "program"

SYNOPSIS

run [-ignore-exit-code] PROGRAM

OUTCOME

The result is HARD_ERROR if the exit code is non-zero, unless -ignore-exit-code is given.

NOTES

The program must terminate.

timeout

Sets the timeout of individual OS processes

SYNOPSIS

timeout = INTEGER: Sets the timeout to a number of seconds.
timeout = none: Sets no timeout.

SYNOPSIS

contents PATH : TEXT-MATCHER: Asserts that the contents of PATH satisfies TEXT-MATCHER.

where

PATH

The PATH of the file who's contents is checked.

Accepted relativities (default is "current directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

OUTCOME

The result is HARD_ERROR if PATH is not an existing regular file.

NOTES

Symbolic links are followed

dir-contents

Tests the contents of a directory

SYNOPSIS

dir-contents PATH : [DIR-CONTENTS-OPTIONS] FILES-MATCHER: Asserts that the files in the directory PATH satisfies FILES-MATCHER.

where

PATH

The PATH of the directory who's contents is checked.

Accepted relativities (default is "current directory"):

-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

DIR-CONTENTS-OPTIONS

Forms:

-recursive [-min-depth INTEGER] [-max-depth INTEGER]

Includes sub directory contents (recursively).

Sub directory contents may be limited by specifying limits for the depth.

Depth 0 is the direct contents of the tested directory.

By default, only the direct contents of the tested directory is included - sub directory contents is excluded.

OUTCOME

The result is HARD_ERROR if PATH is not an existing directory.

NOTES

The '.' and '..' directory entries are not included in the set of matched files.

Symbolic links are followed

exists

Tests the existence, and optionally properties, of a file

SYNOPSIS

exists [!] PATH [FILE-PROPERTIES]

where

!

Negates the assertion.

PATH

Accepted relativities (default is "current directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

FILE-PROPERTIES

Applies a FILE-MATCHER on PATH, if it exists.

Forms:

: FILE-MATCHER

OUTCOME

When not negated
PASS iff:
PATH exists, and has the given properties.
When negated
PASS iff:
PATH does not exist, or PATH does not have the given properties.

NOTES

Symbolic links are not followed in the test of existence (so a broken symbolic link is considered to exist).

exit-code

Tests the exit code from the "action to check", or from a "program"

SYNOPSIS

exit-code INTEGER-MATCHER

$ SHELL-COMMAND-LINE

OUTCOME

PASS iff the exit code from SHELL-COMMAND-LINE is 0.

NOTES

If stdin must be supplied, or the exit code should be ignored, then use the "run" instruction.

%

Runs a program installed on the current system (in the OS PATH), and PASS iff its exit code is 0

SYNOPSIS

% STRING [PROGRAM-ARGUMENT]...

OUTCOME

PASS iff the exit code from the program is 0.

NOTES

If stdin must be supplied, or the exit code should be ignored, then use the "run" instruction.

run

Runs a "program", and PASS iff its exit code is 0

SYNOPSIS

run [-ignore-exit-code] PROGRAM

OUTCOME

PASS if the exit code is zero, or -ignore-exit-code is given.
Otherwise FAIL.

NOTES

The program must terminate.

cd

Sets the "current directory"

SYNOPSIS

cd PATH

where

PATH

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory
-rel-cd	current directory

NOTES

In the "assert" phase, this instruction is mostly useful as a helper for writing assertions. The instruction is not an assertion on its own.

copy

Copies files and directories into the "sandbox directory structure"

SYNOPSIS

copy SOURCE [DESTINATION]

where

SOURCE

The PATH of an existing file or directory.

Accepted relativities (default is "home directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory
-rel-cd	current directory

DESTINATION

The PATH of an existing directory, or a non-existing path.

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

DESCRIPTION

If DESTINATION is not given
SOURCE is installed in the "current directory", as a file/directory with the basename of SOURCE.
If DESTINATION is given, but does not exist
SOURCE is copied as a file/directory with the path of DESTINATION
(the basename of SOURCE is not preserved).
Intermediate directories as created, if required.
If DESTINATION does exist
it must be a directory, and SOURCE is copied into that directory, as a file/directory with the basename of SOURCE.

NOTES

If given, DESTINATION must appear on the same line as SOURCE.

As many attributes as possible of the copied files are preserved (this depends on the Python implementation).

def

Defines a symbol

SYNOPSIS

def TYPE SYMBOL-NAME = VALUE

where

TYPE, VALUE

TYPE	VALUE
string	RICH-STRING
list	LIST
path	PATH
integer-matcher	INTEGER-MATCHER
line-matcher	LINE-MATCHER
file-matcher	FILE-MATCHER
files-matcher	FILES-MATCHER
files-condition	FILES-CONDITION
files-source	FILES-SOURCE
text-source	TEXT-SOURCE
text-matcher	TEXT-MATCHER
text-transformer	TEXT-TRANSFORMER
program	PROGRAM

PATH

Accepted relativities (default is "current directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory
-rel-cd	current directory
-rel SYMBOL-NAME	value of path symbol
-rel-here	location of the current source file

Note: When a PATH value is defined to be relative the "current directory", it means that it is relative the directory that is current when the symbol is REFERENCED,
not when it is defined.

NOTES

SYMBOL-NAME must not have been defined earlier.

The defined symbol is available in all following instructions and phases.

In the "assert" phase, this instruction is mostly useful as a helper for writing assertions. The instruction is not an assertion on its own.

dir

Creates or modifies a directory

SYNOPSIS

dir PATH

Creates an empty directory.

The path must not exist.

dir PATH = FILES-SOURCE

Creates a directory with contents given by FILES-SOURCE.

The path must not exist.

dir PATH += FILES-SOURCE

Adds contents to an existing directory.

Contents is given by FILES-SOURCE.

The path must be an existing directory.

where

PATH

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

Intermediate directories as created, if required.

NOTES

In the "assert" phase, this instruction is mostly useful as a helper for writing assertions. The instruction is not an assertion on its own.

env

Manipulates environment variables

SYNOPSIS

env NAME = VALUE

Sets the environment variable NAME to VALUE.

Elements of the form "${var_name}" in VALUE, will be replaced with the value of the environment variable "var_name", or the empty string, if there is no environment variable with that name.

env unset NAME

Removes the environment variable NAME.

where

NAME

STRING

VALUE

TEXT-SOURCE

NOTES

The manipulation affects all following phases.

In the "assert" phase, this instruction is mostly useful as a helper for writing assertions. The instruction is not an assertion on its own.

file

Creates or modifies a regular file

SYNOPSIS

file PATH

Creates an empty regular file.

The path must not exist.

file PATH = TEXT-SOURCE

Creates a regular file with contents given by TEXT-SOURCE.

The path must not exist.

file PATH += TEXT-SOURCE

Appends contents to an existing regular file.

Contents is given by TEXT-SOURCE.

The path must be an existing regular file.

where

PATH

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

timeout

Sets the timeout of individual OS processes

SYNOPSIS

timeout = INTEGER: Sets the timeout to a number of seconds.
timeout = none: Sets no timeout.

NOTES

In the "assert" phase, this instruction is mostly useful as a helper for writing assertions. The instruction is not an assertion on its own.

$

Executes a command using the current operating system's shell

SYNOPSIS

$ SHELL-COMMAND-LINE

OUTCOME

The result is HARD_ERROR if SHELL-COMMAND-LINE exits with a non-zero exit code.

NOTES

If stdin must be supplied, or the exit code should be ignored, then use the "run" instruction.

%

Runs a program installed on the current system (in the OS PATH)

SYNOPSIS

% STRING [PROGRAM-ARGUMENT]...

OUTCOME

The result is HARD_ERROR if the program exits with a non-zero exit code.

NOTES

If stdin must be supplied, or the exit code should be ignored, then use the "run" instruction.

cd

Sets the "current directory"

SYNOPSIS

cd PATH

where

PATH

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory
-rel-cd	current directory

copy

Copies files and directories into the "sandbox directory structure"

SYNOPSIS

copy SOURCE [DESTINATION]

where

SOURCE

The PATH of an existing file or directory.

Accepted relativities (default is "home directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory
-rel-cd	current directory

DESTINATION

The PATH of an existing directory, or a non-existing path.

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

DESCRIPTION

If DESTINATION is not given
SOURCE is installed in the "current directory", as a file/directory with the basename of SOURCE.
If DESTINATION is given, but does not exist
SOURCE is copied as a file/directory with the path of DESTINATION
(the basename of SOURCE is not preserved).
Intermediate directories as created, if required.
If DESTINATION does exist
it must be a directory, and SOURCE is copied into that directory, as a file/directory with the basename of SOURCE.

NOTES

If given, DESTINATION must appear on the same line as SOURCE.

As many attributes as possible of the copied files are preserved (this depends on the Python implementation).

def

Defines a symbol

SYNOPSIS

def TYPE SYMBOL-NAME = VALUE

where

TYPE, VALUE

TYPE	VALUE
string	RICH-STRING
list	LIST
path	PATH
integer-matcher	INTEGER-MATCHER
line-matcher	LINE-MATCHER
file-matcher	FILE-MATCHER
files-matcher	FILES-MATCHER
files-condition	FILES-CONDITION
files-source	FILES-SOURCE
text-source	TEXT-SOURCE
text-matcher	TEXT-MATCHER
text-transformer	TEXT-TRANSFORMER
program	PROGRAM

PATH

Accepted relativities (default is "current directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory
-rel-cd	current directory
-rel SYMBOL-NAME	value of path symbol
-rel-here	location of the current source file

Note: When a PATH value is defined to be relative the "current directory", it means that it is relative the directory that is current when the symbol is REFERENCED,
not when it is defined.

NOTES

SYMBOL-NAME must not have been defined earlier.

The defined symbol is available in all following instructions and phases.

dir

Creates or modifies a directory

SYNOPSIS

dir PATH

Creates an empty directory.

The path must not exist.

dir PATH = FILES-SOURCE

Creates a directory with contents given by FILES-SOURCE.

The path must not exist.

dir PATH += FILES-SOURCE

Adds contents to an existing directory.

Contents is given by FILES-SOURCE.

The path must be an existing directory.

where

PATH

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

Intermediate directories as created, if required.

env

Manipulates environment variables

SYNOPSIS

env NAME = VALUE

Sets the environment variable NAME to VALUE.

Elements of the form "${var_name}" in VALUE, will be replaced with the value of the environment variable "var_name", or the empty string, if there is no environment variable with that name.

env unset NAME

Removes the environment variable NAME.

where

NAME

STRING

VALUE

TEXT-SOURCE

NOTES

The manipulation affects all following phases.

file

Creates or modifies a regular file

SYNOPSIS

file PATH

Creates an empty regular file.

The path must not exist.

file PATH = TEXT-SOURCE

Creates a regular file with contents given by TEXT-SOURCE.

The path must not exist.

file PATH += TEXT-SOURCE

Appends contents to an existing regular file.

Contents is given by TEXT-SOURCE.

The path must be an existing regular file.

where

PATH

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

run

Runs a "program"

SYNOPSIS

run [-ignore-exit-code] PROGRAM

OUTCOME

The result is HARD_ERROR if the exit code is non-zero, unless -ignore-exit-code is given.

NOTES

The program must terminate.

timeout

Sets the timeout of individual OS processes

SYNOPSIS

timeout = INTEGER: Sets the timeout to a number of seconds.
timeout = none: Sets no timeout.

Test Suites

Specification of test suite functionality

Introduction

A test suite is written as a plain text file:

[cases]

a.case
group-dir/*.case

[suites]

sub-suite.suite
sub-dir/*.suite

If the file 'example.suite' contains this text, then Exactly can execute it:

> exactly suite example.suite
...
OK

A suite file can have any name - Exactly does not put any restriction on file names.

Structure

A suite is made up of "sections".

Some sections correspond to test case "phases".

All sections are optional.

Test cases and sub suites

A test suite contains test cases and sub suites.

These are specified in sections:

[cases]

Lists the test cases that are part of the suite.

This is the default section.

[suites]

Lists test suites (sub suites) that are part of the suite.

Common test case contents

A test suite has a section corresponding to every test case phase.

These sections specify test case contents that is common to all cases in the suite.
The contents is included in every test case in the suite.

Note that the contents is only included in test cases listed directly in the suite - not in sub suites.
The reason for this is that it should be easy to run a test case standalone, and Exactly has no functionality for looking up the ultimate root suite that a test case belongs to.

Sections:

[conf]: Common configuration, for all test cases in the suite
[setup]: Common contents of the "setup" phase, for all test cases in the suite
[act]: Common contents of the "act" phase, for all test cases in the suite
[before-assert]: Common contents of the "before-assert" phase, for all test cases in the suite
[assert]: Common contents of the "assert" phase, for all test cases in the suite
[cleanup]: Common contents of the "cleanup" phase, for all test cases in the suite

Additional test case configuration

The "conf" section has additional configuration possibilities, compared to those of the "conf" test case phase.

File syntax

Syntax is line oriented.

"[NAME]" on a single line declares the start of section "NAME".

The order of sections is irrelevant.

A section may appear any number of times. The contents of all appearances are accumulated.

Empty lines, and lines beginning with "#" are ignored, unless part of an instruction.

Outcome

Reporting

Outcome is reported via an exit code and stdout.
stderr may contain helpful information.

Output on stdout is handled by the selected "suite reporter".
The "suite reporter" also handles the exit code, in some scenarios.

Scenarios

Invalid command line arguments
Invalid argument or an argument that denotes a non-existing file.
Exit code 64
stdout empty
Invalid suite
Syntax error in a suite file, or a reference to a non-existing case or sub-suite file.
Exit code 3
stdout Depends on the selected "suite reporter"
Complete execution
The suite has been executed.
Individual test cases may have failed, or may not be executable (due to syntax error, e.g.).
Exit code Depends on the selected "suite reporter"
stdout Depends on the selected "suite reporter"

Sections

Test cases and sub suites

[cases]

Lists the test cases that are part of the suite.

This is the default section.

Each line specifies zero or more test cases to include in the suite, by giving the names of files that contain individual test cases.

File names are relative the location of the test suite file.

A test case file can have any name - Exactly does not put any restriction on file names.

Each line consists of a single file name glob pattern.
E.g.:

file.case
*.case
**/the.case
**/test-cases/*.case

[suites]

Lists test suites (sub suites) that are part of the suite.

Each line specifies zero or more test suites to include in the suite, by giving the names of files that contain individual test suites.

File names are relative the location of the test suite file.

A test suite file can have any name - Exactly does not put any restriction on file names.

Each line consists of a single file name glob pattern.
E.g.:

file.suite
*.suite
**/the.suite
**/test-suites/*.suite

Default suite file

A directory serves as a suite file, if it contains the default suite file "exactly.suite" (which must be a test suite).

Common test case contents and configuration

[conf]

Common configuration, for all test cases in the suite

Corresponds to the [conf] test case phase, with additional configuration possibilities.

The section contents is included before the contents of the "conf" phase of each test case.

Sequence of instructions - same as the [conf] test case phase.

Additional instructions

preprocessor

Sets a "preprocessor" to transform each test case file in the suite

[setup]

Common contents of the "setup" phase, for all test cases in the suite

Corresponds to the [setup] test case phase.

The section contents is included before the contents of the "setup" phase of each test case.

Identical to contents of the [setup] test case phase.

[act]

Common contents of the "act" phase, for all test cases in the suite

Corresponds to the [act] test case phase.

The section contents is included before the contents of the "act" phase of each test case.

Identical to contents of the [act] test case phase.

[before-assert]

Common contents of the "before-assert" phase, for all test cases in the suite

Corresponds to the [before-assert] test case phase.

The section contents is included before the contents of the "before-assert" phase of each test case.

Identical to contents of the [before-assert] test case phase.

[assert]

Common contents of the "assert" phase, for all test cases in the suite

Corresponds to the [assert] test case phase.

The section contents is included before the contents of the "assert" phase of each test case.

Identical to contents of the [assert] test case phase.

[cleanup]

Common contents of the "cleanup" phase, for all test cases in the suite

Corresponds to the [cleanup] test case phase.

The section contents is included after the contents of the "cleanup" phase of each test case.

Identical to contents of the [cleanup] test case phase.

junit

Outputs JUnit XML.

Output syntax

https://github.com/windyroad/JUnit-Schema/
as of 2016-11-14.

A suite with sub suites is reported as a 'testsuites' element.

A suite without sub suites is reported as a 'testsuite' element.

Exit code

Unconditionally 0.

progress

Reports execution progress in a human readable form.

This is the default "suite reporter".

Output syntax

Reports one event per line:

beginning of test suite
execution of test case
end of test suite

Beginning and end of test suites wraps the test cases that are contained directly in the test suite (i.e. it does not wrap test cases that are contained in sub suites).

Last line is an exit identifier, that depends on the outcome of the suite, and is related to the exit code.
A summary is printed on stderr.

Exit code

Exit codes, and corresponding exit identifiers printed as the last line of stdout:

Exit code	Exit identifier	When
0	OK	All test cases could be executed, and result was one of PASS, SKIPPED, XFAIL.
4	ERROR	At least one test case could not be executed, or was executed with a result other than those above: FAIL, HARD_ERROR, INTERNAL_ERROR, SYNTAX_ERROR, VALIDATION_ERROR, XPASS.
3	INVALID_SUITE	There was an error reading the test suite. No test cases have been executed.

preprocessor

Sets a "preprocessor" to transform each test case file in the suite

SYNOPSIS

preprocessor = COMMAND-LINE

DESCRIPTION

The "preprocessor" is only used for the test cases in the current suite - not in sub suites.

COMMAND-LINE is a command line (with optional arguments), using Unix shell syntax.

Concepts

action to check

The action that is executed in the "act" phase and checked by the "assert" phase.

Description

The "action to check" (ATC) is specified jointly by the configured "actor" and the contents of the "act" phase.

It is executed as an OS process.

The OS process environment is that of the "act" phase.

Depending on which "actor" is configured, the ATC may be, for example:

executable program (with arguments)
source code file
source code
shell command line

actor

Resolves the "action to check" by reading the contents of the "act" phase.

Description

The "actor" concept makes it possible to have different kind of actions executed in the "act" phase.
For example:

executable program file
source code file
source code
shell command

An actor determines the syntax and semantics of the "act" phase contents.

A test case uses a single actor.

Default actor is: "command line" - Executes a command line, in terms of a PROGRAM.

If the contents of [act] is empty, or consists of only whitespace and comment lines, then the "null" actor is used.

Other actors are configured via the "actor" configuration parameter, or the command line option "--actor".

configuration parameter

A value set in the "conf" phase that determine how the remaining phases are executed.

Description

act-home: Default location of files referenced from "act" phase, that should probably not be modified.
actor: Resolves the "action to check" by reading the contents of the "act" phase.
home: Default location of persistent helper files, that should probably not be modified.
status: The status of the test case - if it is expected to PASS or FAIL, or should be skipped.

current directory

The current directory of the environment in which instructions and OS processes are executed.

Description

During the execution of a test case, there is a "current directory" (CD).
It is initialized to the act/ sub directory of the "sandbox directory structure".

The CD is the same for all instructions and phases, unless changed by the "cd" instruction.
A change of CD stay in effect for all following instructions and phases.

Using the "current directory"

Instruction arguments of type "path" that are relative to the CD (via the "-rel-cd" option).

OS processes executed from within a test case have the CD as Present Working Directory (PWD) when the process starts.
This applies to the "act" phase (as a whole), and instructions that execute OS processes.
The "act" phase is always executed as a single OS process execution.

Change of PWD in a process do not change the CD of following instructions and phases.

"def" instruction

The "def" is special because a "symbol" definition is evaluated each time it is referenced - not when it is defined (i.e. not when the "def" instruction is executed).

Configuration parameters

Each of these directories can be set in [conf], since they are "configuration parameters".
They cannot be set after [conf].

Relative paths

PATH values have an option for referring to each of these directories.

Builtin symbols

There is a builtin PATH "symbol" for each of these directories.

Directories

act-home directory

Default location of files referenced from "act" phase, that should probably not be modified.

Default value	The directory where the test case file is located.
Configuration parameter	act-home
Set by instruction	act-home
Builtin symbol	EXACTLY_ACT_HOME
Path relativity option	-rel-act-home

home directory

Default location of persistent helper files, that should probably not be modified.

Default value	The directory where the test case file is located.
Configuration parameter	home
Set by instruction	home
Builtin symbol	EXACTLY_HOME
Path relativity option	-rel-home

instruction

The building block of all phases except [act].

Description

All phases except [act] is a sequence of "instructions", zero or more.

Executing a phase means executing all instructions in the phase, in the order they appear in the test case source file.

Each phase has its own instruction set.

Semantics

In [assert], instructions serve as assertions by representing boolean expressions.
E.g. asserting that the action to check did not output anything to stdout:

stdout empty

In all other phases, and also for some of the instructions in [assert], the purpose of an instruction is its side effects for setting up the execution environment of [act], [assert], or cleaning up after the test ([cleanup]).
E.g. creating a file:

file my-file.txt = "contents of my file"

Syntax

Forms:

PROPERTY = VALUE

For example:

timeout = 10

stdin   = "contents of stdin"

INSTRUCTION-NAME ARGUMENT...

For example:

file my-file.txt = "contents of my file"

run % python3 -c :> import sys; sys.stdout.write("Hello world")

An instruction starts on a new line, beginning with the name of the instruction, followed by arguments.

Each instruction has its own syntax for arguments.
Most common syntax is that of options and arguments resembling the Unix shell command line interface.
One difference though, is that the order of options is usually significant.

Some instructions may span multiple lines, while some must use only a single line.
The syntax is not always consistent.

Most arguments are values of one of Exactly's type system, and thus use a common syntax.

Arguments may refer to symbols defined by the "def" instruction.

Comments may not appear inside instructions.

preprocessor

A command line that transforms a test case file as the first step of processing it

Description

A "preprocessor" is a command line (with optional arguments), using Unix shell syntax.

When executed, it is given a single (additional) argument: the path of the test case file to transform.

The result of the transformation is the output on stdout.

An exit code other than 0 indicates error.

A test case file is preprocessed only if a "preprocessor" is given via the "--preprocessor" option, or via a test suite.

sandbox directory structure

A set of temporary directories used by a single execution of a test case. One of them is the initial "current directory".

Description

Each execution of a test case uses its own "sandbox directory structure" (SDS).

It is created in a platform dependent location for temporary files.

The SDS is created just before the "setup" phase is executed (after validation) and deleted when test case execution ends (unless "--keep" is used).

The SDS is used to store temporary files used by instructions (among other things). One such example is output from external processes - stdout and stderr. Thus, the SDS must have enough space to store the output from all such executed processes.

Directory structure

act/
tmp/
result/
internal/
- tmp/
- log/

act/

The current directory when [setup] begins.

It will be the "current directory" for the "act" phase, and following phases, unless it is changed by the "cd" instruction.

(Files and directories that [setup] creates are installed into the "current directory", if no instruction options are used to change this.)

EXACTLY_ACT	The value of this "symbol" is the absolute path of this directory.
-rel-act	The path relativity option that refers to this directory.

result/

Stores OS process outcome of [act], so that [assert] may check it.

This directory is initially empty.
It is populated when the "act" phase is executed with the following files (with obvious contents):

stderr
stdout
exit-code

EXACTLY_RESULT	The value of this "symbol" is the absolute path of this directory (after the "act" phase has been executed).
-rel-result	The path relativity option that refers to this directory.

tmp/

Reserved for custom helper files, used by the test case implementation.

Exactly does not touch this directory.
The test case can use it as a place where it is safe to put temporary files without the risk of name clashes with files from other program.

EXACTLY_TMP	The value of this "symbol" is the absolute path of this directory.
-rel-tmp	The path relativity option that refers to this directory.

Every symbol has a type according to Exactly's type system.

The name of a symbol must be unique within the whole test case - symbol names share a global name space.

A symbol name (SYMBOL-NAME) is: A combination of alphanumeric characters and underscores.

Definition

A symbol is defined by the "def" instruction:

def string SYMBOL_NAME = "the symbol value"

This defines a symbol SYMBOL_NAME to be the value "the symbol value", which is a value of type string.

The type must be given explicitly.
For example:

def list             LIST_SYMBOL = first second "the third"

def text-transformer REPLACE_ID  = replace [0-9]{10} ID

Reference

A symbol must be defined before it is referenced.

Once defined, a symbol is available to all instructions following the definition - both in the phase where it is defined and in following phases.

A symbol reference may appear in strings and as instruction arguments, in places where an argument of a certain type is expected.
Thus, symbols cannot be used for instruction names or arbitrary instruction arguments (as in unix shell scripts, e.g.).

There are two forms of symbol references:

Plain name	SYMBOL-NAME
Special syntax	@[SYMBOL-NAME]@

The special syntax is designed to reduce the risk of clashes with the syntax of other programming languages and shell script, e.g.
This is important since Exactly has limited capabilities for escaping and quoting.

The special syntax must be used wherever a string value is accepted.

Also, strings that resemble the special syntax will not be considered as errors, since it is assumed that these strings might have the syntax needed for a different purpose.
For example, the following strings does not constitute "symbol" references, and Exactly will not complain:

@[NOT/A_VALID_SYMBOL_NAME]@
@[VALID_SYMBOL_NAME ]@
@[VALID_SYMBOL_NAME]

The plain "symbol" name is used when a "symbol" reference is unambiguous. Otherwise, the special syntax is needed.
For example:

def string S = "reference to @[SYMBOL_NAME]@"

def list   L = first @[STRING_SYMBOL]@ "third element"

stdout -transformed-by ( filter LINE_MATCHER_SYMBOL ) ! is-empty

stdout equals -contents-of -rel PATH_SYMBOL @[DIR_NAME_SYMBOL]@/@[BASE_NAME_SYMBOL]@

test case directory structure

Persistent and temporary directories used in the execution of a test case.

Description

Consists of two sets of directories:

Home directory structure

A set of persistent directories for files used by every execution of a test case.

act-home directory	Default location of files referenced from "act" phase, that should probably not be modified.
home directory	Default location of persistent helper files, that should probably not be modified.

Sandbox directory structure

A set of temporary directories used by a single execution of a test case. One of them is the initial "current directory".

act directory	The current directory when [setup] begins
result directory	Stores OS process outcome of [act], so that [assert] may check it
tmp directory	Reserved for custom helper files, used by the test case implementation

Exactly has support for referring to all of these directories from a test case via the "path" type.

timeout

Timeout of OS processes executed from within a test case.

Description

The timeout is per OS process. It does not apply to the test case as a whole.

It can be changed by the "timeout" instruction.

A change applies to all following instructions and phases.

Default

type

Type system for symbols and instruction arguments.

Description

Exactly has a type system specialized for test cases.

Every symbol, and most instruction arguments, have a type.

There are two categories of types:

Data types
These types represent pure data, and can be converted to strings.
Types involving logic
These types involve data together with logic that is executed.
Most of these types have syntax that resembles common syntax of expressions, with primitives and operators.

Each type has its own specialized syntax.

Every value is constant.

Summary of types

Data types
list A sequence of zero or more string elements
path A file path, with special support for directories in the "test case directory structure"
string A sequence of characters

Types involving logic

file-matcher	Matches properties of an existing file - type, name and contents
files-condition	A condition of existence of a set of named files
files-matcher	Matches a set of files (e.g. the contents of a directory)
files-source	Produces a set of files (for populating a directory)
integer-matcher	Matches an integer
line-matcher	Matches individual lines of a text
program	An external program, with optional arguments, and optional transformation of the output
text-matcher	Matches a text
text-source	Produces a text, from various sources
text-transformer	Transforms a text

Configuration parameters

act-home

Default location of files referenced from "act" phase, that should probably not be modified.

Default Value

The directory where the test case file is located.

Description

Instructions and phases may use files that are supposed to exist before the test case is executed.

E.g., the "act" phase (by default) references an program that is expected to be an executable file.
If the path to this file is relative, then it is relative the "act-home".

The symbol EXACTLY_ACT_HOME is the absolute path of this directory.

The option "-rel-act-home" (accepted by many instructions) refers to this directory.

actor

Resolves the "action to check" by reading the contents of the "act" phase.

Default Value

"command line" - Executes a command line, in terms of a PROGRAM

home

Default location of persistent helper files, that should probably not be modified.

Default Value

The directory where the test case file is located.

Description

Instructions and phases may use predefined input in terms of files that are supposed to exist before the test case is executed.

Many instructions use exiting files. E.g. for copying them into the sandbox.

The symbol EXACTLY_HOME is the absolute path of this directory.

The option "-rel-home" (accepted by many instructions) refers to this directory.

status

The status of the test case - if it is expected to PASS or FAIL, or should be skipped.

Default Value

PASS

Description

PASS

The test case is executed and the "assert" phase is expected to PASS.

FAIL

The test case is executed and the "assert" phase is expected to FAIL.

Outcome of the test case is XFAIL, if [assert] FAILs.
If [assert] PASS, the result is XPASS.

SKIP

The test case is not executed.

Outcome of the test case is SKIPPED.

Actors

command line

Executes a command line, in terms of a PROGRAM

This is the default "actor".

"act" phase contents

A single PROGRAM element.

Any number of empty lines and comment lines are allowed.

Syntax of "act" phase contents

Comment lines are lines beginning with "#" (optionally preceded by space).

SYNOPSIS

PROGRAM

Notes

Path relativities

When PROGRAM has the form of the PATH of an executable file:

Accepted relativities (default is "act-home directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

Stdin

If PROGRAM defines stdin, then the stdin of the OS process is the stdin defined for PROGRAM followed by the stdin set in the "setup" phase.

Transformations

If PROGRAM includes a transformation (via a TEXT-TRANSFORMER), then the transformation is applied to stdout of the OS process.

file interpreter

Executes a source code file using an ACT-INTERPRETER

"act" phase contents

A single line which is a file name followed by optional arguments.

The file name and arguments are given as arguments to the ACT-INTERPRETER.

Syntax of "act" phase contents

Comment lines are lines beginning with "#" (optionally preceded by space).

SYNOPSIS

FILE [PROGRAM-ARGUMENT]...

where

FILE

The path of an existing source code file.

Accepted relativities (default is "act-home directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

null

Ignores the contents of the [act] phase. Exit code is unconditionally 0, and there is no output on neither stdout nor stderr

The null actor is useful when the test case does not test a program, but rather properties of the operating system environment.

"act" phase contents

Ignored.

Syntax of "act" phase contents

There are no syntax requirements.

source interpreter

Treats the [act] phase as source code to be executed by an ACT-INTERPRETER

The contents of the "act" phase is stored in a file, and the name of this file is given as the last argument to the interpreter.

"act" phase contents

Source code to be interpreted by the interpreter.

Syntax of "act" phase contents

All lines of the "act" phase are part of the source code.
There is no recognition or special handling of comment lines and empty lines.

Types

Data types

list

A sequence of zero or more string elements

Description

Used for argument lists, etc.

Useful when the "action to check" is a program with arguments, and for the "run" instruction, e.g.

path

A file path, with special support for directories in the "test case directory structure"

Description

Rendering of path values

All path values are rendered as absolute paths.

This means, e.g., that values can be passed to external programs, which may use them without consideration of the current directory of the OS process.

Note: A path value that is defined using the "def" instruction with relativity of "current directory" ("-rel-cd") is evaluated when it is REFERENCED, not when it is defined.

This means that the rendered value is different if it is referenced in different locations with different "current directory".

string

A sequence of characters

Description

Used for arbitrary string values, relative file names etc.

file-matcher

Matches properties of an existing file - type, name and contents

files-condition

A condition of existence of a set of named files

files-matcher

Matches a set of files (e.g. the contents of a directory)

files-source

Produces a set of files (for populating a directory)

integer-matcher

Matches an integer

line-matcher

Matches individual lines of a text

Description

A line is represented by its

line number
text contents

Line numbers start at 1.

The line separator is not included in the text contents.

The line separator depends on the current OS ('\n', '\r\n', e.g.).

program

An external program, with optional arguments, and optional transformation of the output

Description

A "program" is executed as an OS process.

The environment in which a program is executed is described in "Specification of test case functionality" / "OS process environment".

text-matcher

Matches a text

text-source

Produces a text, from various sources

Description

Produces a text, when referenced.

The produced text may differ when a "text-source" is referenced from different locations.
One such example is a "text-source" that is the output from a "program" that produces different output on different executions.

text-transformer

Transforms a text

Syntax elements

Data types

LIST

A sequence of zero or more string elements

[STRING|SYMBOL-REFERENCE]...

where

SYMBOL-REFERENCE

A reference to a "symbol" defined as either "string", "path" or "list".

If it is a list, it is concatenated with the surrounding elements - lists cannot be nested.

Description

A list continues until END-OF-LINE or an unquoted ")".

An unquoted "\" at END-OF-LINE makes the list continue on the next line.

PATH

A file path, with special support for directories in the "test case directory structure"

[RELATIVITY] FILE-NAME

where

FILE-NAME

Forms:

STRING

A relative or absolute path, using Posix syntax.

Note: The following strings are reserved words: "(", ")", "[", "]", "{", "}", "=", "|", ":", "!", "&&", "||".
To use any of them as a file name, it must be quoted.

RELATIVITY

An option that specifies the directory that FILE-NAME is relative to.

The available relativities depends on the instruction, and also on the argument's role in the instruction.

Summary of options:

Options for directories in the "home directory structure"
-rel-home home directory
-rel-act-home act-home directory
Options for directories in the "sandbox directory structure"
-rel-act act directory
-rel-tmp tmp directory
-rel-result result directory
Option for "current directory"
-rel-cd
This options is needed since many instructions have a default relativity other than the "current directory".
Option for a path denoted by a "symbol"
-rel SYMBOL-NAME
This option is always available.
Option for the location of the current source file
-rel-here
This option is only available when defining a path "symbol" using the "def" instruction.

SYMBOL-NAME

A "symbol" with type "path".

Description

Relativity

If FILE-NAME is an absolute path, then RELATIVITY must not be given.

If FILE-NAME is a relative path, then if RELATIVITY is ...

given
FILE-NAME is relative to the directory specified by RELATIVITY
not given
FILE-NAME is relative to the default relativity root directory,

The default relativity depends on the instruction, and also on the argument's role in the instruction.

Notes

If FILE-NAME begins with a reference to a "path" "symbol", then it is an absolute path.

For example:

def path MY_PATH_SYMBOL = ...

... @[MY_PATH_SYMBOL]@/a/path/relative/to/MY_PATH_SYMBOL

Rendering of path values

All path values are rendered as absolute paths.

This means, e.g., that values can be passed to external programs, which may use them without consideration of the current directory of the OS process.

Note: A path value that is defined using the "def" instruction with relativity of "current directory" ("-rel-cd") is evaluated when it is REFERENCED, not when it is defined.

This means that the rendered value is different if it is referenced in different locations with different "current directory".

STRING

A sequence of characters

Forms:

CHARACTER...

A "naked" sequence of characters.

CHARACTER may not be whitespace.

Any SYMBOL-REFERENCE appearing in the string is substituted.

"CHARACTER..."

Characters surrounded by "soft quotes" (").

Any SYMBOL-REFERENCE appearing in the string is substituted.

'CHARACTER...'

Characters surrounded by "hard quotes" (').

Any SYMBOL-REFERENCE appearing in the string is NOT substituted.

where

SYMBOL-REFERENCE: A reference to a "symbol" defined as either "string", "path" or "list".

Description

Concatenation

String fragments of the different forms may be concatenated by putting them side by side (without intervening whitespace).

Reserved words

The following strings are reserved words: "(", ")", "[", "]", "{", "}", "=", "|", ":", "!", "&&", "||".

To use any of them as a string, it must be quoted.

Conversion of "path" and "list"

References to these types can be converted to strings, either by putting the SYMBOL-REFERENCE inside soft quotes, or by using a naked SYMBOL-REFERENCE (in most places).

An empty "list" is rendered as an empty string.

A non-empty "list" is rendered by separating the elements with a single space.

A "path" is rendered as an absolute path (even if relativity is -rel-cd).

FILE-MATCHER

Matches properties of an existing file - type, name and contents

Forms:

type TYPE

Matches files who's type is TYPE.

TYPE is one of:

dir	File must be a directory
file	File must be a regular file
symlink	File must be a symbolic link

Symbolic links are followed (unless TYPE is symlink).

contents TEXT-MATCHER

Matches regular files who's contents satisfies TEXT-MATCHER.

The result is HARD_ERROR for files are not regular files.
Symbolic links are followed.

Note: TEXT-MATCHER may not contain infix operators (unless inside parentheses).

dir-contents [DIR-CONTENTS-OPTIONS] FILES-MATCHER

Matches directories who's contents satisfies FILES-MATCHER.

The '.' and '..' directory entries are not included in the set of matched files.

The result is HARD_ERROR for files are not directories.
Symbolic links are followed.

Note: FILES-MATCHER may not contain infix operators (unless inside parentheses).

run [PATH-ARGUMENT-POSITION] PROGRAM

Runs a program, and matches iff its exit code is 0.

The path of the file to match is given as an argument according to PATH-ARGUMENT-POSITION, if given, or as the last argument, if PATH-ARGUMENT-POSITION is not given.

Note: Transformations of the output associated with PROGRAM are ignored.

path (GLOB-PATTERN|~ REGEX)

Matches files who's absolute path matches the given pattern.

name (GLOB-PATTERN|~ REGEX)

Matches files who's final path component (base name) matches the given pattern.

stem (GLOB-PATTERN|~ REGEX)

Matches files who's "stem" part of its name matches the given pattern.

E.g., the stem of 'a.tar.gz' is 'a'.

suffixes (GLOB-PATTERN|~ REGEX)

Matches files who's "suffixes" part of its name matches the given pattern.

E.g., the suffixes of 'a.tar.gz' is '.tar.gz'.

suffix (GLOB-PATTERN|~ REGEX)

Matches files who's "suffix" part of its name matches the given pattern.

E.g., the suffix of 'a.tar.gz' is '.gz'.

constant (false|true)

Unconditionally false or true.

! FILE-MATCHER

Matches files not matched by the given matcher.

FILE-MATCHER || FILE-MATCHER

Matches files matched by any matcher.

FILE-MATCHER && FILE-MATCHER

Matches files matched by every matcher.

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a file-matcher.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a file-matcher.

An unquoted string that is not a reserved word is interpreted as the name of a symbol.

( FILE-MATCHER )

where

DIR-CONTENTS-OPTIONS

Forms:

-recursive [-min-depth INTEGER] [-max-depth INTEGER]

Includes sub directory contents (recursively).

Sub directory contents may be limited by specifying limits for the depth.

Depth 0 is the direct contents of the tested directory.

By default, only the direct contents of the tested directory is included - sub directory contents is excluded.

PATH-ARGUMENT-POSITION

Specifies which argument(s) to the program that will be the path of the file.

Forms:

-path-arg-last: The path of the file will be the last argument to PROGRAM.
-path-arg-marker MARKER: The path of the file will replace every argument to PROGRAM that equals MARKER.

Description

File name parts

path	name	stem	suffixes	suffix
/dir/a.tar.gz	a.tar.gz	a	.tar.gz	.gz
/dir/f.txt	f.txt	f	.txt	.txt
/dir/f	f	f
/dir/f.	f.	f	.	.
/dir/.x.y	.x.y		.x.y	.y

Notes

Operator precedence

Evaluation

||, &&: Operands are evaluated lazily, from left to right.

Syntax

Operators and parentheses must be separated by whitespace.

FILES-CONDITION

A condition of existence of a set of named files

Forms:

LITERAL

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a files-condition.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a files-condition.

An unquoted string that is not a reserved word is interpreted as the name of a symbol.

( FILES-CONDITION )

where

LITERAL

A sequence of file names. Each together with an optional FILE-MATCHER.

Forms:

{ FILE-CONDITION... }

There can be only one FILE-CONDITION per line.

All parts must be separated by whitespace.

FILE-CONDITION

A condition of existence of a file with a given name.

Forms:

FILE-NAME

FILE-NAME : FILE-MATCHER

A condition that includes matching of the named file by FILE-MATCHER.

Multiple FILE-MATCHERs may be associated with a single file name (via multiple FILE-CONDITIONs).
In this case, the matchers are combined using &&, in order of appearance.

The ":" before FILE-MATCHER must appear on the same line as the file name.

All parts must be separated by whitespace.

FILE-NAME

Forms:

STRING

A relative path, using Posix syntax.

Notes

Syntax

Parentheses must be separated by whitespace.

FILES-MATCHER

Matches a set of files (e.g. the contents of a directory)

Forms:

is-empty

Matches iff the set of files is empty.

matches [-full] FILES-CONDITION

Matches iff the set of files contains every file in FILES-CONDITION.

If -full is given, the set of files must contain no other files than those in FILES-CONDITION.

every file : FILE-MATCHER

Matches iff every file satisfies FILE-MATCHER.

Note: FILE-MATCHER may not contain infix operators (unless inside parentheses).

any file : FILE-MATCHER

Matches iff any file satisfies FILE-MATCHER.

Note: FILE-MATCHER may not contain infix operators (unless inside parentheses).

num-files INTEGER-MATCHER

Matches iff the number of files satisfies INTEGER-MATCHER.

Note: INTEGER-MATCHER may not contain infix operators (unless inside parentheses).

-selection FILE-MATCHER FILES-MATCHER

Applies FILES-MATCHER to the sub set of files matched by FILE-MATCHER.

Note: If combined with -with-pruned - pruning is done before -selection, regardless of their mutual order.

Note: FILE-MATCHER and FILES-MATCHER may not contain infix operators (unless inside parentheses).

-with-pruned FILE-MATCHER FILES-MATCHER

Excludes contents of directories matched by FILE-MATCHER.

Symbolic links are followed.

Note: FILE-MATCHER is only applied to directories (and symbolic links to directories).

Note: FILE-MATCHER and FILES-MATCHER may not contain infix operators (unless inside parentheses).

constant (false|true)

Unconditionally false or true.

! FILES-MATCHER

Matches sets of files not matched by the given matcher.

FILES-MATCHER || FILES-MATCHER

Matches sets of files matched by any matcher.

FILES-MATCHER && FILES-MATCHER

Matches sets of files matched by every matcher.

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a files-matcher.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a files-matcher.

An unquoted string that is not a reserved word is interpreted as the name of a symbol.

( FILES-MATCHER )

Notes

Operator precedence

Evaluation

||, &&: Operands are evaluated lazily, from left to right.

Syntax

Operators and parentheses must be separated by whitespace.

FILES-SOURCE

Produces a set of files (for populating a directory)

Forms:

FILE-LIST

A sequence of files, created or modified, relative the populated directory.

dir-contents-of PATH

A copy of the contents of PATH (recursive).

PATH must be an existing directory.

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a files-source.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a files-source.

An unquoted string that is not a reserved word is interpreted as the name of a symbol.

( FILES-SOURCE )

where

FILE-LIST

{ FILE-SPEC... }

There can be only one FILE-SPEC per line.

Files are created/modified in the order listed.

All parts must be separated by whitespace.

FILE-SPEC

Creation or modification of a named file, relative the populated directory.

If given, "=" and "+=" must appear on the same line as FILE-NAME.

Forms:

file FILE-NAME

Creates an empty regular file.

The path must not exist.

file FILE-NAME = TEXT-SOURCE

Creates a regular file with contents given by TEXT-SOURCE.

The path must not exist.

file FILE-NAME += TEXT-SOURCE

Appends contents to an existing regular file.

Contents is given by TEXT-SOURCE.

The path must be an existing regular file.

dir FILE-NAME

Creates an empty directory.

The path must not exist.

dir FILE-NAME = FILES-SOURCE

Creates a directory with contents given by FILES-SOURCE.

The path must not exist.

dir FILE-NAME += FILES-SOURCE

Adds contents to an existing directory.

Contents is given by FILES-SOURCE.

The path must be an existing directory.

FILE-NAME

Forms:

STRING

A relative path, using Posix syntax.

The path is relative the populated directory.

Must not contain "..".

Intermediate directories as created, if required.

Notes

Syntax

Parentheses must be separated by whitespace.

Additional reserved words:

dir
file

INTEGER-MATCHER

Matches an integer

Forms:

== INTEGER

Matches integers equal to INTEGER.

!= INTEGER

Matches integers not equal to INTEGER.

< INTEGER

Matches integers less than INTEGER.

<= INTEGER

Matches integers less than or equal to INTEGER.

> INTEGER

Matches integers greater than INTEGER.

>= INTEGER

Matches integers greater than or equal to INTEGER.

constant (false|true)

Unconditionally false or true.

! INTEGER-MATCHER

Matches integers not matched by the given matcher.

INTEGER-MATCHER || INTEGER-MATCHER

Matches integers matched by any matcher.

INTEGER-MATCHER && INTEGER-MATCHER

Matches integers matched by every matcher.

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as an integer-matcher.

SYMBOL-NAME

Reference to a symbol, that must have been defined as an integer-matcher.

An unquoted string that is not a reserved word is interpreted as the name of a symbol.

( INTEGER-MATCHER )

Notes

Operator precedence

Evaluation

||, &&: Operands are evaluated lazily, from left to right.

Syntax

Operators and parentheses must be separated by whitespace.

LINE-MATCHER

Matches individual lines of a text

Forms:

contents TEXT-MATCHER

Matches lines who's text contents matches TEXT-MATCHER.

line-num INTEGER-MATCHER

Matches lines who's line number matches INTEGER-MATCHER.

Note: Line numbers start at 1.

Note: INTEGER-MATCHER may not contain infix operators (unless inside parentheses).

constant (false|true)

Unconditionally false or true.

! LINE-MATCHER

Matches lines not matched by the given matcher.

LINE-MATCHER || LINE-MATCHER

Matches lines matched by any matcher.

LINE-MATCHER && LINE-MATCHER

Matches lines matched by every matcher.

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a line-matcher.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a line-matcher.

An unquoted string that is not a reserved word is interpreted as the name of a symbol.

( LINE-MATCHER )

Notes

Operator precedence

Evaluation

||, &&: Operands are evaluated lazily, from left to right.

Syntax

Operators and parentheses must be separated by whitespace.

PROGRAM

An external program, with optional arguments, and optional transformation of the output

Forms:

PROGRAM'
( PROGRAM' )

where

PROGRAM'

PGM-AND-ARGS [STDIN] [TRANSFORMATION-OF-OUTPUT]

PGM-AND-ARGS

A program followed by arguments until END-OF-LINE.

Forms:

PGM-FOR-ARG-LIST [PROGRAM-ARGUMENT]...

A program with a list of arguments.

An argument list continues until END-OF-LINE or an unquoted ")".

An unquoted "\" at END-OF-LINE makes the argument list continue on the next line.

$ SHELL-COMMAND-LINE

A shell command.

SHELL-COMMAND-LINE is the remaining part of the current line.
It is passed as a single string to the operating system's shell.

PGM-FOR-ARG-LIST

An executable program.

Forms:

PATH

The PATH of an executable file.

Accepted relativities (default is "home directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

Note: The relativities are different for the "command line" "actor".

% STRING

A program installed on the current OS - a program in the OS PATH.

@ SYMBOL-NAME

Reference to a program that has been defined using the "def" instruction.

Arguments, stdin and transformations are appended to the arguments, stdin and transformations of the referenced program.

-python

The Python interpreter (Python >= 3.6).

Since Exactly is written in Python, the Python interpreter is guaranteed to be available on the system.

STDIN

Forms:

-stdin TEXT-SOURCE

Supplies stdin to the program.

If the program is a reference to a PROGRAM symbol, this stdin is appended to the stdin defined for the referenced program.

Note: Must appear on a separate line.
(If not, it will be interpreted as arguments.)

TRANSFORMATION-OF-OUTPUT

Forms:

-transformed-by TEXT-TRANSFORMER

Transforms the output from the program.

Depending on the context, either stdout or stderr is transformed.

Note: Must appear on a separate line.
(If not, it will be interpreted as arguments.)

Note: TEXT-TRANSFORMER may not contain infix operators (unless inside parentheses).

TEXT-MATCHER

Matches a text

Forms:

is-empty

Matches iff the text is empty.

equals TEXT-SOURCE

Matches iff the text is equal to TEXT-SOURCE.

== TEXT-SOURCE

An alias for "equals".

matches [-full] REGEX

Matches iff REGEX matches any part of the text.

If -full is given, then REGEX must match the full text.

~ [-full] REGEX

An alias for "matches".

every line : LINE-MATCHER

Matches iff every line satisfies LINE-MATCHER.

Note: LINE-MATCHER may not contain infix operators (unless inside parentheses).

any line : LINE-MATCHER

Matches iff any line satisfies LINE-MATCHER.

Note: LINE-MATCHER may not contain infix operators (unless inside parentheses).

num-lines INTEGER-MATCHER

Matches iff the number of lines of the text matches INTEGER-MATCHER.

Note: INTEGER-MATCHER may not contain infix operators (unless inside parentheses).

run PROGRAM

Runs a program, and matches iff its exit code is 0.

The text to match is given as stdin.
If PROGRAM defines stdin, then the text to match is appended to that stdin.

Note: Transformations of the output associated with PROGRAM are ignored.

-transformed-by TEXT-TRANSFORMER TEXT-MATCHER

Applies TEXT-MATCHER to the original text transformed by TEXT-TRANSFORMER.

Note: TEXT-TRANSFORMER and TEXT-MATCHER may not contain infix operators (unless inside parentheses).

constant (false|true)

Unconditionally false or true.

! TEXT-MATCHER

Matches texts not matched by the given matcher.

TEXT-MATCHER || TEXT-MATCHER

Matches texts matched by any matcher.

TEXT-MATCHER && TEXT-MATCHER

Matches texts matched by every matcher.

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a text-matcher.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a text-matcher.

An unquoted string that is not a reserved word is interpreted as the name of a symbol.

( TEXT-MATCHER )

Notes

Operator precedence

Evaluation

||, &&: Operands are evaluated lazily, from left to right.

Syntax

Operators and parentheses must be separated by whitespace.

TEXT-SOURCE

Produces a text, from various sources

Forms:

TEXT-SOURCE'
( TEXT-SOURCE' )

where

TEXT-SOURCE'

SYMBOL-REFERENCE [TRANSFORMATION]

A reference to a "symbol" defined as either "text-source" or "string".

RICH-STRING [TRANSFORMATION]

-contents-of SOURCE-FILE-PATH [TRANSFORMATION]

The contents of an existing regular file.

(-stdout-from|-stderr-from) [-ignore-exit-code] PROGRAM

The output from a "program" - either stdout or stderr.

If PROGRAM defines a transformation of its output (via -transformed-by), the transformation is applied to either stdout (-stdout-from), or stderr (-stderr-from).

The result is HARD_ERROR if the exit code is non-zero, unless -ignore-exit-code is given.

The program must terminate.

SOURCE-FILE-PATH

The PATH of an existing regular file.

Accepted relativities (default is "home directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

TRANSFORMATION

Transforms the original contents.

Forms:

-transformed-by TEXT-TRANSFORMER

Note: TEXT-TRANSFORMER may not contain infix operators (unless inside parentheses).

TEXT-TRANSFORMER

Transforms a text

Forms:

filter MATCHER

Keeps lines matched by MATCHER, and discards lines not matched.

MATCHER is one of:

LINE-MATCHER: Applies a LINE-MATCHER.
-line-nums LINE-NUMBER-RANGE...: A line matches iff it's line number matches any LINE-NUMBER-RANGE.

Note: The line separator depends on the current OS ('\n', '\r\n', e.g.).

grep [-full] REGEX

Shortcut for "filter contents matches".

replace [LINES-SELECTION] [-preserve-new-lines] REGEX STRING

Replaces every string matching REGEX (on a single line) with STRING.

Backslash escapes in STRING are processed. That is, \n is converted to a single newline character, \r is converted to a carriage return, and so forth.

Unknown escapes such as \& are left alone.

Back-references, such as \6, are replaced with the substring matched by group 6 in REGEX.

Every line ends with "\n", except the last line, which may or may not end with "\n".
If -preserve-new-lines is given, this "\n" is excluded from the replacement.

Note: Lines are separated by "\n", regardless of the current OS.

char-case (-to-lower|-to-upper)

-to-lower: Converts all cased characters to lowercase.
-to-upper: Converts all cased characters to uppercase.

strip [-trailing-space|-trailing-new-lines]

Removes all whitespace at the beginning and end of the text (by default).

-trailing-space

Removes all whitespace at the end of the text.

-trailing-new-lines

Removes every "\n" at the end of the text.

Note: Lines are separated by "\n", regardless of the current OS.

run [-ignore-exit-code] PROGRAM

Runs a program, with input and output via stdin and stdout.

If PROGRAM defines stdin, then the text to transform is appended to that stdin.

The result is HARD_ERROR if the exit code is non-zero, unless -ignore-exit-code is given.

replace-test-case-dirs

Replaces every occurrence of a string that equals the absolute path of a TCDS directory (on a single line) with the name of the corresponding builtin symbol.

Paths/symbols that are replaced:

EXACTLY_ACT
EXACTLY_ACT_HOME
EXACTLY_HOME
EXACTLY_TMP

If EXACTLY_HOME and EXACTLY_ACT_HOME are equal, then paths will be replaced with EXACTLY_HOME.

identity

Gives output that is identical to the input.

TEXT-TRANSFORMER | TEXT-TRANSFORMER

Composition of text-transformers.

The output of the text-transformer to the left is given as input to the text-transformer to the right.

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a text-transformer.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a text-transformer.

An unquoted string that is not a reserved word is interpreted as the name of a symbol.

( TEXT-TRANSFORMER )

where

LINE-NUMBER-RANGE

A consecutive sequence of line numbers.

Negative numbers denote line numbers relative to the end.
-1 is the last line number, -2 is the second to last line number, etc.

Forms:

INTEGER: The single line number INTEGER.
:INTEGER: Line numbers from 1 to INTEGER (including).
INTEGER:: Line numbers starting from INTEGER.
INTEGER:INTEGER: Line numbers from INTEGER (to the left) to INTEGER (to the right) (including).

Note: Line numbers start at 1.

Note: INTEGER must not contain ":"

LINES-SELECTION

Forms:

-at LINE-MATCHER

Limits replacement to lines matching LINE-MATCHER.

Note: LINE-MATCHER may not contain infix operators (unless inside parentheses).

Notes

Syntax

Operators and parentheses must be separated by whitespace.

ACT-INTERPRETER

An interpreter program that executes source code of the "act" phase

Forms:

PATH [PROGRAM-ARGUMENT]...

The PATH of an executable file.

Accepted relativities (default is "home directory"):

-rel-home	home directory
-rel-act-home	act-home directory

% STRING [PROGRAM-ARGUMENT]...

A program installed on the current OS - a program in the OS PATH.

-python [PROGRAM-ARGUMENT]...

The Python interpreter (Python >= 3.6).

Since Exactly is written in Python, the Python interpreter is guaranteed to be available on the system.

Description

An external program that interprets a source code file, who's path is given as a command line argument.

When the program is applied, the path of the source code file is added as the last argument.

Referenced symbols must have been defined in the "setup" phase.

GLOB-PATTERN

A file name glob pattern

Description

Unix shell-style wildcards.

Patterns:

?
Matches any single character.
*
Matches any number of any characters including none.
[CHARACTERS]
Matches a single character that is listed in CHARACTERS.
[CHARACTER-CHARACTER]
Matches a single character in the given range.
[!CHARACTERS]
Matches a single character that is not listed in CHARACTERS.

Directory specifications:

/
Directory separator.

For a literal match, wrap the meta-characters in brackets. For example, '[?]' matches the character '?'.

INTEGER

An integer expression

STRING

where

STRING

An expression using Python syntax.

The expression must evaluate to an integer (a Python "int").

PROGRAM-ARGUMENT

An individual string, or a list of strings, with additional features for text-until-end-of-line and references to existing files

Forms:

RICH-STRING

Note: If a RICH-STRING that spans whole lines is used, it will be the last element in the list.

SYMBOL-REFERENCE

A reference to a "symbol" defined as either "string", "path" or "list".

A path symbol gives a single argument that is its absolute path.

A list symbol gives a list of arguments.

Note: To pass a list as a single argument, convert it to a string by surrounding it with soft quotes. The elements will be separated by a single space.

-existing-file PATH-OF-EXISTING

A PATH, with additional check for existence.

PATH must be a regular file.

Symbolic links are followed.

Gives a single argument of an absolute path.

-existing-dir PATH-OF-EXISTING

A PATH, with additional check for existence.

PATH must be a directory.

Symbolic links are followed.

Gives a single argument of an absolute path.

-existing-path PATH-OF-EXISTING

A PATH, with additional check for existence.

PATH can be any type of file.

Symbolic links are followed.

Gives a single argument of an absolute path.

where

PATH-OF-EXISTING

The PATH of an existing file.

Accepted relativities (default is "home directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

Description

An individual argument, or a list of arguments, in case of an unquoted reference to a "list" symbol.

An argument list is an ordinary "list" value, with additional features for text-until-end-of-line and references to existing files.

REGEX

A regular expression, using Python syntax

[-ignore-case] RICH-STRING

where

-ignore-case: Makes the matching ignore case.

RICH-STRING

A string value, with additional forms

Forms:

STRING

A normal string.

:> TEXT-UNTIL-END-OF-LINE

The text after ":>" until the end of the line becomes a single string.

Whitespace at both ends is removed.

Any SYMBOL-REFERENCE appearing in TEXT-UNTIL-END-OF-LINE is substituted.

<<MARKER [LINE]... MARKER

A sequence of lines, resembling shell "here document" syntax.

<<EOF
first line
...
last line
EOF

Any single-word string may be used as MARKER, as "EOF" in the example. What matters is that MARKER at the start and end of the input matches.

Each LINE must end with new-line.

Any SYMBOL-REFERENCE appearing in the text is substituted.

SHELL-COMMAND-LINE

A shell command line, as the remaining part of the current line

Description

A SHELL-COMMAND-LINE is passed as a single string to the operating system's shell, so all features of the shell can be used.

Us of the shell is of course not portable since it depends on the current operating system environment's shell.

On POSIX, the shell defaults to /bin/sh.
On Windows, the COMSPEC environment variable specifies the default shell.

SYMBOL-NAME

The name of a symbol

Description

A combination of alphanumeric characters and underscores.

SYMBOL-REFERENCE

A reference to a symbol using special reference syntax

@[SYMBOL-NAME]@

Builtin symbols

String symbols

NEW_LINE

The new-line character ("\n"), the line separator of strings handled by Exactly

Type: string

DESCRIPTION

Exactly separates lines by "\n", regardless of the current OS.

OS_LINE_SEP

The string that separates text lines on the current OS ("\n", "\r\n", e.g.).

Type: string

TAB

The tab character.

exactly [OPTION]... FILE

DESCRIPTION

Runs the test case in file FILE.

OPTIONS

--actor COMMAND-LINE

Specifies the "source interpreter" actor, by giving the program that serves as the interpreter.

COMMAND-LINE is a command line (with optional arguments), using Unix shell syntax.

Note that an actor specified in the test case has precedence over the actor given here.

See also

Concept "actor"
Instruction "actor" (in phase [conf])
Concept "shell syntax"

-k, --keep

Runs the test case as normal, but the "sandbox directory structure" is preserved, and its root directory is the only output on stdout.
If execution of the test case cannot be started (due to invalid syntax, e.g.), then nothing is output on stdout.

See also

Concept "sandbox directory structure"

--act

[before-assert] and [assert] are skipped; and instead of "reporting" the result, the outcome of [act] is emitted:

output on stdout/stderr from [act] becomes the output on stdout/stderr;
the exit code from [act] becomes the exit code from the program.

If an error occurs, the normal error information is emitted to stderr (following the output from [act]).

--preprocessor COMMAND-LINE

A command line that transforms a test case file as the first step of processing it.

COMMAND-LINE is a command line (with optional arguments), using Unix shell syntax.

See also

Concept "preprocessor"
Suite instruction "preprocessor" (in section [conf])
Concept "shell syntax"

--suite FILE

Runs the test case as if it were part of the given suite.

This overrides the default suite file "exactly.suite".

FILES

If there exists a file "exactly.suite" in the same directory as FILE, then this file must be a test suite, and the test case is run as part of this suite.

OUTCOME

Outcome is reported either as an exit code, or as an exit code together with an exit identifier printed as a single line on stdout.

See test case specification for details.

Invalid command line arguments: Exit with exit code 64.

Summary of exit codes and identifiers

Exit identifier	Exit code
FAIL	32
FILE_ACCESS_ERROR	65
HARD_ERROR	128
INTERNAL_ERROR	129
PASS	0
PRE_PROCESS_ERROR	65
SKIPPED	0
SYNTAX_ERROR	65
VALIDATION_ERROR	65
XFAIL	33
XPASS	33

Test Suites

Runs a test suite

SYNOPSIS

exactly suite [--reporter REPORTER] [--actor COMMAND-LINE] FILE

DESCRIPTION

Runs the test suite in file FILE.

OPTIONS

--actor COMMAND-LINE

Specifies a default "source interpreter" "actor" to use for every test case in the suite.

COMMAND-LINE is a command line (with optional arguments), using Unix shell syntax.

Note: An "actor" specified in the test suite or individual test cases will have precedence over the "actor" specified by this option.

See also

Concept "actor"
Concept "shell syntax"
Instruction "actor" (in phase [conf])

--reporter REPORTER

Specifies in which format to report the execution of the test suite (stdout, stderr, exit code).

Options: "progress","junit" (default "progress").

See also

Concept "suite reporter"
Suite reporter "progress"
Suite reporter "junit"

FILES

If FILE is a directory that contains a file "exactly.suite", then this file becomes the suite file argument.

OUTCOME

Reporting

Outcome is reported via an exit code and stdout.
stderr may contain helpful information.

Output on stdout is handled by the selected "suite reporter".
The "suite reporter" also handles the exit code, in some scenarios.

Scenarios

Invalid command line arguments
Invalid argument or an argument that denotes a non-existing file.
Exit code 64
stdout empty
Invalid suite
Syntax error in a suite file, or a reference to a non-existing case or sub-suite file.
Exit code 3
stdout Depends on the selected "suite reporter"
Complete execution
The suite has been executed.
Individual test cases may have failed, or may not be executable (due to syntax error, e.g.).
Exit code Depends on the selected "suite reporter"
stdout Depends on the selected "suite reporter"

Symbol Inspection

Reports the usage of symbols in a test case or test suite

SYNOPSIS

exactly symbol [TEST-CASE-OPTION]... FILE

exactly symbol suite [TEST-SUITE-OPTION]... FILE

Reports all user defined symbols in the case/suite.

Each symbol is reported on a separate line, together with its type and the number of references to it.

exactly symbol [TEST-CASE-OPTION]... FILE SYMBOL-NAME [--ref]

exactly symbol suite [TEST-SUITE-OPTION]... FILE SYMBOL-NAME [--ref]

Reports information about a symbol.

If only SYMBOL-NAME is given, the definition of the symbol is reported.

If --ref is given, all references to the symbol are reported.

OPTIONS

TEST-CASE-OPTION: Corresponds to the options for running a test case.
TEST-SUITE-OPTION: Corresponds to the options for running a test suite.

FILES

When reporting from a test case
If there exists a file "exactly.suite" in the same directory as FILE, then this file must be a test suite, and the test case is run as part of this suite.
When reporting from a test suite
If FILE is a directory that contains a file "exactly.suite", then this file becomes the suite file argument.

OUTCOME

The report is printed on stdout.

Errors are reported with exit codes and exit identifiers corresponding to the outcome of running the corresponding test case or test suite. But the case/suite is not executed, so no execution errors occur.

Help System

Exactlys help system.

exactly help [ARGUMENT]...

SYNOPSIS

help

Gives a brief description of the program.

help help

Displays this help.

help htmldoc

Generates a HTML version of all help information available in the program.

help case

Describes the test case command line syntax.

help case spec

Specification of the test case functionality.

help PHASE

Describes a test case phase.

help PHASE instructions

Lists instructions in PHASE.

help PHASE INSTRUCTION

Describes an instruction in a test case phase.

help instructions

Lists instructions per test case phase.

help INSTRUCTION

Describes all test case instructions with the given name.

help suite

Describes the test suite command line syntax.

help suite spec

Specification of the test suite functionality.

help suite SECTION

Describes a test suite section.

help suite SECTION INSTRUCTION

Describes an instruction in a test suite section.

help symbol

Describes the symbol usages report command line syntax.

help ENTITY-TYPE [ENTITY-NAME]

Lists all entities of a type; or describes a given entity.

concept	Concepts
directive	Directives
confparam	Configuration parameters
actor	Actors
type	Types
syntax	Syntax elements
builtin	Builtin symbols
reporter	Suite reporters

Exit code	Depends on the selected "suite reporter"
stdout	Depends on the selected "suite reporter"