Reference Manual for Exactly

Exactly version 0.12.3

Table of Contents

Test Cases
Test Suites
Concepts
Configuration parameters
Types
- Data types
- Logic types
Actors
Syntax elements
Builtin symbols
Command line syntax

Test Cases

Specification of test case functionality

Introduction

A test case is written as a plain text file:

[act]

helloworld

[assert]

exit-code == 0

stdout equals <<EOF
Hello, World!
EOF

If the file "helloworld.case" contains this test case, then Exactly can execute it:

> exactly helloworld.case
PASS

"PASS" means that Exactly was able to execute the "helloworld" program, and that all assertions were satisfied.

It also means that the executable "helloworld" file was found in in the same directory as the test case file.

File structure

[act] marks the beginning of the "act" phase.

The "act" phase contains the "action to check" - the thing that is tested by the test case.

It must consist of a single command line, starting with the name of an executable file (by default).
The file must be located in the same directory as the test case file (by default).

[assert] marks the beginning of the "assert" phase.

The "assert" phase contains assertions, such as "exit-code" and "stdout".

The assertions determines the outcome of the test case.
Each assertion either PASS or FAIL. If any assertion FAIL, then the outcome of the test case as a whole is FAIL. Otherwise it is PASS.

Structure

Phases

A test case is a sequence of "phases".
Executing a test case means executing all of it's phases.

All phases are optional.

The phases are (in order of execution):

[conf]

Configures the execution of the remaining phases by setting "configuration parameters".

[setup]

Sets up the environment that the "action to check" (the "act" phase) is executed in.

[act]

Contains the action to check; executes it and stores the outcome for later inspection.

This is the default phase.

[before-assert]

Prepares for the "assert" phase.

[assert]

Assertions on the outcome of the "action to check" (the "act" phase) that determine the outcome of the test case.

[cleanup]

Cleans up pollution from earlier phases that exist outside of the sandbox (e.g. in a database).

The "act" phase, "action to check" and "actors"

The contents of the "act" phase represents an "action to check" (ATC), and executing the "act" phase means executing this as an OS process.

The "actor" concept makes it possible to have different kind of ATCs. E.g. executable program files, source code files and shell commands.

The actor interprets the contents of the "act" phase, and thereby resolves the ATC.

A test case configures which actor to use.

The special "null" actor specifies a no-operations ATC. It is used if the test case does not contain an "act" phase.

Examples

An ATC that is a Python source file, executed with arguments:

[conf]

actor = -file python

[act]

my-python-logic_symbol_utils.py argument1 "second argument"

An ATC that is a shell command line:

[act]

$ echo 'This is a' shell command line > &2

Instructions

All phases except [act] is a sequence of instructions, zero or more.

Executing a phase with instructions means executing all it's instructions, in the order they appear in the test case.

Instructions in different phases serve different purposes.

For example, the instruction set for the [assert] phase contains instructions that serve as assertions.

Relation to test suites

Inclusion of phase contents from suites

A test suite can contain test case contents.
When a test case is run as part of a suite, this contents is included in the case.

Part of suite

A test case is considered part of a suite, depending on how it is run.

When run in the following ways, it is part of a suite:

Standalone
- A suite file is given via command line arguments
- A file "exactly.suite" exits in the same directory as the test case file
  The file "exactly.suite" must be a test suite.
  Note that the test case need not be listed in "exactly.suite".
Via test suite
The test case file is listed in the suite.

Execution environment

Directory structure and Current directory

Files and directories used by tests are organized in the "test case directory structure" (TCDS).
It consists of two parts:

Sandbox directory structure (SDS)
A set of temporary directories used by a single execution of a test case. one of them is the initial "current directory".
Home directory structure (HDS)
A set of persistent directories for files used by every execution of a test case.

Sandbox directory structure and Current directory

The SDS is created in a platform dependent location for temporary files.

The current directory is set to one of the directories inside the SDS at the beginning of execution.
This means that files and directories created in the current directory will be removed at the end of the execution.

The current directory is the same for all phases, unless changed by the "cd" instruction.
A change of "current directory" stay in effect for all following instructions and phases.

Home directory structure

The HDS organizes files that exist before the execution and that should probably not be modified.

One of the directories in the HDS is the "act-home" directory. It is the location of the executable program file that is tested - i.e. the location of files referenced from the "act" phase.

All directories in the HDS are initialized to the directory that contains the test case file. They can be changed by the "conf" phase.

File references

Exactly has functionality for referencing files in the TCDS using the "path" type.

The "path" data type has syntax for expressing paths relative to any of the TCDS directories.

The directory that a "path" value is relative to is called the "relativity" of the value.

Arguments to instructions that are paths use the "path" data type.
Each argument have different accepted "relativities" and different default "relativity".
The defaults are chosen to be the most likely "relativity". The accepted "relativities" are chosen to prevent modification of files in the HDS.

Symbols

A "symbol" is a named constant defined by the "def" instruction.

Once defined, it is available to all following instructions in the phase where it is defined, and in all following phases.

Symbols may be used to define reusable values used by instructions and the "act" phase.

OS process environment

Exactly executes OS processes as part of a test case. Part of their environment is controlled by Exactly.

Current directory

The current directory (PWD) is the same as for instructions.

It is changed by the "cd" instruction.

Environment variables

All OS environment variables that are set when Exactly is started are available in OS processes run from the test case.

Environment variables can be manipulated by the "env" instruction.

Timeout

Timeout for all OS processes is determined by the "configuration parameter" "timeout".

File syntax

Syntax is line oriented.

Top level elements start at the beginning of a line, and line ends mark the end of elements, although some may span several lines (e.g. expressions inside parentheses).

Phases

"[NAME]" on a single line declares the start of phase NAME.
This line marks the start the "assert" phase, for example:

[assert]

The following lines will belong to this phase.

File contents before the first phase declaration belong to the default phase, which is [act].

The order of the different phases in the test case file is irrelevant. The phases are always executed in the same order, regardless of the order they appear in the test case file.

A phase can be declared more than once.
Contents of multiple declarations are merged, and executed in the order it appears in the file.
Here, "exit-code" is executed before "stderr":

[assert]

exit-code == 0

[act]

helloworld

[assert]

stderr empty

Phase contents

All phases except the "act" phase consist of a sequence of "instructions" (see below).

The contents of the "act" phase depends on which "actor" is used.
By default, it is expected to contain a single command line.

Instructions

Instructions start at the beginning of the line with a space separated identifier that is the name of the instruction.

The name may optionally be followed by arguments. Most instructions use a syntax for options, arguments and quoting that resembles the unix shell. The order of options is relevant, though (unlike many unix tools).
The exact syntax depends on the particular instruction.

An instruction may span several lines, as this form of "stdout" does:

stdout equals <<EOF
Hello, World!
EOF

Instruction descriptions

An instruction may optionally be preceded by a "description" - a free text within quotes that is displayed together with the instruction source line in error messages.
The purpose of a description is to describe the meaning of the instruction using text that is easier to understand than the source line.
A description is a quoted string using shell syntax.

For example, a free text may be easier to understand than a shell command:

[assert]

'PATH should contain /usr/local/bin'

$ tr ':' '\n' < ../result/stdout | grep '^/usr/local/bin$'

A description may span several lines.

File inclusion

Parts of a test case can be put in an external file, using the "including" directive:

including external-part-of-test-case.xly

Comments and empty lines

Lines beginning with "#" are comments.
Comments may only appear on lines between instructions and phase headers.

Empty lines that are not part of an instruction are ignored.

Empty lines, and lines with comment line syntax, may be part of instructions and the "act" phase, though, as in the "stdout" instruction here:

stdout equals <<EOF
this assertion expects 4 lines of output
# this is the second line of the expected output

the above empty line is part of the expected output
EOF

Processing steps

Test case file "processing" starts after the command line arguments have been parsed, and Exactly has been told to run a test case.

A test case file is processed in a number of steps, where the actual execution of the test is the last step.

The outcome is reported by an exit code and an exit identifier printed as a single line on stdout.

If a step before execution fails, then the outcome is reported and the processing is halted.

Preprocessing
Transforms the test case file.
This step is applied only if a preprocessor has been given by "--preprocessor", or set in a test suite.
Fails if the preprocessor program cannot be executed, or if it exits with a non-zero exit code.
Exit code 3
Exit identifier PRE_PROCESS_ERROR
Syntax checking and directives processing
Checks the syntax of all elements in the test case file - phases, their instructions, and the action to check of the [act] phase.
Processes directives ("including").
Fails if a syntax error is found, or if a directive fails.
- Syntax error
  Exit code 3
  Exit identifier SYNTAX_ERROR
- File inclusion error
  Exit code 3
  Exit identifier FILE_ACCESS_ERROR
Validation
Checks references to symbols and external resources (files).
Fails if a reference to a symbol that has not been defined or has invalid type, is found, or if a reference to a non-existing file is found, e.g.
Exit code 1
Exit identifier VALIDATION_ERROR
Execution
Executes the actual test.
Executes the phases in the predefined order.
Executing a phase with instructions means executing all instructions in the order they appear in the test case file.
Execution halts if an instruction encounters an error, or, in the case of assertion instructions, if the assertion fails.

Exit code	3
Exit identifier	PRE_PROCESS_ERROR

Exit code	3
Exit identifier	SYNTAX_ERROR

Exit code	3
Exit identifier	FILE_ACCESS_ERROR

Outcome

Reporting

Outcome is reported either as an exit code, or as an exit code together with an exit identifier printed as a single line on stdout.

Scenarios

Complete execution

The outcome of a completely executed test case depends on two things:

The "test case status" set by the "conf" phase.
The default is PASS.
The outcome of the "assert" phase.

Together, these determine the outcome of the test case as a whole:

Status	[assert]	Test Case
PASS	PASS	PASS
PASS	FAIL	FAIL
FAIL	PASS	XPASS
FAIL	FAIL	XFAIL
SKIP		SKIPPED

The outcome is reported by an exit code and an exit identifier printed as a single line on stdout.

Test Case	Exit identifier	Exit code
PASS	PASS	0
FAIL	FAIL	2
XPASS	XPASS	5
XFAIL	XFAIL	4
SKIPPED	SKIPPED	77

Error during validation

If validation fails, the test case is not executed.

The outcome is reported by an exit code and an exit identifier printed as a single line on stdout.

Exit code	1
Exit identifier	VALIDATION_ERROR

Error during execution

If an error occur during the execution of a test, then this will halt the test and be reported as an error, and not as a failed test.

Note that an error will be reported even if both the "act" and "assert" phases have been executed successfully.

The outcome is reported by an exit code and an exit identifier printed as a single line on stdout.

The execution may be interrupted by the following causes:

Hard error
An instruction fails to do it's job. E.g. an instruction in the "setup" phase fails to create a file.
Exit code 99
Exit identifier HARD_ERROR
Implementation error
An error in the implementation of Exactly is detected.
Exit code 100
Exit identifier IMPLEMENTATION_ERROR

Exit code	99
Exit identifier	HARD_ERROR

Exit code	100
Exit identifier	IMPLEMENTATION_ERROR

Other errors

If parsing of command line arguments fails, Exactly halts with exit code 2 (no exit identifier is printed).

Otherwise an exit identifier is printed as a single line on stdout.

The following errors may occur:

File access
Failure of accessing a file referenced via "including".
Exit code 3
Exit identifier FILE_ACCESS_ERROR
Preprocessing
Fails if the preprocessor program cannot be executed, or if it exits with a non-zero exit code.
Exit code 3
Exit identifier PRE_PROCESS_ERROR
Syntax checking
Fails if the test case contains a syntax error.
Also fails if the test case is run as part of a test suite, and the test suite contains a syntax error.
Exit code 3
Exit identifier SYNTAX_ERROR

Exit code	3
Exit identifier	FILE_ACCESS_ERROR

Exit code	3
Exit identifier	PRE_PROCESS_ERROR

Exit code	3
Exit identifier	SYNTAX_ERROR

Summary of exit codes and identifiers

Exit identifier	Exit code
FAIL	2
FILE_ACCESS_ERROR	3
HARD_ERROR	99
IMPLEMENTATION_ERROR	100
PASS	0
PRE_PROCESS_ERROR	3
SKIPPED	77
SYNTAX_ERROR	3
VALIDATION_ERROR	1
XFAIL	4
XPASS	5

Phases

[conf]

Configures the execution of the remaining phases by setting "configuration parameters".

Consists of zero or more instructions.

Accepts the "including" directive.

Each instruction sets one of the "configuration parameters".

Phase execution order

This is the first phase.

If the "status" is set to SKIP, then the remaining phases are not executed.
Otherwise, the "setup" phase follows.

Instructions

act-home	Sets the "act-home" directory
actor	Specifies the "actor" that will execute the "act" phase
home	Sets the "home" directory
status	Sets the "test case status" configuration parameter
timeout	Sets the timeout of sub processes executed by instructions and the "act" phase.

[setup]

Sets up the environment that the "action to check" (the "act" phase) is executed in.

E.g.

populating the current directory with files and directories
setting the contents of stdin
setting environment variables
setting up external resources, such as databases.

Consists of zero or more instructions.

Accepts the "including" directive.

Each instruction should probably have some side effect that affects the "action to check".

Phase execution order

If the "status" is set to SKIP, then this phase is not executed.
Otherwise:

This phase follows the "conf" phase, and is the first phase that is executed in the sandbox.

If any of the instructions fail, then execution jumps to the "cleanup" phase, and the test case halts with an error.
Otherwise, the "act" phase follows.

Environment

Current directory
At the beginning of the phase, the "current directory" is the act directory in the "sandbox directory structure".
Environment variables
Environment variables are inherited from the parent process.

Instructions

$	Executes a command using the current operating system's shell
cd	Sets the "current directory"
copy	Copies files and directories into the "sandbox directory structure"
def	Defines a symbol
dir	Creates a directory
env	Manipulates environment variables
file	Creates a file
run	Runs a "program"
stdin	Sets the contents of stdin for the "action to check"

[act]

Contains the action to check; executes it and stores the outcome for later inspection.

Executes the "action to check" as an OS process.
It's outcome is stored as files in the result directory of the "sandbox directory structure":

exit code	result/exitcode
stdout	result/stdout
stderr	result/stderr

The actor configured for the test case determines what the "act" phase will execute.

Default actor is: "command line" - Executes a command line, in terms of a PROGRAM

If the "act" phase is not specified, or if it's contents is empty, then the "null" actor is used.

This is the default phase.

The source of the "action to check".

The syntax depends on which "actor" is used.

The "including" directive cannot be used.

Some escape sequences exist to make it possible to have contents that would otherwise be treated as phase headers.
The escape sequences are only recognized at the first non-space characters of a line.

Source	Translation
\[	[
\\	\

Phase execution order

If the "status" is set to SKIP, then this phase is not executed.
Otherwise:

This phase is executed directly after the "setup" phase.

If the "action to check" cannot be executed, then execution jumps to the "cleanup" phase, and the test case halts with an error.
Otherwise, the "before-assert" phase follows.

Environment

Current directory
The "current directory" is the same as at the end of the previous phase.
(which is the act directory in the "sandbox directory structure", if it has not been changed.)
Environment variables
Environment variables are inherited from the previous phase.
stdin
The contents set by the "stdin" instruction in the "setup" phase.
Empty otherwise.
Timeout
The value specified via the "timeout" configuration parameter.

[before-assert]

Prepares for the "assert" phase.

E.g. processing files and setting "current directory", in order to make the assertions in the "assert" phase smooth.

Note that such preparations might as well be done as part of the "assert" phase.

Doing the preparations in this phase can make the "assert" phase cleaner, since the "assert" phase then can consist solely of instructions that do "real" assertions (as opposed to preparations for assertions).

The drawback is that if an instruction in this phase fails, then no assertions will have been executed and one will not know if any of the assertions would pass.

Consists of zero or more instructions.

Accepts the "including" directive.

Each instruction should probably have some side effect that supports the instructions in the "assert" phase.

Phase execution order

If the "status" is set to SKIP, then this phase is not executed.
Otherwise:

This phase is executed directly after the "act" phase.

If any of the instructions fail, then execution jumps to the "cleanup" phase, and the test case halts with an error.
Otherwise, the "assert" phase follows.

Environment

Current directory
The "current directory" is the same as at the end of the "setup" phase.
(which is the act directory in the "sandbox directory structure", if it has not been changed.)
Environment variables
Environment variables are inherited from the previous phase.

Instructions

$	Executes a command using the current operating system's shell
cd	Sets the "current directory"
def	Defines a symbol
dir	Creates a directory
env	Manipulates environment variables
file	Creates a file
run	Runs a "program"

[assert]

Assertions on the outcome of the "action to check" (the "act" phase) that determine the outcome of the test case.

If any of the instructions in this phase FAIL, then the outcome of the test case is FAIL, otherwise it is PASS.

An exception to the above is if an instruction encounters an unexpected problem that makes it neither PASS nor FAIL, in which case the outcome of the test case will indicate this unexpected problem instead of PASS or FAIL.

Consists of zero or more instructions.

Accepts the "including" directive.

Every instructions in the "assert" phase is an assertion that report its outcome as either PASS or FAIL.

If an instruction encounter a problem that prevents it from completing its work, it reports this as HARD_ERROR.

In practice, though, some instructions may have more of a purpose of "preparation" for an assertion - they are helpers for the assertions.
Such a preparation might be to sort a file that the "action to check" has produced, so that it's possible to compare it with a constant file containing the expected sorted output, e.g.

This kind of preparation can also be done in the "before-assert" phase, but it might be more suitable to put it in the "assert" phase right before the instruction that does the "true" assertion.

Assertions
Instructions that serve as assertions.
Assertions and helpers
Instructions that may serve as either an assertion or a helper for an assertion.
Helpers
Instructions that prepare things for assertion instructions, and that are not assertions in them selves.
Typically, if these instructions cannot do their job, they report HARD_ERROR, instead of FAIL.
Note that the preparation done by such an instruction can perhaps be done in [before-assert].

Phase execution order

If the "status" is set to SKIP, then this phase is not executed.
Otherwise:

This phase is executed directly after the "before-assert" phase.

This phase is followed by the "cleanup" phase.

If any of the instructions FAIL, remaining instructions in the phase will be skipped and the execution will jump directly to the "cleanup" phase.

Environment

Current directory
The "current directory" is the same as at the end of the previous phase.
(which is the act directory in the "sandbox directory structure", if it has not been changed.)
Environment variables
Environment variables are inherited from the previous phase.

Instructions

Assertions

contents	Tests the contents of a file
dir-contents	Tests the contents of a directory
exists	Tests the existence, and optionally properties, of a file
exit-code	Tests the exit code
stderr	Tests the contents of stderr from the "action to check", or from a "program"
stdout	Tests the contents of stdout from the "action to check", or from a "program"

Assertions and helpers
$ Executes a command using the current operating system's shell, and PASS iff its exit code is 0
run Runs a "program", and PASS or FAIL depending on its exit code
Helpers
cd Sets the "current directory"
def Defines a symbol
dir Creates a directory
env Manipulates environment variables
file Creates a file

[cleanup]

Cleans up pollution from earlier phases that exist outside of the sandbox (e.g. in a database).

Pollution can come from an earlier phase such as [setup], or the "action to check" of the "act" phase.

Consists of zero or more instructions.

Accepts the "including" directive.

Each instruction probably has some side effect that does some cleanup.

Phase execution order

If the "status" is set to SKIP, then this phase is not executed.
Otherwise:

This phase is always executed.

If an instruction in an earlier phase encountered an unexpected error, then execution has jumped directly from that instruction to this phase.
If no unexpected error has occurred, then this phase follows the "assert" phase.

This is the last phase.

Environment

Current directory
The "current directory" is the same as at the end of the previous phase.
(which is the act directory in the "sandbox directory structure", if it has not been changed.)
Environment variables
Environment variables are inherited from the previous phase.

Instructions

$	Executes a command using the current operating system's shell
cd	Sets the "current directory"
def	Defines a symbol
dir	Creates a directory
env	Manipulates environment variables
file	Creates a file
run	Runs a "program"

Directives

including

Includes test case contents from an external file

SYNOPSIS

including FILE

where

FILE

The absolute or relative path of an existing file.

If FILE is relative, then it's relative to the directory of the current source file.

Paths uses Posix syntax.

DESCRIPTION

The effect of including a file is equivalent to having the contents of the included file in the including file; except that the current phase of the including file cannot be changed by an included file.

The default phase of the included file is the phase from which the file is included.

The included file may contain contents of different phases, by declaring different phases just as in a main test case file.
But the phase of the including file is not changed.

Instructions per phase

[conf]

The instructions in the "conf" phase.

act-home

Sets the "act-home" directory

SYNOPSIS

act-home = DIRECTORY

where

DIRECTORY

The absolute or relative path of an existing directory.

If DIRECTORY is relative, then it's relative to the location of the current source file - the file that contains the instruction.

Paths uses Posix syntax.

actor

Specifies the "actor" that will execute the "act" phase

SYNOPSIS

actor = -command: Specifies that the "command line" "actor" should be used.
actor = -file EXECUTABLE [ARGUMENT]...: Specifies that the "file interpreter" "actor" should be used, with an executable program as interpreter.
actor = -file $ COMMAND: Specifies that the "file interpreter" "actor" should be used, with a shell command as interpreter.
actor = -source EXECUTABLE [ARGUMENT]...: Specifies that the "source interpreter" "actor" should be used, with an executable program as interpreter.
actor = -source $ COMMAND: Specifies that the "source interpreter" "actor" should be used, with a shell command as interpreter.

DESCRIPTION

The "actor" specified by this instruction has precedence over all other ways to specify the actor.

home

Sets the "home" directory

SYNOPSIS

home = DIRECTORY

where

DIRECTORY

The absolute or relative path of an existing directory.

If DIRECTORY is relative, then it's relative to the location of the current source file - the file that contains the instruction.

Paths uses Posix syntax.

status

Sets the "test case status" configuration parameter

SYNOPSIS

status = STATUS

where

STATUS

PASS

The test case is executed and the "assert" phase is expected to PASS.

FAIL

The test case is executed and the "assert" phase is expected to FAIL.

Outcome of the test case is XFAIL, if [assert] FAILs.
If [assert] PASS, the result is XPASS.

SKIP

The test case is not executed.

Outcome of the test case is SKIPPED.

DESCRIPTION

The default "test case status" is PASS.

timeout

Sets the timeout of sub processes executed by instructions and the "act" phase.

SYNOPSIS

timeout = INTEGER

where

INTEGER: Timeout in seconds.

DESCRIPTION

Default: No timeout.

The timeout is per instruction, and for the "act" phase. It does not apply to the test case as a whole.

$

Executes a command using the current operating system's shell

SYNOPSIS

$ SHELL-COMMAND-LINE

where

SHELL-COMMAND-LINE

Something that is understood by the current operating system's shell.

It is the literal text until end of line.

DESCRIPTION

The result is HARD_ERROR if SHELL-COMMAND-LINE exits with a non-zero exit code.

cd

Sets the "current directory"

SYNOPSIS

cd PATH

where

PATH

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory

copy

Copies files and directories into the "sandbox directory structure"

SYNOPSIS

copy SOURCE [DESTINATION]

where

SOURCE

The PATH of an existing file or directory.

Accepted relativities (default is "home directory"):

-rel-home	home directory
-rel-act-home	act-home directory

DESTINATION

The PATH of an existing directory, or a non-existing file.

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

DESCRIPTION

If DESTINATION is not given
SOURCE is installed in the "current directory", as a file/directory with the basename of SOURCE.
If DESTINATION is given, but does not exist
SOURCE is copied as a file/directory with the path of DESTINATION
(the basename of SOURCE is not preserved).
If DESTINATION does exist
it must be a directory, and SOURCE is copied into that directory, as a file/directory with the basename of SOURCE.

If given, DESTINATION must appear on the same line as SOURCE.

As many attributes as possible of the copied files are preserved (this depends on the Python implementation).

def

Defines a symbol

SYNOPSIS

def TYPE SYMBOL-NAME = VALUE

where

TYPE, VALUE

TYPE	VALUE
string	(STRING\|HERE-DOCUMENT)
list	[ELEMENT]...
path	PATH
line-matcher	LINE-MATCHER
file-matcher	FILE-MATCHER
files-matcher	FILES-MATCHER
string-matcher	STRING-MATCHER
files-condition	FILES-CONDITION
string-transformer	STRING-TRANSFORMER
program	PROGRAM

PATH

Accepted relativities (default is "current directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory
-rel-cd	current directory
-rel SYMBOL-NAME	value of path symbol
-rel-here	location of the current source file

NOTE: When a PATH value is defined to be relative the "current directory", it means that it is relative the directory that is current when the symbol is REFERENCED,
not when it is defined.

STRING

A single word, or a single quoted text.

SYMBOL-NAME

A combination of alphanumeric characters and underscores.

DESCRIPTION

Defines the symbol SYMBOL-NAME to be a value of the given type.

SYMBOL-NAME must not have been defined earlier.

The defined symbol is available in all following instructions and phases.

dir

Creates a directory

SYNOPSIS

dir PATH

where

PATH

The PATH of a non-existing file.

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

DESCRIPTION

Creates parent directories if needed.

Does nothing if the given directory already exists.

env

Manipulates environment variables

SYNOPSIS

env NAME = STRING

Sets the environment variable NAME to STRING.

Elements of the form "${var_name}" in STRING, will be replaced with the value of the environment variable "var_name", or the empty string, if there is no environment variable with that name.

env unset NAME

Removes the environment variable NAME.

DESCRIPTION

The manipulation affects all following phases.

file

Creates a file

SYNOPSIS

file PATH: Creates an empty file.
file PATH = CONTENTS: Creates a file with contents given by CONTENTS.

where

PATH

The PATH of a non-existing file.

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

CONTENTS

STRING

HERE-DOCUMENT

-file SOURCE-FILE-PATH [TRANSFORMATION]

The contents of an existing file.

Any SYMBOL-REFERENCE appearing in the file SOURCE-FILE-PATH is NOT substituted.

(-stderr-from|-stdout-from) [-ignore-exit-code] PROGRAM

The output from a "program".

PROGRAM includes arguments until end of line, and an optional TRANSFORMATION on a following line.

The result is HARD_ERROR if the exit code is non-zero, unless -ignore-exit-code is given.

(Note: -ignore-exit-code do not apply to programs run by string transformers applied via PROGRAM.)

The program must terminate.

SOURCE-FILE-PATH

The PATH of an existing file.

Accepted relativities (default is "current directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

TRANSFORMATION

Transforms the original contents.

Forms:

-transformed-by STRING-TRANSFORMER

run

Runs a "program"

SYNOPSIS

run [-ignore-exit-code] PROGRAM

DESCRIPTION

The result is HARD_ERROR if the exit code is non-zero, unless -ignore-exit-code is given.

The program must terminate.

stdin

Sets the contents of stdin for the "action to check"

SYNOPSIS

stdin = (STRING|HERE-DOCUMENT|-file PATH)

Sets stdin to be the contents of a string, HERE-DOCUMENT or file.

Any SYMBOL-REFERENCE appearing in the file PATH is NOT substituted.

where

PATH

The PATH of the file that will be the contents of stdin.

Accepted relativities (default is "home directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

$

Executes a command using the current operating system's shell

SYNOPSIS

$ SHELL-COMMAND-LINE

where

SHELL-COMMAND-LINE

Something that is understood by the current operating system's shell.

It is the literal text until end of line.

DESCRIPTION

The result is HARD_ERROR if SHELL-COMMAND-LINE exits with a non-zero exit code.

cd

Sets the "current directory"

SYNOPSIS

cd PATH

where

PATH

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory

def

Defines a symbol

SYNOPSIS

def TYPE SYMBOL-NAME = VALUE

where

TYPE, VALUE

TYPE	VALUE
string	(STRING\|HERE-DOCUMENT)
list	[ELEMENT]...
path	PATH
line-matcher	LINE-MATCHER
file-matcher	FILE-MATCHER
files-matcher	FILES-MATCHER
string-matcher	STRING-MATCHER
files-condition	FILES-CONDITION
string-transformer	STRING-TRANSFORMER
program	PROGRAM

PATH

Accepted relativities (default is "current directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory
-rel-cd	current directory
-rel SYMBOL-NAME	value of path symbol
-rel-here	location of the current source file

NOTE: When a PATH value is defined to be relative the "current directory", it means that it is relative the directory that is current when the symbol is REFERENCED,
not when it is defined.

STRING

A single word, or a single quoted text.

SYMBOL-NAME

A combination of alphanumeric characters and underscores.

DESCRIPTION

Defines the symbol SYMBOL-NAME to be a value of the given type.

SYMBOL-NAME must not have been defined earlier.

The defined symbol is available in all following instructions and phases.

dir

Creates a directory

SYNOPSIS

dir PATH

where

PATH

The PATH of a non-existing file.

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

DESCRIPTION

Creates parent directories if needed.

Does nothing if the given directory already exists.

env

Manipulates environment variables

SYNOPSIS

env NAME = STRING

Sets the environment variable NAME to STRING.

Elements of the form "${var_name}" in STRING, will be replaced with the value of the environment variable "var_name", or the empty string, if there is no environment variable with that name.

env unset NAME

Removes the environment variable NAME.

DESCRIPTION

The manipulation affects all following phases.

file

Creates a file

SYNOPSIS

file PATH: Creates an empty file.
file PATH = CONTENTS: Creates a file with contents given by CONTENTS.

where

PATH

The PATH of a non-existing file.

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

CONTENTS

STRING

HERE-DOCUMENT

-file SOURCE-FILE-PATH [TRANSFORMATION]

The contents of an existing file.

Any SYMBOL-REFERENCE appearing in the file SOURCE-FILE-PATH is NOT substituted.

(-stderr-from|-stdout-from) [-ignore-exit-code] PROGRAM

The output from a "program".

PROGRAM includes arguments until end of line, and an optional TRANSFORMATION on a following line.

The result is HARD_ERROR if the exit code is non-zero, unless -ignore-exit-code is given.

(Note: -ignore-exit-code do not apply to programs run by string transformers applied via PROGRAM.)

The program must terminate.

SOURCE-FILE-PATH

The PATH of an existing file.

Accepted relativities (default is "current directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory
-rel-cd	current directory

TRANSFORMATION

Transforms the original contents.

Forms:

-transformed-by STRING-TRANSFORMER

run

Runs a "program"

SYNOPSIS

run [-ignore-exit-code] PROGRAM

DESCRIPTION

The result is HARD_ERROR if the exit code is non-zero, unless -ignore-exit-code is given.

The program must terminate.

SYNOPSIS

contents PATH STRING-MATCHER: Asserts that the contents of PATH satisfies STRING-MATCHER.

where

PATH

The PATH of the file who's contents is checked.

Accepted relativities (default is "current directory"):

-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

DESCRIPTION

The result is HARD_ERROR if PATH is not an existing regular file.
Symbolic links are followed.

dir-contents

Tests the contents of a directory

SYNOPSIS

dir-contents PATH [DIR-CONTENTS-OPTIONS] FILES-MATCHER: Asserts that the files in the directory PATH satisfies FILES-MATCHER.

where

PATH

The PATH of the directory who's contents is checked.

Accepted relativities (default is "current directory"):

-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

DIR-CONTENTS-OPTIONS

By default, only the direct contents of the tested directory is included - sub directory contents is excluded.

Forms:

-recursive [-min-depth INTEGER] [-max-depth INTEGER]

Includes sub directory contents (recursively).

Sub directory contents may be limited by specifying limits for the depth.

Depth 0 is the direct contents of the tested directory.

DESCRIPTION

The result is HARD_ERROR if PATH is not an existing directory.
Symbolic links are followed.

exists

Tests the existence, and optionally properties, of a file

SYNOPSIS

exists [!] PATH [PROPERTIES]

where

!

Negates the assertion.

PATH

Accepted relativities (default is "current directory"):

-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

PROPERTIES

Applies a FILE-MATCHER on PATH, if it exists.

Forms:

: FILE-MATCHER

DESCRIPTION

Symbolic links are not followed in the test of existence (so a broken symbolic link is considered to exist).

When not negated, the assertion will PASS if, and only if:
PATH exists, and has the specified properties.

When negated, the assertion will FAIL if, and only if:
PATH does not exist, or PATH does not have the specified properties.

exit-code

Tests the exit code

SYNOPSIS

exit-code [!] INTEGER-MATCHER

where

!: Negates the assertion.

DESCRIPTION

DESCRIPTION

The assertion PASS if, and only if, the exit code from SHELL-COMMAND-LINE is 0.
All other exit codes makes the assertion FAIL.

run

Runs a "program", and PASS or FAIL depending on its exit code

SYNOPSIS

run [-ignore-exit-code] PROGRAM

DESCRIPTION

PASS if the exit code is zero, or -ignore-exit-code is given.
Otherwise, FAIL.

The program must terminate.

cd

Sets the "current directory"

SYNOPSIS

cd PATH

where

PATH

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory

DESCRIPTION

Note: In the "assert" phase, this instruction is mostly useful as a helper for writing assertions. The instruction is not an assertion on its own.

def

Defines a symbol

SYNOPSIS

def TYPE SYMBOL-NAME = VALUE

where

TYPE, VALUE

TYPE	VALUE
string	(STRING\|HERE-DOCUMENT)
list	[ELEMENT]...
path	PATH
line-matcher	LINE-MATCHER
file-matcher	FILE-MATCHER
files-matcher	FILES-MATCHER
string-matcher	STRING-MATCHER
files-condition	FILES-CONDITION
string-transformer	STRING-TRANSFORMER
program	PROGRAM

PATH

Accepted relativities (default is "current directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory
-rel-cd	current directory
-rel SYMBOL-NAME	value of path symbol
-rel-here	location of the current source file

NOTE: When a PATH value is defined to be relative the "current directory", it means that it is relative the directory that is current when the symbol is REFERENCED,
not when it is defined.

STRING

A single word, or a single quoted text.

SYMBOL-NAME

A combination of alphanumeric characters and underscores.

DESCRIPTION

Defines the symbol SYMBOL-NAME to be a value of the given type.

SYMBOL-NAME must not have been defined earlier.

The defined symbol is available in all following instructions and phases.

dir

Creates a directory

SYNOPSIS

dir PATH

where

PATH

The PATH of a non-existing file.

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

DESCRIPTION

Note: In the "assert" phase, this instruction is mostly useful as a helper for writing assertions. The instruction is not an assertion on its own.

Creates parent directories if needed.

Does nothing if the given directory already exists.

env

Manipulates environment variables

SYNOPSIS

env NAME = STRING

Sets the environment variable NAME to STRING.

Elements of the form "${var_name}" in STRING, will be replaced with the value of the environment variable "var_name", or the empty string, if there is no environment variable with that name.

env unset NAME

Removes the environment variable NAME.

DESCRIPTION

Note: In the "assert" phase, this instruction is mostly useful as a helper for writing assertions. The instruction is not an assertion on its own.

The manipulation affects all following phases.

file

Creates a file

SYNOPSIS

file PATH: Creates an empty file.
file PATH = CONTENTS: Creates a file with contents given by CONTENTS.

where

PATH

The PATH of a non-existing file.

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

CONTENTS

STRING

HERE-DOCUMENT

-file SOURCE-FILE-PATH [TRANSFORMATION]

The contents of an existing file.

Any SYMBOL-REFERENCE appearing in the file SOURCE-FILE-PATH is NOT substituted.

(-stderr-from|-stdout-from) [-ignore-exit-code] PROGRAM

The output from a "program".

PROGRAM includes arguments until end of line, and an optional TRANSFORMATION on a following line.

The result is HARD_ERROR if the exit code is non-zero, unless -ignore-exit-code is given.

(Note: -ignore-exit-code do not apply to programs run by string transformers applied via PROGRAM.)

The program must terminate.

SOURCE-FILE-PATH

The PATH of an existing file.

Accepted relativities (default is "current directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory
-rel-cd	current directory

TRANSFORMATION

Transforms the original contents.

Forms:

-transformed-by STRING-TRANSFORMER

$

Executes a command using the current operating system's shell

SYNOPSIS

$ SHELL-COMMAND-LINE

where

SHELL-COMMAND-LINE

Something that is understood by the current operating system's shell.

It is the literal text until end of line.

DESCRIPTION

The result is HARD_ERROR if SHELL-COMMAND-LINE exits with a non-zero exit code.

cd

Sets the "current directory"

SYNOPSIS

cd PATH

where

PATH

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory

def

Defines a symbol

SYNOPSIS

def TYPE SYMBOL-NAME = VALUE

where

TYPE, VALUE

TYPE	VALUE
string	(STRING\|HERE-DOCUMENT)
list	[ELEMENT]...
path	PATH
line-matcher	LINE-MATCHER
file-matcher	FILE-MATCHER
files-matcher	FILES-MATCHER
string-matcher	STRING-MATCHER
files-condition	FILES-CONDITION
string-transformer	STRING-TRANSFORMER
program	PROGRAM

PATH

Accepted relativities (default is "current directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory
-rel-cd	current directory
-rel SYMBOL-NAME	value of path symbol
-rel-here	location of the current source file

NOTE: When a PATH value is defined to be relative the "current directory", it means that it is relative the directory that is current when the symbol is REFERENCED,
not when it is defined.

STRING

A single word, or a single quoted text.

SYMBOL-NAME

A combination of alphanumeric characters and underscores.

DESCRIPTION

Defines the symbol SYMBOL-NAME to be a value of the given type.

SYMBOL-NAME must not have been defined earlier.

The defined symbol is available in all following instructions and phases.

dir

Creates a directory

SYNOPSIS

dir PATH

where

PATH

The PATH of a non-existing file.

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

DESCRIPTION

Creates parent directories if needed.

Does nothing if the given directory already exists.

env

Manipulates environment variables

SYNOPSIS

env NAME = STRING

Sets the environment variable NAME to STRING.

Elements of the form "${var_name}" in STRING, will be replaced with the value of the environment variable "var_name", or the empty string, if there is no environment variable with that name.

env unset NAME

Removes the environment variable NAME.

DESCRIPTION

The manipulation affects all following phases.

file

Creates a file

SYNOPSIS

file PATH: Creates an empty file.
file PATH = CONTENTS: Creates a file with contents given by CONTENTS.

where

PATH

The PATH of a non-existing file.

Accepted relativities (default is "current directory"):

-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

CONTENTS

STRING

HERE-DOCUMENT

-file SOURCE-FILE-PATH [TRANSFORMATION]

The contents of an existing file.

Any SYMBOL-REFERENCE appearing in the file SOURCE-FILE-PATH is NOT substituted.

(-stderr-from|-stdout-from) [-ignore-exit-code] PROGRAM

The output from a "program".

PROGRAM includes arguments until end of line, and an optional TRANSFORMATION on a following line.

The result is HARD_ERROR if the exit code is non-zero, unless -ignore-exit-code is given.

(Note: -ignore-exit-code do not apply to programs run by string transformers applied via PROGRAM.)

The program must terminate.

SOURCE-FILE-PATH

The PATH of an existing file.

Accepted relativities (default is "current directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-result	result directory
-rel-cd	current directory

TRANSFORMATION

Transforms the original contents.

Forms:

-transformed-by STRING-TRANSFORMER

run

Runs a "program"

SYNOPSIS

run [-ignore-exit-code] PROGRAM

DESCRIPTION

The result is HARD_ERROR if the exit code is non-zero, unless -ignore-exit-code is given.

The program must terminate.

Test Suites

Specification of test suite functionality

Introduction

A test suite is written as a plain text file:

[cases]

a.case
group-dir/*.case

[suites]

sub-suite.suite
sub-dir/*.suite

If the file 'example.suite' contains this text, then Exactly can execute it:

> exactly suite example.suite
...
OK

A suite file can have any name - Exactly does not put any restriction on file names.

Structure

A suite is made up of "sections".

Some sections correspond to test case "phases".

All sections are optional.

Test cases and sub suites

A test suite contains test cases and sub suites.

These are specified in sections:

[cases]

Lists the test cases that are part of the suite.

This is the default section.

[suites]

Lists test suites (sub suites) that are part of the suite.

Common test case contents

A test suite has a section corresponding to every test case phase.

These sections specify test case contents that is common to all cases in the suite.
The contents is included in every test case in the suite.

Note that the contents is only included in test cases listed directly in the suite - not in sub suites.
The reason for this is that it should be easy to run a test case standalone, and Exactly has no functionality for looking up the ultimate root suite that a test case belongs to.

Sections:

[conf]: Common configuration, for all test cases in the suite
[setup]: Common contents of the "setup" phase, for all test cases in the suite
[act]: Common contents of the "act" phase, for all test cases in the suite
[before-assert]: Common contents of the "before-assert" phase, for all test cases in the suite
[assert]: Common contents of the "assert" phase, for all test cases in the suite
[cleanup]: Common contents of the "cleanup" phase, for all test cases in the suite

Additional test case configuration

The "conf" section has additional configuration possibilities, compared to those of the "conf" test case phase.

File syntax

Syntax is line oriented.

"[NAME]" on a single line declares the start of section "NAME".

The order of sections is irrelevant.

A section may appear any number of times. The contents of all appearances are accumulated.

Empty lines, and lines beginning with "#" are ignored, unless part of an instruction.

Outcome

Reporting

Outcome is reported via an exit code and stdout.
stderr may contain helpful information.

Output on stdout is handled by the selected "suite reporter".
The "suite reporter" also handles the exit code, in some scenarios.

Scenarios

Invalid command line arguments
Invalid argument or an argument that denotes a non-existing file.
Exit code 2
stdout empty
Invalid suite
Syntax error in a suite file, or a reference to a non-existing case or sub-suite file.
Exit code 3
stdout Depends on the selected "suite reporter"
Complete execution
The suite has been executed.
Individual test cases may have failed, or may not be executable (due to syntax error, e.g.).
Exit code Depends on the selected "suite reporter"
stdout Depends on the selected "suite reporter"

Sections

Test cases and sub suites

[cases]

Lists the test cases that are part of the suite.

This is the default section.

Each line specifies zero or more test cases to include in the suite, by giving the names of files that contain individual test cases.

File names are relative the location of the test suite file.

A test case file can have any name - Exactly does not put any restriction on file names.

Each line consists of a single file name glob pattern.
E.g.:

file.case
*.case
**/the.case
**/test-cases/*.case

[suites]

Lists test suites (sub suites) that are part of the suite.

Each line specifies zero or more test suites to include in the suite, by giving the names of files that contain individual test suites.

File names are relative the location of the test suite file.

A test suite file can have any name - Exactly does not put any restriction on file names.

Each line consists of a single file name glob pattern.
E.g.:

file.suite
*.suite
**/the.suite
**/test-suites/*.suite

Default suite file

A directory serves as a suite file, if it contains the default suite file "exactly.suite" (which must be a test suite).

Common test case contents and configuration

[conf]

Common configuration, for all test cases in the suite

Corresponds to the [conf] test case phase, with additional configuration possibilities.

The section contents is included before the contents of the "conf" phase of each test case.

Sequence of instructions - same as the [conf] test case phase.

Additional instructions

preprocessor

Sets a "preprocessor" to transform each test case file in the suite

[setup]

Common contents of the "setup" phase, for all test cases in the suite

Corresponds to the [setup] test case phase.

The section contents is included before the contents of the "setup" phase of each test case.

Identical to contents of the [setup] test case phase.

[act]

Common contents of the "act" phase, for all test cases in the suite

Corresponds to the [act] test case phase.

The section contents is included before the contents of the "act" phase of each test case.

Identical to contents of the [act] test case phase.

[before-assert]

Common contents of the "before-assert" phase, for all test cases in the suite

Corresponds to the [before-assert] test case phase.

The section contents is included before the contents of the "before-assert" phase of each test case.

Identical to contents of the [before-assert] test case phase.

[assert]

Common contents of the "assert" phase, for all test cases in the suite

Corresponds to the [assert] test case phase.

The section contents is included before the contents of the "assert" phase of each test case.

Identical to contents of the [assert] test case phase.

[cleanup]

Common contents of the "cleanup" phase, for all test cases in the suite

Corresponds to the [cleanup] test case phase.

The section contents is included after the contents of the "cleanup" phase of each test case.

Identical to contents of the [cleanup] test case phase.

junit

Outputs JUnit XML.

Output syntax

https://github.com/windyroad/JUnit-Schema/
as of 2016-11-14.

A suite with sub suites is reported as a 'testsuites' element.

A suite without sub suites is reported as a 'testsuite' element.

Exit code

Unconditionally 0.

progress

Reports execution progress in a human readable form.

This is the default "suite reporter".

Output syntax

Reports one event per line:

beginning of test suite
execution of test case
end of test suite

Beginning and end of test suites wraps the test cases that are contained directly in the test suite (i.e. it does not wrap test cases that are contained in sub suites).

Last line is an exit identifier, that depends on the outcome of the suite, and is related to the exit code.
A summary is printed on stderr.

Exit code

Exit codes, and corresponding exit identifiers printed as the last line of stdout:

Exit code	Exit identifier	When
0	OK	All test cases could be executed, and result was one of PASS, SKIPPED, XFAIL.
4	ERROR	At least one test case could not be executed, or was executed with a result other than those above: FAIL, HARD_ERROR, IMPLEMENTATION_ERROR, VALIDATION_ERROR, XPASS.
3	INVALID_SUITE	There was an error reading the test suite. No test cases have been executed.

preprocessor

Sets a "preprocessor" to transform each test case file in the suite

SYNOPSIS

preprocessor = SHELL-COMMAND-LINE

DESCRIPTION

The "preprocessor" is only used for the test cases in the current suite - not in sub suites.

SHELL-COMMAND-LINE uses "shell syntax".

Concepts

action to check

The action that is executed in the "act" phase.

Description

The "action to check" (ATC) is specified jointly by the configured "actor" and the contents of the "act" phase.

It is executed as an OS process.

Depending on which "actor" is configured, the ATC may be, for example:

executable program (with arguments)
source code file
source code
shell command

actor

Resolves the "action to check" by interpreting the contents of the "act" phase.

Description

The "actor" concept makes it possible to have different kind of actions executed in the "act" phase.
For example:

executable program file
source code file
source code
shell command

An actor determines the syntax and semantics of the "act" phase contents.

A test case uses a single actor.

Default actor is: "command line" - Executes a command line, in terms of a PROGRAM.

Other actors are configured via the "actor" configuration parameter, or the command line option "--actor".

configuration parameter

A value set in the "conf" phase that determine how the remaining phases are executed.

Description

act-home: Default location of files referenced from "act" phase, that should probably not be modified.
actor: Resolves the "action to check" by interpreting the contents of the "act" phase.
home: Default location of persistent helper files, that should probably not be modified.
status: The status of the test case - if it is expected to PASS or FAIL, or should be skipped.
timeout: Timeout of sub processes executed by instructions and the "act" phase.

current directory

The current directory of the environment in which instruction and OS processes are executed.

Description

During the execution of a test case, there is a "current directory" (CD).
It is initialized to the act/ sub directory of the "sandbox directory structure".

The CD is the same for all instructions and phases, unless changed by the "cd" instruction.
A change of CD stay in effect for all following instructions and phases.

Using the "current directory"

Instruction arguments of type "path" that are relative to the CD (via the "-rel-cd" option).

OS processes executed from within a test case have the CD as Present Working Directory (PWD) when the process starts.
This applies to the [act] phase (as a whole), and the "run" and "$" instructions.
The [act] phase is always executed as a single OS process execution.

Change of PWD in a process do not change the CD of following instructions and phases.

"def" instruction

The "def" is special because a "symbol" definition is evaluated each time it is referenced - not when it is defined (i.e. not when the "def" instruction is executed).

This means, that when a "path" "symbol" is defined with a relativity of "current directory" ("-rel-cd") it is relative the directory that is current when the symbol is REFERENCED,
not when it is defined.

directive

Processing during file reading and syntax checking.

Description

A directive is processed during file reading and syntax checking.
Nothing happens at later processing steps - i.e. unlike instructions, there is no execution.

A consequence of this is that symbols cannot be used.

environment variable

OS environment variables available to processes executed from within a test case.

Description

All OS environment variables that are set when Exactly is started are available in processes run from a test case.

Environment variables can be manipulated by the "env" instruction, in many phases.

A change of environment variables stay in effect for all following instructions and phases.

home directory structure

A set of persistent directories for files used by every execution of a test case.

Description

Typically, the directories in the "home directory structure" (HDS) exist before the test case is executed. They contain files read by, but not modified by, the test case.

Directories:

act-home directory

Default location of files referenced from "act" phase, that should probably not be modified.

Default value	The directory where the test case file is located.
Configuration parameter	act-home
Set by instruction	act-home
Builtin symbol	EXACTLY_ACT_HOME
Path relativity option	-rel-act-home

home directory

Default location of persistent helper files, that should probably not be modified.

Default value	The directory where the test case file is located.
Configuration parameter	home
Set by instruction	home
Builtin symbol	EXACTLY_HOME
Path relativity option	-rel-home

Configuration parameter

Each of these directories can be set in [conf], since they are "configuration parameters".
They cannot be set after [conf].

Relative paths

PATH values have an option for referring to each of these directories.

Builtin symbols

There is a builtin PATH "symbol" for each of these directories.

instruction

The building block of all phases except [act].

Description

All phases except [act] is a sequence of "instructions", zero or more.

Executing a phase means executing all instructions in the phase, in the order they appear in the test case source file.

Each phase has its own instruction set.

Semantics

In [assert], instructions serve as assertions by representing boolean expressions.
E.g. asserting that the action to check did not output anything to stdout:

stdout empty

In all other phases, and also for some of the instructions in [assert], the purpose of an instruction is it's side effects for setting up the execution environment of [act], [assert], or cleaning up after the test ([cleanup]).
E.g. creating a file:

file my-file.txt = "contents of my file"

Syntax

Forms:

PROPERTY = VALUE

For example:

timeout = 10

stdin   = "contents of stdin"

INSTRUCTION-NAME ARGUMENT...

For example:

file my-file.txt = "contents of my file

run % python -c :> import sys; sys.stdout.write("Hello world")

An instruction starts on a new line, beginning with the name of the instruction, followed by arguments.

Each instruction has it's own syntax for arguments.
Most common syntax is that of options and arguments resembling the Unix shell command line interface.
One difference though, is that the order of options is usually significant.

Some instructions may span multiple lines, while some must use only a single line.
The syntax is not always consistent.

Most arguments are values of one of Exactly's type system, and thus use a common syntax.

Arguments may refer to symbols defined by the "def" instruction.

Comments may not appear inside instructions.

preprocessor

A shell command that transforms a test case file as the first step of processing it

Description

A "preprocessor" is a shell command (with optional arguments), using Unix shell syntax.

When executed, it is given a single (additional) argument: the name of the test case file to transform.

The result of the transformation is the output on stdout.

An exit code other than 0 indicates error.

A test case file is preprocessed only if a "preprocessor" is given via the "--preprocessor" option, or via a test suite.

sandbox directory structure

A set of temporary directories used by a single execution of a test case. One of them is the initial "current directory".

Description

Each execution of a test case uses its own "sandbox directory structure" (SDS).

It is created in a platform dependent location for temporary files.

The SDS is created just before the "setup" phase is executed (after validation) and deleted when test case execution ends (unless "--keep" is used).

The SDS is used to store temporary files used by instructions (among other things). One such example is output from external processes - stdout and stderr. Thus, the SDS must have enough space to store the output from all such executed processes.

Directory structure

result/
tmp/
act/
internal/
- tmp/
- log/

act/

The current directory when [setup] begins.

It will be the "current directory" for the "act" phase, and following phases, unless it is changed by the "cd" instruction.

(Files and directories that [setup] creates are installed into the "current directory", if no instruction options are used to change this.)

EXACTLY_ACT	The value of this "symbol" is the absolute path of this directory.
-rel-act	The path relativity option that refers to this directory.

result/

Stores OS process outcome of [act], so that [assert] may check it.

This directory is initially empty.
It is populated when the "act" phase is executed with the following files (with obvious contents):

stderr
stdout
exitcode

EXACTLY_RESULT	The value of this "symbol" is the absolute path of this directory (after the "act" phase has been executed).
-rel-result	The path relativity option that refers to this directory.

tmp/

Reserved for custom helper files, used by the test case implementation.

Exactly does not touch this directory.
The test case can use it as a place where it is safe to put temporary files without the risk of name clashes with files from other program.

EXACTLY_TMP	The value of this "symbol" is the absolute path of this directory.
-rel-tmp	The path relativity option that refers to this directory.

Every symbol has a type according to Exactly's type system.

The name of a symbol must be unique within the whole test case - symbol names share a global name space.

A symbol name (SYMBOL-NAME) is: A combination of alphanumeric characters and underscores.

Definition

A symbol is defined by the "def" instruction:

def string SYMBOL_NAME = "the symbol value"

This defines a symbol SYMBOL_NAME to be the value "the symbol value", which is a value of type "string".

The type must be given explicitly.
For example:

def list               LIST_SYMBOL = first second "the third"

def string-transformer REPLACE_ID  = replace [0-9]{10} ID

Reference

A symbol must be defined before it is referenced.

Once defined, a symbol is available to all instructions following the definition - both in the phase where it is defined and in following phases.

A symbol reference may appear in string and as instruction arguments, in places where an argument of a certain type is expected.
Thus, symbols cannot be used for instruction names or arbitrary instruction arguments (as in unix shell scripts, e.g.).

There are two forms of symbol references:

Plain name	SYMBOL-NAME
Special syntax	@[SYMBOL-NAME]@

The special syntax is designed to reduce the risk of clashes with the syntax of other programming languages and shell script, e.g.
This is important since Exactly has limited capabilities for escaping and quoting.

Also, strings that resemble the special syntax will not be considered as errors, since it is assumed that these strings might have the syntax needed for a different purpose.
For example, the following strings does not constitute "symbol" references, and Exactly will not complain:

@[NOT/A_VALID_SYMBOL_NAME]@
@[VALID_SYMBOL_NAME ]@
@[VALID_SYMBOL_NAME]

The plain "symbol" name is used when a "symbol" reference is unambiguous. Otherwise, the special syntax is needed.
For example:

def string S = "reference to @[SYMBOL_NAME]@"

def list   L = first @[STRING_SYMBOL]@ "third element"

stdout -transformed-by ( filter LINE_MATCHER_SYMBOL ) ! empty

stdout equals -file -rel PATH_SYMBOL @[DIR_NAME_SYMBOL]@/@[BASE_NAME_SYMBOL]@

test case directory structure

Persistent and temporary directories used in the execution of a test case.

Description

Consists of two sets of directories:

Home directory structure

A set of persistent directories for files used by every execution of a test case.

act-home directory	Default location of files referenced from "act" phase, that should probably not be modified.
home directory	Default location of persistent helper files, that should probably not be modified.

Sandbox directory structure

A set of temporary directories used by a single execution of a test case. One of them is the initial "current directory".

act directory	The current directory when [setup] begins
result directory	Stores OS process outcome of [act], so that [assert] may check it
tmp directory	Reserved for custom helper files, used by the test case implementation

Exactly has support for referring to all of these directories from a test case via the "path" type.

type

Type system for symbols and instruction arguments.

Description

Exactly has a type system specialized for test cases.

Every symbol, and most instruction arguments, have a type.

There are two categories of types

Data types
list A sequence of zero or more strings.
path A file path, with special support for directories in the "test case directory structure"
string A sequence of characters.

Logic types

file matcher	Matches properties of an existing file - type, name and contents.
files condition	A condition of existence of a set of named files
files matcher	Matches a set of files (e.g. the contents of a directory).
line matcher	Matches individual text lines.
program	An external program, with optional arguments, and optional transformation of the output.
string matcher	Matches a string (a sequence of new-line separated text lines).
string transformer	Transforms a string (a sequence of new-line separated text lines).

A data type values is pure data.

A logic type values is a pure function.

Each data type has it's own specialized syntax.
The logic types have syntax that resembles common syntax of expressions.

Configuration parameters

act-home

Default location of files referenced from "act" phase, that should probably not be modified.

Default Value

Data types

list

A sequence of zero or more strings.

Description

Used for argument lists, etc.

Useful when the "action to check" is a program with arguments, and for the "run" instruction, e.g.

path

A file path, with special support for directories in the "test case directory structure"

Description

Rendering of "path" values

All "path" values are rendered as absolute paths.

This means, e.g., that values can be passed to external programs, which may use them without consideration of the current directory of the program process.

Note though, that a "path" value that is defined using the "def" instruction with relativity of "current directory" ("-rel-cd") is evaluated when it is REFERENCED,
not when it is defined.

This means that the rendered value is different if it is referenced in different locations with different "current directory".

string

A sequence of characters.

Description

Used for arbitrary string values, and relative file names.

When used as a relative file name, Posix syntax should be used.

file matcher

Matches properties of an existing file - type, name and contents.

SYNOPSIS

name (GLOB-PATTERN|-regex REG-EX)

Matches files who's ...

name : matches a file name glob pattern, or
base name : matches a regular expression, using python syntax

type TYPE

Matches files with the given type. Symbolic links are followed (unless matched type is symlink). TYPE is one of:

dir	File must be a directory
file	File must be a regular file
symlink	File must be a symbolic link

contents STRING-MATCHER

Matches regular files who's contents satisfies STRING-MATCHER.

The result is HARD_ERROR for a file that is not a regular file.
Symbolic links are followed.

dir-contents [DIR-CONTENTS-OPTIONS] FILES-MATCHER

Matches directories who's contents satisfies FILES-MATCHER.

The result is HARD_ERROR for a file that is not a directory.
Symbolic links are followed.

run [PATH-ARGUMENT-POSITION] PROGRAM

Runs a program. Matches iff the exit code is 0.

The path of the file to match is given as an argument according to PATH-ARGUMENT-POSITION, if given, or as the last argument, if PATH-ARGUMENT-POSITION is not given.

Transformations of the output from PROGRAM are ignored.

constant (false|true)

Unconditionally false or true.

! FILE-MATCHER

Matches files not matched by the given matcher.

FILE-MATCHER && FILE-MATCHER

Matches files matched by every matcher.

FILE-MATCHER || FILE-MATCHER

Matches files matched by any matcher.

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a file matcher.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a file matcher.

A string that is not the name of a file matcher is interpreted as the name of a symbol.

( FILE-MATCHER )

where

DIR-CONTENTS-OPTIONS

By default, only the direct contents of the tested directory is included - sub directory contents is excluded.

Forms:

-recursive [-min-depth INTEGER] [-max-depth INTEGER]

Includes sub directory contents (recursively).

Sub directory contents may be limited by specifying limits for the depth.

Depth 0 is the direct contents of the tested directory.

PATH-ARGUMENT-POSITION

Specifies which argument(s) to the program that will be the path of the file.

Forms:

-path-arg-last: The path of the file will be the last argument to PROGRAM.
-path-arg-marker MARKER: The path of the file will replace every argument to PROGRAM that equals MARKER.

Description

Prefix operators have the highest precedence.
All binary operators have the same precedence.

Operators and parentheses must be separated by whitespace.

files condition

A condition of existence of a set of named files

SYNOPSIS

CONSTANT

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a files condition.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a files condition.

A string that is not the name of a files condition is interpreted as the name of a symbol.

( FILES-CONDITION )

where

CONSTANT

A sequence of file names. Each together with an optional FILE-MATCHER.

Forms:

{ FILE-CONDITION... }

There can be only one FILE-CONDITION per line.

All parts must be separated by space.

FILE-CONDITION

A condition of existence of a file with a given name.

Forms:

FILE-NAME

FILE-NAME : FILE-MATCHER

A condition that includes matching of the named file by a FILE-MATCHER.

If there are multiple FILE-MATCHERs associated with a single file name (via multiple FILE-CONDITIONs), then the matchers are combined using &&, in order of appearance.

The ":" before the FILE-MATCHER must appear on the same line as the file name.

All parts must be separated by space.

FILE-NAME

A FILE-NAME is a STRING.

The file name must be a relative path, using Posix syntax.

Description

Operators and parentheses must be separated by whitespace.

files matcher

Matches a set of files (e.g. the contents of a directory).

SYNOPSIS

empty

Tests that the set of files is empty.

matches [-full] FILES-CONDITION

Tests that the set of files contains every file in FILES-CONDITION.

If -full is given, the set of files must contain no other files than those in FILES-CONDITION.

every file : FILE-MATCHER

Tests that every file satisfies the given FILE-MATCHER.

any file : FILE-MATCHER

Tests that any file satisfies the given FILE-MATCHER.

num-files INTEGER-MATCHER

Tests the number of files.

-selection FILE-MATCHER FILES-MATCHER

Applies FILES-MATCHER to the sub set of files matched by FILE-MATCHER.

-prune FILE-MATCHER FILES-MATCHER

Excludes contents of directories matched by FILE-MATCHER.

Symbolic links are followed.

Note: FILE-MATCHER is only applied to directories (and symbolic links to directories).

constant (false|true)

Unconditionally false or true.

! FILES-MATCHER

Matches sets of files not matched by the given matcher.

FILES-MATCHER && FILES-MATCHER

Matches sets of files matched by every matcher.

FILES-MATCHER || FILES-MATCHER

Matches sets of files matched by any matcher.

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a files matcher.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a files matcher.

A string that is not the name of a files matcher is interpreted as the name of a symbol.

( FILES-MATCHER )

Description

Prefix operators have the highest precedence.
All binary operators have the same precedence.

Operators and parentheses must be separated by whitespace.

line matcher

Matches individual text lines.

SYNOPSIS

matches REG-EX

Matches lines that contains a given REG-EX.

line-num INTEGER-MATCHER

Matches lines with a given line number.

Line numbers start at 1.

constant (false|true)

Unconditionally false or true.

! LINE-MATCHER

Matches lines not matched by the given matcher.

LINE-MATCHER && LINE-MATCHER

Matches lines matched by every matcher.

LINE-MATCHER || LINE-MATCHER

Matches lines matched by any matcher.

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a line matcher.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a line matcher.

A string that is not the name of a line matcher is interpreted as the name of a symbol.

( LINE-MATCHER )

Description

Prefix operators have the highest precedence.
All binary operators have the same precedence.

Operators and parentheses must be separated by whitespace.

program

An external program, with optional arguments, and optional transformation of the output.

Description

A "program" is executed as an OS process.

The timeout of the OS process is determined by the "timeout" configuration parameter.

The environment in which a program is executed is described in "Specification of test case functionality" / "OS process environment".

string matcher

Matches a string (a sequence of new-line separated text lines).

SYNOPSIS

empty

Matches if the string is empty.

equals (STRING|HERE-DOCUMENT|-file PATH-OF-EXPECTED)

Matches if the string is equal to a given string, HERE-DOCUMENT or contents of a file.

Any SYMBOL-REFERENCE appearing in the file PATH-OF-EXPECTED is NOT substituted.

matches [-full] REG-EX

Matches if REG-EX matches any part of the string.

If -full is given, then REG-EX must match the full string.

every line : LINE-MATCHER

Tests that every line satisfies the given LINE-MATCHER.

any line : LINE-MATCHER

Tests that any line satisfies the given LINE-MATCHER.

num-lines INTEGER-MATCHER

Matches if the number of lines of the string matches INTEGER-MATCHER.

-transformed-by STRING-TRANSFORMER STRING-MATCHER

Applies a matcher to a transformed variant of the string.

run PROGRAM

Runs a program. Matches iff the exit code is 0.

The string to match is given as stdin.

Transformations of the output from PROGRAM are ignored.

constant (false|true)

Unconditionally false or true.

! STRING-MATCHER

Matches strings not matched by the given matcher.

STRING-MATCHER && STRING-MATCHER

Matches strings matched by every matcher.

STRING-MATCHER || STRING-MATCHER

Matches strings matched by any matcher.

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a string matcher.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a string matcher.

A string that is not the name of a string matcher is interpreted as the name of a symbol.

( STRING-MATCHER )

where

PATH-OF-EXPECTED

Accepted relativities (default is "home directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

Description

Prefix operators have the highest precedence.
All binary operators have the same precedence.

Operators and parentheses must be separated by whitespace.

string transformer

Transforms a string (a sequence of new-line separated text lines).

SYNOPSIS

replace REG-EX STRING

Every string matching regular expression REG-EX - on a single line - is replaced with STRING.

Backslash escapes in STRING are processed. That is, \n is converted to a single newline character, \r is converted to a carriage return, and so forth.

Unknown escapes such as \& are left alone.

Back-references, such as \6, are replaced with the substring matched by group 6 in REG-EX.

filter LINE-MATCHER

Keeps lines matched by LINE-MATCHER, and discards lines not matched.

run [-ignore-exit-code] PROGRAM

Runs a program, with input and output via stdin and stdout.

The result is HARD_ERROR if the exit code is non-zero, unless -ignore-exit-code is given.

The program must terminate.

identity

Gives output that is identical to the input.

STRING-TRANSFORMER | STRING-TRANSFORMER

Composition of string transformers.

The output of the string transformer to the left is given as input to the string transformer to the right.

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a string transformer.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a string transformer.

A string that is not the name of a string transformer is interpreted as the name of a symbol.

( STRING-TRANSFORMER )

Description

All binary operators have the same precedence.

Operators and parentheses must be separated by whitespace.

Actors

command line

Executes a command line, in terms of a PROGRAM

This is the default "actor".

"act" phase contents

A single PROGRAM element.

Any number of empty lines and comment lines are allowed.

Syntax of "act" phase contents

Comment lines are lines beginning with "#" (optionally preceded by space).

SYNOPSIS

PROGRAM

Notes

Path relativities

When PROGRAM has the form of the PATH of an executable file:

Accepted relativities (default is "act-home directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

Transformations

If PROGRAM includes a transformation (via a STRING-TRANSFORMER), then the transformation is applied to stdout of the OS process.

file interpreter

Executes a source code file using an interpreter

"act" phase contents

A single line which is a file name followed by optional arguments.

The file name and arguments may be quoted according to "shell syntax".

Syntax of "act" phase contents

Comment lines are lines beginning with "#" (optionally preceded by space).

SYNOPSIS

FILE [ARGUMENT]...

where

FILE

The path of an existing source code file.

Accepted relativities (default is "act-home directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

ARGUMENT

A command line argument.

Uses "shell syntax".

null

Ignores the contents of the [act] phase. Exit code is unconditionally 0, and there is no output on neither stdout nor stderr

The null actor is useful when the test case does not test a program, but rather properties of the operating system environment.

"act" phase contents

Ignored.

Syntax of "act" phase contents

There are no syntax requirements.

source interpreter

Treats the [act] phase as source code to be executed by an interpreter

The contents of the "act" phase is stored in a file, and the name of this file is given as the last argument to the given interpreter.
If the interpreter is a shell command, then the quoted file name (according to "shell syntax") is appended to the end of the command string.

The interpreter may be specified, either via the "actor" instruction (both in test cases and test suites), or the "--actor" command line option.

"act" phase contents

Source code to be interpreted by the given interpreter.

Syntax of "act" phase contents

All lines of the "act" phase are part of the source code.
There is no recognition or special handling of comment lines and empty lines.

Syntax elements

Data types

LIST

A sequence of zero or more strings.

[STRING|SYMBOL-REFERENCE]...

where

SYMBOL-REFERENCE

A reference to a "symbol" defined as either "string" or "list".

If it is a "list", it is concatenated with the surrounding elements - lists cannot be nested.

PATH

A file path, with special support for directories in the "test case directory structure"

[RELATIVITY] STRING

where

STRING

A relative or absolute path, using Posix syntax.

It is a value of type "string", and thus uses STRING syntax.

RELATIVITY

An option that specifies the directory that STRING is relative to.

The available relativities depends on the instruction, and also on the argument's role in the instruction.

Summary of options:

Options for directories in the "home directory structure"
-rel-home home directory
-rel-act-home act-home directory
Options for directories in the "sandbox directory structure"
-rel-act act directory
-rel-tmp tmp directory
-rel-result result directory
Option for "current directory"
-rel-cd
This options is needed since many instructions have a default relativity other than the "current directory".
Option for a path denoted by a "symbol"
-rel SYMBOL-NAME
This option is always available.
Option for the location of the current source file
-rel-here
This option is only available when defining a path "symbol" using the "def" instruction.

SYMBOL-NAME

A "symbol" with type "path".

Description

Relativity

If STRING is an absolute path, then RELATIVITY must not be given.

If STRING is a relative path, then if RELATIVITY is ...

given
STRING is relative to the directory specified by RELATIVITY
not given
STRING is relative to the default relativity root directory,

The default relativity depends on the instruction, and also on the argument's role in the instruction.

NOTE

If STRING begins with a reference to a "path" "symbol", then it is an absolute path.

For example:

def path MY_PATH_SYMBOL = ...

... @[MY_PATH_SYMBOL]@/a/path/relative/to/MY_PATH_SYMBOL

Rendering of "path" values

All "path" values are rendered as absolute paths.

This means, e.g., that values can be passed to external programs, which may use them without consideration of the current directory of the program process.

Note though, that a "path" value that is defined using the "def" instruction with relativity of "current directory" ("-rel-cd") is evaluated when it is REFERENCED,
not when it is defined.

This means that the rendered value is different if it is referenced in different locations with different "current directory".

STRING

A sequence of characters.

Forms:

CHARACTER...

A "naked" sequence of characters.

CHARACTER may not be whitespace.

Any SYMBOL-REFERENCE appearing in the string is substituted.

"CHARACTER..."

Characters surrounded by "soft quotes" (").

Any SYMBOL-REFERENCE appearing in the string is substituted.

'CHARACTER...'

Characters surrounded by "hard quotes" (').

Any SYMBOL-REFERENCE appearing in the string is NOT substituted.

where

SYMBOL-REFERENCE

A reference to a "symbol" defined as either "string", "list" or "path".

An empty "list" value is rendered as an empty string.

A non-empty "list" value is rendered by separating the elements with a single space.

A "path" value is rendered as absolute paths (even when relativity is -rel-cd).

FILE-MATCHER

Matches properties of an existing file - type, name and contents.

Forms:

name (GLOB-PATTERN|-regex REG-EX)

Matches files who's ...

name : matches a file name glob pattern, or
base name : matches a regular expression, using python syntax

type TYPE

Matches files with the given type. Symbolic links are followed (unless matched type is symlink). TYPE is one of:

dir	File must be a directory
file	File must be a regular file
symlink	File must be a symbolic link

contents STRING-MATCHER

Matches regular files who's contents satisfies STRING-MATCHER.

The result is HARD_ERROR for a file that is not a regular file.
Symbolic links are followed.

dir-contents [DIR-CONTENTS-OPTIONS] FILES-MATCHER

Matches directories who's contents satisfies FILES-MATCHER.

The result is HARD_ERROR for a file that is not a directory.
Symbolic links are followed.

run [PATH-ARGUMENT-POSITION] PROGRAM

Runs a program. Matches iff the exit code is 0.

The path of the file to match is given as an argument according to PATH-ARGUMENT-POSITION, if given, or as the last argument, if PATH-ARGUMENT-POSITION is not given.

Transformations of the output from PROGRAM are ignored.

constant (false|true)

Unconditionally false or true.

! FILE-MATCHER

Matches files not matched by the given matcher.

FILE-MATCHER && FILE-MATCHER

Matches files matched by every matcher.

FILE-MATCHER || FILE-MATCHER

Matches files matched by any matcher.

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a file matcher.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a file matcher.

A string that is not the name of a file matcher is interpreted as the name of a symbol.

( FILE-MATCHER )

where

DIR-CONTENTS-OPTIONS

By default, only the direct contents of the tested directory is included - sub directory contents is excluded.

Forms:

-recursive [-min-depth INTEGER] [-max-depth INTEGER]

Includes sub directory contents (recursively).

Sub directory contents may be limited by specifying limits for the depth.

Depth 0 is the direct contents of the tested directory.

PATH-ARGUMENT-POSITION

Specifies which argument(s) to the program that will be the path of the file.

Forms:

-path-arg-last: The path of the file will be the last argument to PROGRAM.
-path-arg-marker MARKER: The path of the file will replace every argument to PROGRAM that equals MARKER.

Description

Prefix operators have the highest precedence.
All binary operators have the same precedence.

Operators and parentheses must be separated by whitespace.

FILES-CONDITION

A condition of existence of a set of named files

Forms:

CONSTANT

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a files condition.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a files condition.

A string that is not the name of a files condition is interpreted as the name of a symbol.

( FILES-CONDITION )

where

CONSTANT

A sequence of file names. Each together with an optional FILE-MATCHER.

Forms:

{ FILE-CONDITION... }

There can be only one FILE-CONDITION per line.

All parts must be separated by space.

FILE-CONDITION

A condition of existence of a file with a given name.

Forms:

FILE-NAME

FILE-NAME : FILE-MATCHER

A condition that includes matching of the named file by a FILE-MATCHER.

If there are multiple FILE-MATCHERs associated with a single file name (via multiple FILE-CONDITIONs), then the matchers are combined using &&, in order of appearance.

The ":" before the FILE-MATCHER must appear on the same line as the file name.

All parts must be separated by space.

FILE-NAME

A FILE-NAME is a STRING.

The file name must be a relative path, using Posix syntax.

Description

Operators and parentheses must be separated by whitespace.

FILES-MATCHER

Matches a set of files (e.g. the contents of a directory).

Forms:

empty

Tests that the set of files is empty.

matches [-full] FILES-CONDITION

Tests that the set of files contains every file in FILES-CONDITION.

If -full is given, the set of files must contain no other files than those in FILES-CONDITION.

every file : FILE-MATCHER

Tests that every file satisfies the given FILE-MATCHER.

any file : FILE-MATCHER

Tests that any file satisfies the given FILE-MATCHER.

num-files INTEGER-MATCHER

Tests the number of files.

-selection FILE-MATCHER FILES-MATCHER

Applies FILES-MATCHER to the sub set of files matched by FILE-MATCHER.

-prune FILE-MATCHER FILES-MATCHER

Excludes contents of directories matched by FILE-MATCHER.

Symbolic links are followed.

Note: FILE-MATCHER is only applied to directories (and symbolic links to directories).

constant (false|true)

Unconditionally false or true.

! FILES-MATCHER

Matches sets of files not matched by the given matcher.

FILES-MATCHER && FILES-MATCHER

Matches sets of files matched by every matcher.

FILES-MATCHER || FILES-MATCHER

Matches sets of files matched by any matcher.

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a files matcher.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a files matcher.

A string that is not the name of a files matcher is interpreted as the name of a symbol.

( FILES-MATCHER )

Description

Prefix operators have the highest precedence.
All binary operators have the same precedence.

Operators and parentheses must be separated by whitespace.

LINE-MATCHER

Matches individual text lines.

Forms:

matches REG-EX

Matches lines that contains a given REG-EX.

line-num INTEGER-MATCHER

Matches lines with a given line number.

Line numbers start at 1.

constant (false|true)

Unconditionally false or true.

! LINE-MATCHER

Matches lines not matched by the given matcher.

LINE-MATCHER && LINE-MATCHER

Matches lines matched by every matcher.

LINE-MATCHER || LINE-MATCHER

Matches lines matched by any matcher.

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a line matcher.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a line matcher.

A string that is not the name of a line matcher is interpreted as the name of a symbol.

( LINE-MATCHER )

Description

Prefix operators have the highest precedence.
All binary operators have the same precedence.

Operators and parentheses must be separated by whitespace.

PROGRAM

An external program, with optional arguments, and optional transformation of the output.

PGM-AND-ARGS [TRANSFORMATION]

where

PGM-AND-ARGS

A program followed by arguments until end of line.

Forms:

PGM-FOR-ARG-LIST [ARGUMENT]...

A program with a list of arguments.

$ SHELL-COMMAND-LINE

A shell command.

SHELL-COMMAND-LINE is the remaining part of the current line.
It is passed as a single string to the operating system's shell.

PGM-FOR-ARG-LIST

An executable program.

Forms:

PATH

The PATH of an executable file.

Accepted relativities (default is "home directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

Note: The relativities are different for the "command line" "actor".

% STRING

A program installed on the current system - a program in the OS PATH.

@ SYMBOL-NAME

Reference to a program that has been defined using the def instruction.

SYMBOL-NAME must have been defined as a "program".

Arguments and transformations are appended to existing arguments and transformations of SYMBOL-NAME.

-python

The Python interpreter (Python >= 3.5.4).

Since Exactly is written in Python, the Python interpreter is guaranteed to be available on the system.

ARGUMENT

An individual argument, or a list of arguments, in case of an unquoted reference to a "list" "symbol".

An argument list is an ordinary "list" value, with additional features for text-until-end-of-line and references to existing files.

Forms:

STRING

"path" values are handled as "string" values the usual way - rendered as an absolute path.

LIST

Elements of referenced lists are appended to the preceding arguments - argument lists are not nested.

To pass a LIST as a single argument to a program, convert it to a STRING by surrounding it with quotes. (The elements will be separated by a single space.)

:> TEXT-UNTIL-END-OF-LINE

The remaining part of the current line becomes a single argument.

-existing-file PATH-OF-EXISTING

A PATH, with additional check for existence.

Symbolic links are followed.

Values are rendered as absolute paths.

-existing-dir PATH-OF-EXISTING

A PATH, with additional check for existence.

Symbolic links are followed.

Values are rendered as absolute paths.

-existing-path PATH-OF-EXISTING

A PATH, with additional check for existence.

Symbolic links are followed.

Values are rendered as absolute paths.

PATH-OF-EXISTING

The PATH of an existing file.

Accepted relativities (default is "home directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

TRANSFORMATION

Transforms the output from the program.

Depending on the context, either stdout or stderr is transformed.

Must appear on a separate line.
If not, it will be interpreted as an ARGUMENT.

Forms:

-transformed-by STRING-TRANSFORMER

STRING-MATCHER

Matches a string (a sequence of new-line separated text lines).

Forms:

empty

Matches if the string is empty.

equals (STRING|HERE-DOCUMENT|-file PATH-OF-EXPECTED)

Matches if the string is equal to a given string, HERE-DOCUMENT or contents of a file.

Any SYMBOL-REFERENCE appearing in the file PATH-OF-EXPECTED is NOT substituted.

matches [-full] REG-EX

Matches if REG-EX matches any part of the string.

If -full is given, then REG-EX must match the full string.

every line : LINE-MATCHER

Tests that every line satisfies the given LINE-MATCHER.

any line : LINE-MATCHER

Tests that any line satisfies the given LINE-MATCHER.

num-lines INTEGER-MATCHER

Matches if the number of lines of the string matches INTEGER-MATCHER.

-transformed-by STRING-TRANSFORMER STRING-MATCHER

Applies a matcher to a transformed variant of the string.

run PROGRAM

Runs a program. Matches iff the exit code is 0.

The string to match is given as stdin.

Transformations of the output from PROGRAM are ignored.

constant (false|true)

Unconditionally false or true.

! STRING-MATCHER

Matches strings not matched by the given matcher.

STRING-MATCHER && STRING-MATCHER

Matches strings matched by every matcher.

STRING-MATCHER || STRING-MATCHER

Matches strings matched by any matcher.

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a string matcher.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a string matcher.

A string that is not the name of a string matcher is interpreted as the name of a symbol.

( STRING-MATCHER )

where

PATH-OF-EXPECTED

Accepted relativities (default is "home directory"):

-rel-home	home directory
-rel-act-home	act-home directory
-rel-act	act directory
-rel-tmp	tmp directory
-rel-cd	current directory

Description

Prefix operators have the highest precedence.
All binary operators have the same precedence.

Operators and parentheses must be separated by whitespace.

STRING-TRANSFORMER

Transforms a string (a sequence of new-line separated text lines).

Forms:

replace REG-EX STRING

Every string matching regular expression REG-EX - on a single line - is replaced with STRING.

Backslash escapes in STRING are processed. That is, \n is converted to a single newline character, \r is converted to a carriage return, and so forth.

Unknown escapes such as \& are left alone.

Back-references, such as \6, are replaced with the substring matched by group 6 in REG-EX.

filter LINE-MATCHER

Keeps lines matched by LINE-MATCHER, and discards lines not matched.

run [-ignore-exit-code] PROGRAM

Runs a program, with input and output via stdin and stdout.

The result is HARD_ERROR if the exit code is non-zero, unless -ignore-exit-code is given.

The program must terminate.

identity

Gives output that is identical to the input.

STRING-TRANSFORMER | STRING-TRANSFORMER

Composition of string transformers.

The output of the string transformer to the left is given as input to the string transformer to the right.

SYMBOL-REFERENCE

Reference to a symbol, that must have been defined as a string transformer.

SYMBOL-NAME

Reference to a symbol, that must have been defined as a string transformer.

A string that is not the name of a string transformer is interpreted as the name of a symbol.

( STRING-TRANSFORMER )

Description

All binary operators have the same precedence.

Operators and parentheses must be separated by whitespace.

GLOB-PATTERN

A file name glob pattern

Description

A file name, with ability to match multiple files.

Pattern matchers:

?
Matches any single character.
*
Matches any number of any characters including none.
[CHARACTERS]
Matches a single character that is listed in CHARACTERS.
[CHARACTER-CHARACTER]
Matches a single character in the given range.
[!CHARACTER-SET]
Matches a single character NOT in CHARACTER-SET.

Directory specifications:

**
The directory where the pattern is applied, and all subdirectories, recursively.
/
Directory separator.

For a literal match, wrap the meta-characters in brackets. For example, '[?]' matches the character '?'.

A patterns cannot denote an absolute path.

HERE-DOCUMENT

A string value, given as a sequence of lines, resembling shell "here document" syntax

Description

<<EOF
first line
...
last line
EOF

Any single-word string may be used instead of "EOF" as marker. What matters is that the maker at start and end of input matches.

Any SYMBOL-REFERENCE appearing in the text is substituted.

INTEGER

An integer expression

Description

A STRING that is an expression that evaluates to an integer (using Python syntax).

May be quoted (to allow space).

SYMBOL-REFERENCEs are substituted.

INTEGER-MATCHER

Matches an integer

OPERATOR INTEGER

Matches if, and only if, the actual value satisfies the comparison.

The actual value that is matched serves as the left hand side in the comparison.

where

OPERATOR: One of != < <= == > >=

REG-EX

A regular expression, using Python syntax

[-ignore-case] STRING

where

-ignore-case: Makes the matching ignore case.

SHELL-COMMAND-LINE

A shell command line, as the remaining part of the current line.

Description

A SHELL-COMMAND-LINE is passed as a single string to the operating system's shell, so all features of the shell can be used.

Us of the shell is of course not portable since it depends on the current operating system environment's shell.

On POSIX, the shell defaults to /bin/sh.
On Windows, the COMSPEC environment variable specifies the default shell.

SYMBOL-NAME

The name of a symbol

Description

A combination of alphanumeric characters and underscores.

SYMBOL-REFERENCE

A reference to a symbol using special reference syntax

@[SYMBOL-NAME]@

Builtin symbols

String symbols

LINE_SEP

The string that separates text lines on the current OS ('\n', '\r\n', e.g.)

Type: string

TAB

The tab character

If EXACTLY_HOME and EXACTLY_ACT_HOME are equal, then paths will be replaced with EXACTLY_HOME.

Paths that are replaced:

EXACTLY_ACT
EXACTLY_ACT_HOME
EXACTLY_HOME
EXACTLY_TMP

TO_LOWER_CASE

Converts all cased characters to lowercase

Type: string transformer

TO_UPPER_CASE

Converts all cased characters to uppercase

Type: string transformer

Command line syntax

Test Cases

Runs a test case

SYNOPSIS

exactly [OPTION]... FILE

DESCRIPTION

Runs the test case in file FILE.

OPTIONS

--actor SHELL-COMMAND-LINE

Specifies the "source interpreter" actor, by giving the program that serves as the interpreter.

SHELL-COMMAND-LINE is a shell command (with optional arguments), using Unix shell syntax.

Note that an actor specified in the test case has precedence over the actor given here.

See also

Concept "actor"
Concept "shell syntax"

-k, --keep

Runs the test case as normal, but the "sandbox directory structure" is preserved, and it's root directory is the only output on stdout.
If execution of the test case cannot be started (due to invalid syntax, e.g.), then nothing is output on stdout.

See also

Concept "sandbox directory structure"

--act

[before-assert] and [assert] are skipped; and instead of "reporting" the result, the outcome of [act] is emitted:

output on stdout/stderr from [act] becomes the output on stdout/stderr;
the exit code from [act] becomes the exit code from the program.

If an error occurs, the normal error information is emitted to stderr (following the output from [act]).

--preprocessor SHELL-COMMAND-LINE

A shell command that transforms a test case file as the first step of processing it.

SHELL-COMMAND-LINE is a shell command (with optional arguments), using Unix shell syntax.

See also

Concept "preprocessor"
Concept "shell syntax"
Suite instruction "preprocessor" (in section [conf])

--suite FILE

Runs the test case as if it were part of the given suite.

This overrides the default suite file "exactly.suite".

FILES

If there exists a file "exactly.suite" in the same directory as FILE, then this file must be a test suite, and the test case is run as part of this suite.

OUTCOME

Outcome is reported either as an exit code, or as an exit code together with an exit identifier printed as a single line on stdout.

See test case specification for details.

Summary of exit codes and identifiers

Exit identifier	Exit code
FAIL	2
FILE_ACCESS_ERROR	3
HARD_ERROR	99
IMPLEMENTATION_ERROR	100
PASS	0
PRE_PROCESS_ERROR	3
SKIPPED	77
SYNTAX_ERROR	3
VALIDATION_ERROR	1
XFAIL	4
XPASS	5

Test Suites

Runs a test suite

SYNOPSIS

exactly suite [--reporter REPORTER] [--actor SHELL-COMMAND-LINE] FILE

DESCRIPTION

Runs the test suite in file FILE.

OPTIONS

--actor SHELL-COMMAND-LINE

Specifies a default "source interpreter" "actor" to use for every test case in the suite.

SHELL-COMMAND-LINE is a shell command (with optional arguments), using Unix shell syntax.

Note: An "actor" specified in the test suite or individual test cases will have precedence over the "actor" specified by this option.

See also

Concept "actor"
Concept "shell syntax"
Instruction "actor" (in phase [conf])

--reporter REPORTER

Specifies in which format to report the execution of the test suite (stdout, stderr, exit code).

Options: "progress","junit" (default "progress").

See also

Concept "suite reporter"
Suite reporter "progress"
Suite reporter "junit"

FILES

If FILE is a directory that contains a file "exactly.suite", then this file becomes the suite file argument.

OUTCOME

Reporting

Outcome is reported via an exit code and stdout.
stderr may contain helpful information.

Output on stdout is handled by the selected "suite reporter".
The "suite reporter" also handles the exit code, in some scenarios.

Scenarios

Invalid command line arguments
Invalid argument or an argument that denotes a non-existing file.
Exit code 2
stdout empty
Invalid suite
Syntax error in a suite file, or a reference to a non-existing case or sub-suite file.
Exit code 3
stdout Depends on the selected "suite reporter"
Complete execution
The suite has been executed.
Individual test cases may have failed, or may not be executable (due to syntax error, e.g.).
Exit code Depends on the selected "suite reporter"
stdout Depends on the selected "suite reporter"

Symbol Usages

Reports the usage of user defined symbols in a test case or test suite

SYNOPSIS

exactly symbol [TEST-CASE-OPTION]... FILE

exactly symbol suite [TEST-SUITE-OPTION]... FILE

Reports all symbols defined in the case/suite.

Each symbol is reported on a separate line, together with its type and the number of references to it.

exactly symbol [TEST-CASE-OPTION]... FILE SYMBOL-NAME [--ref]

exactly symbol suite [TEST-SUITE-OPTION]... FILE SYMBOL-NAME [--ref]

Reports information about a symbol.

If only SYMBOL-NAME is given, the definition of the symbol is reported.

If --ref is given, all references to the symbol are reported.

OPTIONS

TEST-CASE-OPTION: Corresponds to the options for running a test case.
TEST-SUITE-OPTION: Corresponds to the options for running a test suite.

FILES

When reporting from a test case
If there exists a file "exactly.suite" in the same directory as FILE, then this file must be a test suite, and the test case is run as part of this suite.
When reporting from a test suite
If FILE is a directory that contains a file "exactly.suite", then this file becomes the suite file argument.

OUTCOME

The report is printed on stdout.

Errors are reported with exit codes and exit identifiers corresponding to the outcome of running the corresponding test case or test suite. But the case/suite is not executed, so no execution errors occur.

Getting Help

Exactlys help system.

exactly help [ARGUMENT]...

SYNOPSIS

help

Gives a brief description of the program.

help help

Displays this help.

help htmldoc

Generates a HTML version of all help information available in the program.

help case

Describes the test case command line syntax.

help case spec

Specification of the test case functionality.

help PHASE

Describes a test case phase.

help PHASE instructions

Lists instructions in PHASE.

help PHASE INSTRUCTION

Describes an instruction in a test case phase.

help instructions

Lists instructions per test case phase.

help INSTRUCTION

Describes all test case instructions with the given name.

help suite

Describes the test suite command line syntax.

help suite spec

Specification of the test suite functionality.

help suite SECTION

Describes a test suite section.

help suite SECTION INSTRUCTION

Describes an instruction in a test suite section.

help symbol

Describes the symbol usages report command line syntax.

help ENTITY-TYPE [ENTITY-NAME]

Lists all entities of a type; or describes a given entity.

concept	Concepts
confparam	Configuration parameters
type	Types
actor	Actors
syntax	Syntax elements
directive	Directives
builtin	Builtin symbols
reporter	Suite reporters

Exit code	Depends on the selected "suite reporter"
stdout	Depends on the selected "suite reporter"