Testing CLI tools in Go
All your tests pass, but the program crashes when you run it. Sound familiar?
In Part 1 of this series we made
friends with the very useful testscript
package, and saw
how to write and run test scripts from our Go tests.
If all we could do with testscript
were to run existing
programs with certain arguments, and assert that they succeed (or fail),
and produce certain outputs (or not), that would still be pretty
useful.
But we’re not limited to running existing programs. If we
want to test our own binary, for example, we don’t have to go through
all the labour of compiling and installing it first, in order to execute
it in a script. testscript
can save us the trouble. Let’s
see how.
Suppose we’re writing a program named hello
, for
example, whose job is simply to print a “hello world” message on the
terminal. As it’s an executable binary, it will need a main
function. Something like this would do:
func main() {
.Println("hello world")
fmt}
How can we test this? The main
function can’t be called
directly from a Go test; it’s invoked automatically when we run the
compiled binary. We can’t test it directly.
So we’ll instead delegate its duties to some other function that we
can call from a test: hello.Main
, let’s say.
We’ll do nothing in the real main
function except call
hello.Main
to do the printing, and then exit with whatever
status value it returns:
func main() {
.Exit(hello.Main())
os}
Fine. So we can deduce from this what the signature of
hello.Main
needs to be. It takes no arguments, and it
returns an int
value representing the program’s exit
status. Let’s write it:
package hello
import "fmt"
func Main() int {
.Println("hello world")
fmtreturn 0
}
This is a very simple program, of course, so it doesn’t have
any reason to report a non-zero exit status, but it could if it needed
to. Here, we just explicitly return 0
, so the program will
always succeed.
Now that we’ve delegated all the real functionality of the program to
this hello.Main
function, we could call it from a
Go test if we wanted to. But that wouldn’t invoke it as a
binary, only as a regular Go function call, which isn’t really
what we want here. For example, we probably want different values of
os.Args
for each invocation of the program.
What we need to do instead is to tell testscript
to make
our program available to scripts as a binary named hello
.
Here’s what that looks like in our Go test code:
func TestMain(m *testing.M) {
.Exit(testscript.RunMain(m, map[string]func() int{
os"hello": hello.Main,
}))
}
The function TestMain
is special to Go: it doesn’t test
anything itself, but its job is usually to set something up in advance,
before any test actually runs.
What this TestMain
is doing is calling
testscript.RunMain
. What does that do? Well, it
runs all our Go tests, but before it does that, it also sets up any
custom programs that we want to use in scripts.
To do that, we pass this map to RunMain
, connecting the
name of our desired binary (hello
) with its delegate
main
function (hello.Main
):
map[string]func() int{
"hello": hello.Main,
}
This tells testscript
to create an executable binary
named hello
, whose main
function will call
hello.Main
. This binary will be installed in a temporary
directory (not the script’s work directory), and that directory will be
added to the $PATH
variable in the environment of all our
scripts.
If the magic works the way it should, then, we’ll be able to use
exec
in a script to run the hello
program,
just as if we’d compiled and installed it manually. After the tests have
finished, the binary and its temporary directory will be deleted
automatically.
Let’s give it a try. We’ll create a script with the following contents:
exec hello
stdout 'hello world\n'
Now we’ll add a test that runs this script using
testscript.Run
, as before:
func TestHello(t *testing.T) {
.Run(t, testscript.Params{
testscript: "testdata/script",
Dir})
}
Here’s the result of running go test
:
PASS
It worked! But are we really executing the binary
implemented by hello.Main
, or does there just happen to be
some unrelated program named hello
somewhere on our system?
You can’t be too careful these days.
To find out, let’s change the hello.Main
function to
print something slightly different, and see if that makes the test fail.
This ought to prove that testscript
is really running the
program we think it is:
func HelloMain() int {
.Println("goodbye world")
fmtreturn 0
}
Here’s the result:
> exec hello
[stdout]
goodbye world
> stdout 'hello world\n'
FAIL: testdata/script/hello.txtar:2: no match for `hello world\n`
found in stdout
Proof positive that we’re executing the right hello
program, I think you’ll agree. Let’s also check that returning anything
other than 0 from hello.Main
causes the exec
assertion to fail, as we would expect:
func HelloMain() int {
.Println("hello world")
fmtreturn 1
}
Here’s the result:
> exec hello
[stdout]
hello world
[exit status 1]
FAIL: testdata/script/hello.txt:1: unexpected command failure
One thing to be careful of when defining custom commands in this way
is to remember to call os.Exit
with the result of
testscript.RunMain
. For example, suppose we were to write a
TestMain
like this:
func TestMain(m *testing.M) {
.RunMain(m, map[string]func() int{
testscript"hello": hello.Main,
})
// oops, forgot to use 'status'
}
This looks reasonable, but the status value returned by
RunMain
(which is the exit status of our custom command) is
ignored. Implicitly, we exit with a zero exit status, meaning that the
hello
command would always appear to “succeed”, regardless
of what hello.Main
actually returns.
So if you find that your custom command always succeeds, even when
it’s supposed to fail, check that you have the necessary call to
os.Exit
in TestMain
.
Great. Now we can test that our program succeeds and fails when it should. What about more complicated behaviours, such as those involving command-line arguments?
For example, let’s extend our hello
program to take a
command-line argument, and fail if it’s not provided. Since all the real
work is done in hello.Main
, that’s where we need to make
this change:
func Main() int {
if len(os.Args[1:]) < 1 {
.Fprintln(os.Stderr, "usage: hello NAME")
fmtreturn 1
}
.Println("Hello to you,", os.Args[1])
fmtreturn 0
}
This program now has two behaviours. When given an argument, it should print a greeting using that argument, and succeed. On the other hand, when the argument is missing, it should print an error message and fail.
Let’s test both behaviours in the same script:
# With no arguments, fail and print a usage message
! exec hello
! stdout .
stderr 'usage: hello NAME'
# With an argument, print a greeting using that value
exec hello Joumana
stdout 'Hello to you, Joumana'
! stderr .
The ability to define and run custom programs in this way is the key
to using testscript
to test command-line tools. We can
invoke the program with whatever arguments, environment variables, and
supporting files are required to test a given behaviour. In this way we
can test even quite complex behaviours with a minimum of code.
And that’s no surprise, because testscript
is derived directly
from the code used to test the Go tool itself, which is probably as
complex a command-line tool as any. It’s part of the very handy
go-internal
repository:
https://github.com/rogpeppe/go-internal
Checking the test coverage of scripts
One especially neat feature of testscript
is that it can
even provide us with coverage information when testing our
binary. That’s something we’d find hard to do if we built and executed
the binary ourselves, but testscript
makes it work
seemingly by magic:
go test -coverprofile=cover.out
PASS
coverage: 100.0% of statements
Since our hello
script executes both of the two possible
code paths in hello.Main
, it covers it completely. Thus,
100% of statements.
Just to check that this is really being calculated properly, let’s try deliberately reducing the coverage, by testing only the happy path behaviour in our script:
# With an argument, print a greeting using that value
exec hello Joumana
stdout 'hello to you, Joumana'
! stderr .
We’re no longer causing the “if no arguments, error” code path to be executed, so we should see the total coverage go down:
coverage: 60.0% of statements
Since we’ve generated a coverage profile (the file
cover.out
), we can use this with the
go tool cover
command, or our IDE. This coverage profile
can show us exactly which statements are and aren’t executed by
tests (including test scripts). If there are important code paths we’re
not currently covering, we can add or extend scripts so that they test
those behaviours, too.
Test coverage isn’t always the most important guide to the quality of
our tests, since it only proves that statements were executed,
not what they do. But it’s very useful that we can test
command-line tools and other programs as binaries using
testscript
, without losing our test coverage
statistics.
Comparing output with
files using cmp
Let’s look at some more sophisticated ways we can test input and
output from command-line tools using testscript
.
For example, suppose we want to compare the program’s output not against a string or regular expression, but against a prepared file that contains the exact output we expect. This is sometimes referred to as a golden file.
We can supply a golden file as part of the script file itself,
delimiting its contents with a special marker line beginning
and ending with a double hyphen (--
).
Here’s an example:
exec hello
cmp stdout golden.txt
-- golden.txt --
hello world
The marker line containing golden.txt
begins a file
entry: everything following the marker line will be written to
golden.txt
and placed in the script’s work directory before
it starts executing. We’ll have more to say about file entries later in
this series, but first, let’s see what we can do with this
file.
The cmp
assertion can compare two files to see if
they’re the same. If they match exactly, the test passes. If they don’t,
the failure will be accompanied by a diff showing which parts didn’t
match.
If the program’s output doesn’t match the golden file, as it won’t in this example, we’ll see a failure message like this:
> exec hello
[stdout]
hello world
> cmp stdout golden.txt
--- stdout
+++ golden.txt
@@ -1,1 +0,0 @@
-hello world
@@ -0,0 +1,1 @@
+goodbye world
FAIL: testdata/script/hello.txtar:2: stdout and golden.txt differ
Alternatively, we can use !
to negate the comparison, in
which case the files must not match, and the test will fail if
they do:
exec echo hello
! cmp stdout golden.txt
-- golden.txt --
goodbye world
The first argument to cmp
can be the name of a file, but
we can also use the special name stdout
, meaning the
standard output of the previous exec
. Similarly,
stderr
refers to the standard error output.
If the program produces different output depending on the value of
some environment variable, we can use the cmpenv
assertion.
This works like cmp
, but interpolates environment variables
in the golden file:
exec echo Running with home directory $HOME
cmpenv stdout golden.txt
-- golden.txt --
Running with home directory $HOME
When this script runs, the $HOME
in the
echo
command will be expanded to the actual value of the
HOME
environment variable, whatever it is. But because
we’re using cmpenv
instead of cmp
, we
also expand the $HOME
in the golden file to the
same value.
So, assuming the command’s output is correct, the test will pass.
This prevents our test from flaking when its behaviour depends on some
environment variable that we don’t control, such as
$HOME
.
More matching:
exists
, grep
, and -count
Some programs create files directly, without producing any output on
the terminal. If we just want to assert that a given file
exists as a result of running the program, without worrying
about the file’s contents, we can use the exists
assertion.
For example, suppose we have some program myprog
that
writes its output to a file specified by the -o
flag. We
can check for the existence of that file after running the program using
exists
:
exec myprog -o results.txt
exists results.txt
And if we are concerned about the exact contents of the
results file, we can use cmp
to compare it against a golden
file:
exec myprog -o results.txt
cmp results.txt golden.txt
-- golden.txt --
hello world
If the two files match exactly, the assertion succeeds, but otherwise it will fail and produce a diff showing the mismatch. If the results file doesn’t exist at all, that’s also a failure.
On the other hand, if we don’t need to match the entire file, but
only part of it, we can use the grep
assertion to match a
regular expression:
exec myprog -o results.txt
grep '^hello' results.txt
-- golden.txt --
hello world
A grep
assertion succeeds if the file matches the given
expression at least once, regardless of how many matches there
are. On the other hand, if it’s important that there are a specific
number of matches, we can use the -count
flag to
specify how many :
grep -count=1 'beep' result.txt
-- result.txt --
beep beep
In this example, we specified that the pattern beep
should only match once in the target file, so this will fail:
> grep -count=1 'beep' result.txt
[result.txt]
beep beep
FAIL: beep.txtar:1: have 2 matches for `beep`, want 1
Because the script’s work directory is automatically deleted after
the test, we can’t look at its contents—for example, to figure out why
the program’s not behaving as expected. To keep this directory around
for troubleshooting, we can supply the -testwork
flag to
go test
.
This will preserve the script’s work directory intact, and also print
the script’s environment, including the WORK
variable that
tells us where to find that directory:
--- FAIL: Test/hello (0.01s)
testscript.go:422:
WORK=/private/var/folders/.../script-hello
PATH=...
...
That’s it for Part 2; in Part
3, we’ll find out more about this mysterious txtar
format, and we’ll also learn how to supply standard input to
programs running in test scripts. Don’t miss it!
Previous: Test scripts in Go
Next: Files in test scripts