Module QCheck2
QuickCheck-inspired property-based testing
Introduction
This library takes inspiration from Haskell's QuickCheck library. The rough idea is that the programmer describes invariants that values of a certain type need to satisfy ("properties"), as functions from this type to bool. They also need to describe how to generate random values of the type, so that the property is tried and checked on a number of random instances.
This explains the organization of this module:
Genis used to describe how to generate random values. Auxiliary modulePrintcan be used along withTest.maketo build one's own generator instances.
Testis used to describe a single test, that is, a property of type'a -> boolcombined with an'a Gen.tthat is used to generate the test cases for this property. Optional parameters allow to specify the random generator state, number of instances to generate and test, etc.
💡 If you are migrating from QCheck, check the migration guide below.
Examples
- "
List.rev is involutive" (the test passes socheck_exnreturns()):
let test =
QCheck2.(Test.make ~count:1000
~print:Print.(list int)
Gen.(list int)
(fun l -> List.rev (List.rev l) = l));;
QCheck2.Test.check_exn test;;"All lists are sorted" (false property that will fail):
- QCheck tests this property on random lists and finds a counter-example
- QCheck then looks for the smallest counter-example possible (here
[1; 0]) to help you find the problem (called "shrinking")
let test = QCheck2.(
Test.make
~name:"All lists are sorted"
~count:10_000
~print:Print.(list int)
Gen.(list small_nat)
(fun l -> l = List.sort compare l));;
QCheck2.Test.check_exn test;;
Exception:
test `All lists are sorted` failed on ≥ 1 cases:
[1; 0] (after 5 shrink steps)- Generate 20 random trees using
Gen.fix:
type tree = Leaf of int | Node of tree * tree
let leaf x = Leaf x
let node x y = Node (x,y)
let tree_gen = QCheck2.Gen.(sized @@ fix
(fun self n -> match n with
| 0 -> map leaf nat
| n ->
frequency
[1, map leaf nat;
2, map2 node (self (n/2)) (self (n/2))]
));;
QCheck2.Gen.generate ~n:20 tree_gen;;- since
- 0.18
module Tree : sig ... endA tree represents a generated value and its successive shrunk values.
module Gen : sig ... endA generator is responsible for generating pseudo-random values and provide shrinks (smaller values) when a test fails.
module Print : sig ... endPrinting functions and helpers, used to print generated values on test failures.
module Shrink : sig ... endShrinking helper functions.
module Observable : sig ... endAn observable is a random function argument.
module Tuple : sig ... endUtils on combining function arguments.
type 'f fun_reprUsed by QCheck to shrink and print generated functions of type
'fin case of test failure. You cannot and should not use it yourself. Seefun_for more information.
type 'f fun_=|Fun of 'f fun_repr * 'fA function packed with the data required to print/shrink it.
The idiomatic way to use any
fun_Gen.t is to directly pattern match on it to obtain the executable function.For example (note the
Fun (_, f)part):QCheck2.(Test.make Gen.(pair (fun1 Observable.int bool) (small_list int)) (fun (Fun (_, f), l) -> l = (List.rev_map f l |> List.rev l))In this example
fis a generated function of typeint -> bool.The ignored part
_ofFun (_, f)is useless to you, but is used by QCheck during shrinking/printing in case of test failure.See also
Fnfor utils to print and apply such a function.
val fun1 : 'a Observable.t -> ?print:'b Print.t -> 'b Gen.t -> ('a -> 'b) fun_ Gen.tfun1 obs gengenerates random functions that take an argument observable viaobsand map to random values generated withgen. To write functions with multiple arguments, it's better to useTupleorObservable.pairrather than applyingfun_several times (shrinking will be faster).- since
- 0.6
val fun2 : 'a Observable.t -> 'b Observable.t -> ?print:'c Print.t -> 'c Gen.t -> ('a -> 'b -> 'c) fun_ Gen.tSpecialized version of
fun_naryfor functions of 2 arguments, for convenience.- since
- 0.6
val fun3 : 'a Observable.t -> 'b Observable.t -> 'c Observable.t -> ?print:'d Print.t -> 'd Gen.t -> ('a -> 'b -> 'c -> 'd) fun_ Gen.tSpecialized version of
fun_naryfor functions of 3 arguments, for convenience.- since
- 0.6
val fun4 : 'a Observable.t -> 'b Observable.t -> 'c Observable.t -> 'd Observable.t -> ?print:'e Print.t -> 'e Gen.t -> ('a -> 'b -> 'c -> 'd -> 'e) fun_ Gen.tSpecialized version of
fun_naryfor functions of 4 arguments, for convenience.- since
- 0.6
val fun_nary : 'a Tuple.obs -> ?print:'b Print.t -> 'b Gen.t -> ('a Tuple.t -> 'b) fun_ Gen.tfun_nary tuple_obs gengenerates random n-ary functions. Arguments are observed usingtuple_obsand return values are generated usinggen.Example (the property is wrong as a random function may return
false, this is for the sake of demonstrating the syntax):let module O = Observable in Test.make (fun_nary Tuple.(O.int @-> O.float @-> O.string @-> o_nil) bool) (fun (Fun (_, f)) -> f Tuple.(42 @:: 17.98 @:: "foobar" @:: nil))Note that this particular example can be simplified using
fun3directly:let module O = Observable in Test.make (fun3 O.int O.float O.string bool) (fun (Fun (_, f)) -> f 42 17.98 "foobar")- since
- 0.6
module Fn : sig ... endUtils on generated functions.
Assumptions
val assume : bool -> unitassume condchecks the preconditioncond, and does nothing ifcond=true. Ifcond=false, it interrupts the current test (but the test will not be failed).⚠️ This function must only be used in a test, not outside. Example:
Test.make (list int) (fun l -> assume (l <> []); List.hd l :: List.tl l = l)- since
- 0.5.1
val (==>) : bool -> bool -> boolb1 ==> b2is the logical implicationb1 => b2ienot b1 || b2(except that it is strict and will interact better withTest.check_exnand the likes, because they will know the precondition was not satisfied.).⚠️ This function should only be used in a property (see
Test.make), because it raises a special exception in case of failure of the first argument, to distinguish between failed test and failed precondition. Because of OCaml's evaluation order, bothb1andb2are always evaluated; ifb2should only be evaluated whenb1holds, seeassume.
val assume_fail : unit -> 'aassume_fail ()is likeassume false, but can take any type since we know it always fails (likeassert false). This is useful to ignore some branches iniformatch.Example:
Test.make (list int) (function | [] -> assume_fail () | _::_ as l -> List.hd l :: List.tl l = l)- since
- 0.5.1
Tests
A test is a universal property of type foo -> bool for some type foo, with an object of type foo Gen.t used to generate values of type foo.
See Test.make to build a test, and Test.check_exn to run one test simply. For more serious testing, it is better to create a testsuite and use QCheck_runner.
type 'a stat= string * ('a -> int)A statistic on a distribution of values of type
'a. The function MUST return a positive integer.
module TestResult : sig ... endResult of running a test
module Test_exceptions : sig ... endmodule Test : sig ... endA test is a pair of an generator and a property thar all generated values must satisfy.
Sub-tests
exceptionNo_example_found of stringRaised by
find_exampleandfind_example_genif no example was found.
val find_example : ?name:string -> ?count:int -> f:('a -> bool) -> 'a Gen.t -> 'a Gen.tfind_example ~f genusesgento generate some values of type'a, and checks them againstf. If such a value is found, it is returned. Otherwise an exception is raised.⚠️ This should only be used from within a property in
Test.make.- parameter name
Description of the example to find (used in test results/errors).
- parameter count
Number of attempts.
- parameter f
The property that the generated values must satisfy.
- raises No_example_found
If no example is found within
counttries.
- since
- 0.6
val find_example_gen : ?rand:Stdlib.Random.State.t -> ?name:string -> ?count:int -> f:('a -> bool) -> 'a Gen.t -> 'aToplevel version of
find_example.find_example_gen ~f genis roughly the same asGen.generate1 @@ find_example ~f gen.- parameter rand
the random state to use to generate inputs.
- raises No_example_found
if no example was found within
counttries.
- since
- 0.6
Migration to QCheck2
QCheck2 is a major release and as such, there are (as few as possible) breaking changes, as well as functional changes you should be aware of.
Minimal changes
Most of your QCheck (v1) code should be able to compile and run the first time you upgrade your QCheck version to a QCheck2-compatible version. However you may need to do the following minimal changes:
QCheck.Test.makereturn type was changed toQCheck2.Test.tto be able to run both QCheck and QCheck2 tests together. This is transparent if you used type inference, but if you explicitly usedQCheck.Test.tyou will need to change it toQCheck2.Test.t.
Recommended changes
Now you want to actually start using the QCheck2 features (most importantly: free shrinking!). To get started, change all your QCheck references to QCheck2 and follow the compiler errors. Below are the most common situations you may encounter:
- as shrinking is now integrated, several function arguments like
~shrinkor~revhave been removed: you can remove such reverse functions, they will no longer be necessary. - accessor functions like
QCheck.arbitrary.genhave been renamed to consistent names likeget_gen. QCheck.map_keep_inputhas been removed: you can usemapdirectly.Gen.tis no longer public, it is now abstract: it is recommended to use generator composition to make generators.Gen.make_primitivewas added to create generators with finer control (in particular of shrinking).