View on GitHub

OCamlverse

Documenting everything about OCaml

Page Contents

Edit

Unit Testing

Definitions

  • Unit Tests: tests involving specific functions. The function is given a specific input, and the output is compared to the expected output. This allows us to have guarantees about the behavior of code even after we refactor it, or even before we write the code itself.
  • Property Tests: tests which involve specifying the general behavior of a function - given random input, what property should its output have - rather than choosing specific input and output combinations, as we do in unit testing.
  • Expect tests: a form of unit testing. Rather than coming up with specific inputs and their respective outputs, the programmer tries different inputs and gets the dumped state of the function output, which is incorporated into the test itself. This requires printers for every type involved, but allows for exploration and regression testing. See this article.
  • Cram Tests: a form of unit testing that involves using some shell, such as the OCaml toplevel. All input and output is included in the test itself, and the test plays back the input and compares the output to what is expected.

Unit Testing Frameworks

OCaml has many unit testing frameworks to choose from:

  • dune has a useful page explaining how to integrate all unit test options. dune also supports its own simplified version of expect tests using .expect files, which may be enough for many users. dune runtest and dune promote are usually all that’s needed to run and update expect tests (including ppx_expect). dune also includes support for running Cram tests.
  • expect-test: expect tests utilising the ppx system. As discussed here. Tightly coupled with dune. Tests can be in the same files as the code itself or placed separately.
  • ppx_inline_test: Inline unit tests using the ppx system. Works well in concert with ppx_expect.
  • Alcotest: a lightweight, easy to use, and popular test framework. Nice colored output for results.
  • OUnit: an older but still competitive unit test framework for OCaml.
  • QCheck: a library that allows you to perform property tests. Sub-libraries are provided to integrate with both Alcotest and OUnit
  • qtest: write simple inline pragmas within your code to generate unit tests. qtest can also integrate QCheck tests to have random testing, too.
  • mdx: Execute OCaml code blocks inside Markdown documentation files. Also supports Cram tests.
  • Kaputt: a comprehensive testing framework.
  • TestSimple: a lightweight unit testing framework compatible with the Test Anything Protocol.
  • Cucumber.ml: Behavior Driven Development using Cucumber.
  • qcstm: State machine framework for testing imperative code. Built on QCheck. Paper Video
  • Monolith: Specify the behavior of a program and perform random or fuzz testing on it automatically. Paper

Fuzzing

Fuzzing involves hitting your program with random or adversarially optimized input until you find ill-defined behavior.

Fuzz Testing definition