Testing
##########

This section  describes the testing suite for binder.
The testing suite has two implementations, both using the same set of tests
located in the ``test`` subdirectory.
The first implementation is located inside the script ``build-and-run-tests.py`` and is
designed to be used for the builds inside the LLVM source tree. This implementation 
is briefly described above.
The second implementation uses ``cmake/ctest`` and is used in the CI with the external LLVM 
installation. To configure this testing suite, use

.. code-block:: bash

  cmake ...  -DBINDER_ENABLE_TEST=ON ...

Multiple python versions can be and should be used in parallel. The versions are set as a 
comma-separated list passed by the ``BINDER_TEST_PYTHON_VERSIONS`` option. The default list is ``0,2,3``.
The Python version \"0\" corresponds to a \"plain diff\" of the output vs. reference. 
The versions of Python2/Python3 will be the first versions found by cmake, see 
https://cmake.org/cmake/help/latest/module/FindPython.html and https://cmake.org/cmake/help/latest/module/FindPython3.html
for your cmake version. 
To use specific python versions one can use the following

.. code-block:: bash
  
  cmake .... -DBINDER_TEST_PYTHON_VERSIONS=0,2.7.15,3.8.0,3.7.0 ...

The generated codes will be compiled and loaded using the corresponding interpreter.
With an option ``-DBINDER_MOCK_TEST=ON`` one can mock the code generation by binder. 
In this case the reference codes will be used in the python tests.