Guide to the LSB Test Suites

The aim of this document is to help people understand how the installation and setup scripts for the TET based LSB test suites work. Also described is how the test suites and the associated framework are stored in CVS and way in which the source based tarballs can be generated.

Parts of this document may be unclear, out of date or simply wrong. I encourage you to send me feedback on problems you may find or other suggestions. In many cases looking at the shell scripts themselves will give you the best detailed description, but hopefully the information here will act as a good outline to how things are meant to work at a higher level.

Building and running the source based LSB test suites

The source based LSB test suites are distributed as a single tarball (tet_vsxgen_<version>.tar.gz) containing TET and VSXgen and multiple optional tarballs one for each test suite (eg VSX-PCTS, LSB-FHS, VSTHlite).

Installation Process

The install.sh script handles the installation of the TET & VSXgen framework and the individual test suites. This script must be run as the root user. A summary of the process is:

Performs sanity checking of the system (eg byacc must be installed to build the test suites).
Creation of necessary users and groups on the system. If users already exist but have been configured for a different installation directory then this is also modified.
Installation of TET/VSXgen. This also includes the additional framework of scripts used to semi-automate the later setup and running of the tests.
Installation of individual test sets. The test set tarballs are all unpacked in $TET_ROOT/test_sets where $TET_ROOT is set by the user during the installation process. The tarballs are designed such that they do not contain conflicting files.
Configures setup scripts (setup.sh, profile and test_sets/profile) for the installation directory selected by the user.
Display of various licensing information for the TET/VSXgen and the test suites installed.

Setup Process

The setup of the test suites must be done as the vsx0 user. The setup.sh is the starting point for the setup and buidling and running of the installed test suites.

A summary of setup.sh is:

Setup of environment variables and helper shell functions for later use.
Compilation and installation (locally) of TET
Calling of setup_testsets.sh to perform further setup.

setup_testsets.sh is responsible for performing the rest of the setup of the test suite and VSXgen framework and the building of the VSXgen framework, but notably not the building of the test programs.

The setup framework was designed to allow for the configuration of TET/VSXgen based test suites both which have already been written such as VSX-PCTS and LSB-FHS as those which are still being more actively developed and tested such LSB-OS and Users/Groups. The configuration demands are dependent on which specific test suites have been installed on the current system. Prior to this system, specific scripts for each test suite had to be written which contained a lot of common code, but required separate maintenance. To allow for test suite specific configurations the setup_testsets.sh looks for and calls test set specific scripts and various stages of the setup of the test suites. The points at which this is done are:

Prior to the running of the config.sh script (VSXgen supplied script).
After the running of the config.sh script.
After the compilation of the VSXgen framework.
After the built-in automated parameterisation of the test suites.

Refer to the description of the generic structure of the test suite tarballs for more information on how the test suite specific setup scripts are stored and searched for.

A summary of the process of the setup_testsets.sh script:

Sanity checking, and initialisation such as cleanup of locks possibly left over from previously failed setup attempts.
Runs test suite specific scripts.
Execution of VSXgen config.sh script. Installation of Linux specific C routines. Automated 'fixing' of some file produced by the the VSXgen configuration procedure.
Runs test suite specific scripts.
Builds the VSXgen framework (libraries and helper programs).
Runs test suite specific scripts.
Parameterisation common to all test suites is performed.
Runs test suite specific scripts (mainly to perform test suite specific parameterisation).
Optionally allows the user to proceed to building and executing the test suites. If selected then the script calls run_testsets.sh

Build and Execution of the test suites

run_testsets.sh optionally builds (or rebuilds) the test programs that comprise the test suites. It then exeecutes the test programs. A summary of the test results is displayed and the tests which failed are optionally re-executed in another run.

The run_testsets.sh script should not be run directly but instead the script rerun_tests should be called which sets up the environment correctly. This is especially true after a reboot of the machine as features such as loopback devices often need to be reconfigured again.

CVS Storage of the Test Suites

The source code for the test suites are stored in the test_sets module of the LSB project on Source Forge. The README file in the root directory of the module has a description of the directory layout.

In addition to the test suite code, the test_sets module also contains code for the installation scripts, and the VSXgen harness. The VSXgen harness has been modified from the original source. The directory hierarchy containing the VSXgen source also contains intermingled with it part of the installation script framework.

None of the TET code is checked in to CVS and it is used by the test suites unmodified with the single exception of a configured Makefile. The TET code was obtained from the 3.3f release. There are also a few necessary scripts which are not checked in to CVS (I thought they were, but looking at CVS now they aren't).

Creating the TET/VSXgen tarball

This tarball is logically a merge between TET and the contents of various directories in the harness directory of the module. The root directory of the tarball is the same as that of TET. As this process is fairly involved in practice when updating the tarball I haven't built it from the original TET source in quite a while, but instead I unpack the tarball and update the needed files from a checked out CVS tree and then rebuild the tarball. It is also necessary to set the execution permission bit of a few scripts as they were initially checked into CVS without it and it doesn't seem possible to change it now.

What we really should have here is a script which generates the tarball automatically entirely (perhaps with TET being an exception) from CVS. This would ensure we don't accidentally get out of sync and that CVS is always updated properly.

Creating the test suite tarballs

These are generated with the root directory of the tarball being the subdirectory of the test_sets module for each test suite. They should not contain the CVS directories. The cvs export command is very useful for checking out the latest version for creating the tarballs.

Structure of the individual test suites in CVS

cyeoh@samba.org