Run Time Tester Requirements Document

Transcription

1 Run Time Tester Requirements Document P. Sherwood 7 March 2004 Version: 0.4 Status: Draft After review 2 March 2004 Contents 1 Introduction Scope of the Requirements Document Organisation of the Requirements Document Overview Acronym Expansion and Definitions References General Description Perspective User Characteristics Scenarios User Actions Detailed Requirements Functional Requirements Configuration Requirements

2 3.3 Requirements by Job Group Introduction 1.1 Scope of the Requirements Document This documents specifies the requirements for a the Run Time Tester (RTT) software. It describes the purpose of the program, and the ways it is to be used. The interactions with the Atlas software and management tool (CMT) is described. The necessary environment, and how to interact are specified. It is intended to be the main input to the design of the program. 1.2 Organisation of the Requirements Document In the remainder of this section, the purpose of the program is described, and acronym definitions and references are given. Section 2 describes how the RTT is to be used, and its relationship to CMT. Section 3 gives the detailed functional and configurational requirements for the RTT. 1.3 Overview The RTT is a facility for running a set of standard Athena jobs, performing actions on the job results, and publishing information about those results. The facility should be able to submit jobs to different computing resources (in the first instance to a PBS batch queue manager, or run them interactively). The actions to take after the jobs have completed should be easily modified. They should be appropriate to the jobs run, and the results should be easily accessible. The RTT should be setup so that it performs generic actions (run jobs; run tests; run post job actions; report results). The specific actions to be performed will change with time. Although some specific actions need to come with the RTT from the start, special attention should be paid to making it straightforward to modify or remove existing specific tasks, and to add new tasks. 1.4 Acronym Expansion and Definitions ATLAS An experiment at the CERN Large Hadron Collider. 2

3 Athena The ATLAS offline software framework CMT The configuration and build program used to build ATLAS code. RTT Run Time Tester a facility to run ATLAS test jobs. Regression test A test that establishes whether code produces the same results it did at some determined time in the past. Test run One cycle of RTT activity - job submission, tests and other post job activities, results publication - according to how the RTT is configured, for a fixed set of jobs. NICOS ATLAS manager for ATLAS nightly builds. TDS Athena transient data store. Run time environment The conditions needed to run ATLAS jobs. This includes setting up symbols ( e.g. Athena), environment variables ( e.g. LD LIBRARY PATH) and making available necessary auxilliary files. Job group A classification of jobs used by the RTT. Jobs within a group are processed in the same manner when setting up the run time environment, and during the postrun processing. Examples of job groups are: HelloWorld, Reconstruction, Artemis, Atlfast. Release A set of software package tags which refer to versions of software packages which work together. For software to be part of a release, it is tagged (currently this means is given a CVS tag), and is declared (currently using the Tag Collector). Releases occur as numbered releases from time to time, and, as a development aid, once a day (the nightly build). Numbered releases have a long lifetime (typically years), while nightly builds are overwritten every week. 1.5 References CMT in ATLAS: NICOS: 3

4 2 General Description 2.1 Perspective Atlas code is divided into interdependent packages. The code from the packages is compiled into shared libraries, which are loaded on request at run time. A release is a set of packages tags corresponding to versions of the libraries which work together. When an Athena job is run, it is passed a job options file which gives, among other things, the names of the shared libraries to be loaded. CMT sets up environment variables to ensure that the correct versions of the different libraries are loaded. The user tells CMT which release to use by placing this information in her home requirements file and sourcing her home setup.sh script. The RTT uses CMT to establish the connection between the release information and the actual libraries and executables used, and to set up symbols and paths so that when the command athena is typed, the correct version of the executable is used. The RTT uses job option files to specify which shared libraries are to be used, and other Athena algorithm specific information. Details of configuration file information is given in section 3.2. In short, the RTT uses CMT by performing the same operations as those performed by a user using CMT manually in that it is capable of creating a home CMT requirements file and setup script, and by sourcing this script. The RTT assumes certain runtime behaviour. An example is that RTT must know the mechanisms for locating job option files. The RTT should be part of the ATLAS release to accommodate such dependencies. An RTT test run is to be made up of the following steps: job submission, job checking, other post job activities, and reporting. The RTT will use the concept of job group to determine which actions are taken for a particular job. For example, regression tests will be run on jobs in the Atlfast group, while a root macro will be run on the output of jobs in the Artemis group. During the post processing step, job logs, and possibly other files - such as histogram and ntuple files - will be made available, as will the results of any tests that are run. To ensure that the RTT can be used a wide variety of platforms, and to conform to the ATLAS reccomendations on computing languages, the RTT is to be written in Python. 4

5 2.2 User Characteristics Users will pass information to the RTT via parameters in congiguration files. How the RTT reponds to such information is determined by the RTT code. This means that there are three kinds of user. The first kind of user runs a preconfigured RTT in order to obtain test results. The second kind of user is capable of modifying configuration files, and taking action to ensure that whatever actions he is requesting of thee RTT make sense. For example, if he specifies that a job use a particular data set, he must ensure that that data set is available. The third kind of user is capable of understanding the python code, and modifying it to obtain desired behaviour. Such a user can tune the reaction of the RTT to configuration file parameters. 2.3 Scenarios A developer who wishes to have her code tested automatically and at regular intervals, using the latest ATLAS release. Her job-option files are included in the job-configuration file. The test runs are initiated by a cron job which is run once a day. This job waits for the nightly build to complete, and then starts job submission. The release is set to the current nightly release. From this point onwards she can consult the results displayed on the web, to ascertain both the state of Atlas job running machinery (do the jobs run to completion? if not what errors are reported?), and the state of her own code. She can download any files produced by her test jobs which are made available by the RTT. A user who wishes to consult the status of the code for a particular release, in order to see if his jobs will run with that release. He looks at the RTT results on the web to see which test jobs ran for the release he is interested in. Release officials who wish to run tests prior to making a release public. Developers who have checked-out and modified code, and who wish to test it before committing the new material to the code repository. A user wishes to check that the installation of the ATLAS code at a new site gives the same results as at an original site. They obtain access to the database used to hold reference material created the the original site (perhaps by copying the database, if this is stored as a python database file), and the original configuration files, Those parts of the files holding site-specific information (specification of directories, batch queues) are modified appropriately. 5

6 The RTT is then used to submit the jobs, and the regression tests show if the new environment causes the jobs to act differently, 2.4 User Actions As part of the task of setting up and running jobs, the ATLAS run time environment (environment variables, search paths) needs to be established. This will be done using CMT. When the user specifies that she wishes to work with an existing release, the RTT will perform the necessary CMT actions. When working with checked-out code, the user will inform RTT that this is the case (this is an option in the configuration file), and will perform the CMT steps manually. The RTT will use CMT to create a run-time environment in the following manner: 1. Source the CMT setup.sh, providing a Home requirements file. Among other things this action informs CMT of the ATLAS release to use. 2. Source setup.sh for the AtlasRelease package. These steps are to be performed by CMT when it is used to run jobs from a release. When RTT is to work on checked-out code, the developer will perform these actions, and build her code. Typically she will do this by sourcing her home CMT scripts, and using the TestRelease mechanism. The RTT will have two configuration files, one to specify the RTT behaviour, and one to specify job information. Examples of such information are the names of job option files, batch queues to submit to, and data sets to run on. The user will start the RTT with a short script that specifies where the RTT code is, and where the RTT configuration file is. This provides a high degree of flexibility: the user may have a number of different configuration scripts, to provide different RTT customisation. She does not need to own (or check-out) a copy of the scripts. The minimum material she needs to run the RTT is A very short script specifying where to find the RTT (this will usually be the ATLAS install area). A job configuration file. 6

7 3 Detailed Requirements 3.1 Functional Requirements RTT Requirement 1 To ensure that the RTT be available on a wide variety of platforms, and to conform to the ATLAS recommendations on which computing languages are to be used, the RTT is to be written in Python. RTT Requirement 2 The RTT should be runnable in an automatic manner ( e.g. as a cron job), or on demand whereby a client configures the RTT according to his needs and resources. RTT Requirement 3 It should be possible to have the RTT run at regular intervals, in such a way that the results are viewable via a web browser. In particular, the RTT will be run once every 24 hours, after the Atlas nightly builds have finished. The nightly jobs should use the most up-to-date nightly release. The results and any files produced by the jobs should be made available to clients at a predefined URL. RTT Requirement 4 The RTT should be usable with binaries and libraries built from code a developer has checked-out and modified. By definition, this code is not in a release. RTT Requirement 5 RTT run results should be viewable indexed by release. RTT Requirement 6 Release results for nightly builds should be kept until the ATLAS nightly build infrastructure (currently NICOS) provides a new build for that night. The RTT results should then be be overwritten. RTT Requirement 7 Release results for numbered releases should not be overwritten. RTT Requirement 8 The RTT should be in the ATLAS release - this is to ensure that the RTT job option files are installed during the ATLAS builds. RTT Requirement 9 The RTT should be written in a high level scripting language. RTT Requirement 10 The RTT should be part of the ATLAS release. It should have a src directory for the scripts, and a share directory for job option files, and other test fixtures. 7

8 RTT Requirement 11 The RTT should provide job option files for a minimum set of test jobs in its share directory. RTT Requirement 12 The RTT job option files should conform to the CMT conventions for locating job option files. RTT Requirement 13 Job logs and job option files should be made available for downloading via a web browser. RTT Requirement 14 CPU and memory usage information should be made available sparately from the log file, if this information is available in the log file in a standard form. RTT Requirement 15 A dump of the environment variables used for running the jobs will be made available. RTT Requirement 16 Results should be reported as HTML files. This makes the results calculated at remote sites easily accessible. Both remote and local files should be readable with a web browser. RTT Requirement 17 It should be possible to define stable data sets. This is essential for regression tests. Currently, these data sets should be DC1 Zebra data. 3.2 Configuration Requirements The following items should be specified in ASCII configuration files: RTT Requirement 18 The release to use ( e.g , atlrel 3). RTT Requirement 19 The release used to generate reference material. RTT Requirement 20 The build (opt, debug) RTT Requirement 21 If reference material is to be stored. RTT Requirement 22 The location where reference material is to be stored. RTT Requirement 23 The location where result information is to be sent. 8

9 RTT Requirement 24 The location of a work area needed by RTT. RTT Requirement 25 Where jobs should be submitted (PBS, interactive on local machine...) RTT Requirement 26 Which job options files to use. RTT Requirement 27 Queues to submit to for each job - relevant for batch queues. RTT Requirement 28 Data sets to be used for each job. 3.3 Requirements by Job Group The RTT is a general purpose facility to which one can add functionality, which is organised by job group. As a starting point, the functionality requirements for two job groups Atlfast and Artemis are stated. RTT Requirement 29 The Atlfast test jobs run the SimpleTest Athena Algorithm. The results of this algorithm are returned in a comparison file. The RTT will provide the code to examine the comparison file and display either an error condition, or a success condition. RTT Requirement 30 The Artemis test jobs will provide histogram files that are to be displayed. The RTT will run macros to format and write out plots as post script files, and make these files available for downloading or display with a web browser. 9