1. have arrangements to ensure its personnelare free from any commercial, financial andother pressures which might adversely affectthe quality of their work;

2. be organized in such a way that confidencein its independence of judgement and integrityis maintained at all times;...'

1. another implementation with better coverage;

2. a set of implementations from different suppliers; or

3. an implementation which is designed to be a reference implementation.

It is recognised that for some IT&T standards there may be no alternative to using a normal commercial implementation as a reference implementation against which to validate the MOT. In such cases, the publication of the identity of the reference implementation (in order to be open about the nature of the MOT validation conducted) may inadvertently give commercial advantage to the supplier.

The decision to use a normal commercial implementation as a reference implementation, and the choice of which commercial Implementation to use in this way, are decisions to be made by the laboratory.

In some cases, it may be necessary to use multiple reference implementations for MOT validation, in order to ensure that adequate coverage of the MOT behaviour is checked. This arises because a given commercial implementation may only support a subset (or "profile") of the relevant specification(s). This may be acceptable as a temporary solution, particularly if the market Is primarily interested in that subset, but is inadequate as a longer term solution if the testing service is to cover the specification(s) in full.

1. participation in proficiency testing or otherinterlaboratory comparisons;'

1. reference procedures for... verification andmaintenance of equipment;

1. m)

   reference to procedures for... verificationand maintenance of equipment;

'There shall be effective separation betweenneighbouring areas when the activities therein are incompatible. '

Records of MOT validation should include clarification of the coverage of the validation with respect to the test suite being used.

If a testing service is such that there are specified test cases that are run, then if the discrepancies seem to be Isolated to particular test cases, then only those test cases need be suspended from use. If, however, the concept of test cases does not apply or if the discrepancies indicate a general problem with the MOT or test tool, then the MOT or test tool needs to be suspended.

'All equipment shall be properly maintained.... '

In ISO/IEC Guide 25, subclause 8.2 also requires the following:

'Any item of equipment which has been subject to... mishandling, or which gives suspect results, orhas been shown by verification or otherwise to bedefective shall be taken out of service, ... until ithas been repaired and shown by... verification ortest to perform satisfactorily. The laboratory shallexamine the effect of this defect on the previous...tests. '

1. h)

   details of maintenance carried out to-dateand planned for the future;

2. i)

   history of any damage,modification or repair. '

1. computer software is documented andadequate for use;

2. procedures are established andimplemented for protecting the integrity ofdata;...

3. computer and automated equipment ismaintained to ensure proper functioning andprovided with the environmental and operatingconditions necessary to maintain the integrityof... test data;

In compiler testing, instrumented reference implementations have also been used to check the coverage of test suites.

See 4.A for requirements concerning the use of an ordinary commercial implementation as a reference implementation.

For results analysis tools, the use of a subset of the test suite would be replaced by the use of the conformance log produced as a result of running a subset of the test suite.

For compiler testing, a simple regression test against a validated compiler would suffice for re-validation following minor changes.

a) the requirements of this guide are complied with;

b) computer software is documented and adequate for use;

...

d) computer and automated equipment is maintained to ensure proper functioning...'

In some special cases, this requirement will apply to firmware as well as test software.

This requirement also applies to software which is part of a means of validation and which is to be mounted on the system which contains the MOT or test tool to be validated.

Appropriate harmonization agreements should be used, wherever applicable, to check the reliability of MOT and test tool validation results.

There should be a procedure to deal with the discovery that one of the expected results of an MOT validation is erroneous. There should also be a procedure to deal with the situation in which the set of expected results is incomplete (e.g. using those expected results that are available but using analysis from first principles where no expected results exist).

Systems used solely for producing the executable test cases that are to be run need not be validated provided that the executable test cases are themselves validated (i.e. shown to be 'traceable' back to the abstract test suite).

In IT&T, this requirement shall be interpreted as follows.

Checks should be carried out to ensure that the correct version of the MOT is loaded and that there has been no unauthorised access or change to the MOT. The checks should include the execution of a specified subset of the test cases to ensure that the software has loaded correctly and not been corrupted.

The criteria that are applied to the suitability of test methods depend to some extent on the type of testing. Below, there are, firstly, some general criteria which apply to all types of testing; secondly, there are additional criteria which apply only to conformance testing.

Furthermore, repeatability has to be interpreted more flexibly for types of testing that involve making performance measurements. For example, measurements of the time taken to run a specific test case can vary considerably as the result of what seem to be insignificant changes to the environment or parameterization of the software implementation. Thus, the test methods for making performance measurements need to specify the precise environmental conditions to ensure repeatability. The results should be presented so as to avoid giving spurious accuracy. The relevant test methods may involve re-running test cases with a selection of different environmental conditions or parameter settings. In all cases of making performance measurements, the results need to be accompanied by an explanation of the environment in which the test cases were run.

There is no implied time period over which the repeatability should hold true.

Note — this usage of the term 'repeatability' is consistent with the definitions given in ISO 5725, the Guide for determination of repeatability and reproducibility for a standard test method by inter-laboratory tests, and in the International Vocabulary of Basic and General Terms in Metrology.

There is no equivalent of a standardized ICS proforma for some other areas such as compiler testing, although test selection may need to be performed. For Ada compiler testing, for example, there are implicit options available due to limitations of the target hardware upon which the Ada program executes. For instance, if the hardware does not support floating point arithmetic with 20 digit precision, then certain test cases in the test suite are inapplicable and are not processed during testing. Unfortunately, the complete characterization of such inapplicable test cases is not straightforward and therefore rests on analysis by a test operator, rather than upon simple pre-defined rules (or selection by a test tool). Nevertheless, procedures are needed to ensure that the test selection is repeatable, reproducible and objective, otherwise it is probable that such testing is not suitable for accreditation.

This situation can arise, for example, in OSI and telecommunications protocol testing when the testing is to be conducted at the client's site, and to avoid the testing laboratory having to transport the hardware on which the MOT is to run, the client lends the testing laboratory an appropriate computer system on which to mount the MOT software. This computer system may be coupled to a computer network and the client may expect it to be returned after testing in the same state as it was before testing began. In such a case, If test records are left on the system and subsequently accessed by a third party, without the client having accepted responsibility for the test records, then the testing laboratory may be held liable for the breach of confidence.

When the testing laboratory needs to remove such test records from the client's system, care should be taken to try to find a means of deletion which completely removes the records from the system, rather than just flagging them for subsequent removal. To find such a means of deletion may require the client's assistance.

These procedures should not be confused with the ones to be followed when arbitration is called for (i.e. when the client challenges the results produced by the testing laboratory).

a) define and use testing procedures which ensure that any unspecified verdict assignments meet the criteria of repeatability, reproducibility and objectivity with respect to the relevant standard(s); or

b) assign an inconclusive verdict to any unspecified situation until the test suite has been corrected by the maintenance authority for that test suite.

The testing laboratory shall also take appropriate steps to get such (incomplete) test cases revised so that they completely define all allowed verdict assignments.

This is a consequence of ISO/IEC Guide 25 subclauses 10.2 and 13.1 (as quoted above), and the interpretations in 10.B.1,10.B.2 and 13.A.2.

This is a consequence of ISO/IEC Guide 25 subclauses 10.2 and 13.1 (as quoted above), and the interpretations in 10.B.1,10.B.2 and 13.A.2.

It is recognised that if the same test suite is used by other testing laboratories then withdrawal of a test case may only be possible by agreement with the others, to avoid affecting mutual recognition agreements.

In some cases the assignment of verdicts will be made by an MOT or test tool maintained by an organization that is outside the testing laboratory's control. In such cases, the testing laboratory will need to add to its procedures the intervention of a test operator to change provisional verdicts to inconclusive as necessary to meet this requirement.