Year 2000 Test Criteria WG2-0001 CSSA/PMI Y2k SIG - WG2 Last update 1998-07-20 http://www.cinderella.co.za/criteria.txt 1. Test Procedures Several test scenarios have been developed as a result of problems identified with the year 2000. This limited set of tests cannot prove a Component/System to be Year 2000 compliant, but using them will help identify several frequently observed problems. These test procedures are written as general instructions. Specific knowledge of the systems or components under test is required in addition to apply these test cases. The following test procedures provide step by step instructions for performing each test. The results should be recorded step by step as the test is performed to insure accurate records of the test are documented on the "Year 2000 Test Report" form. 1.1 Critical Date Values for Year 2000 Testing The following dates can be tested for proper operation: 1. 0000-00-00 Special Value 2. 1998-12-31 Rollover, Reboot 3. 1999-01-01 Special Value 4. 1999-09-09 Special Value 5. 1999-12-31 Special Value, Rollover, Reboot 6. 2000-01-01 Day of Week, Day of Year 7. 2000-02-28 Rollover, Reboot 8. 2000-02-29 Rollover, Reboot, Day of Week 9. 2000-03-01 Day of Week 10. 2000-12-31 Rollover, Reboot, Day of Week, Day of Year 11. 2001-01-01 Day of Week, Day of Year 12. 2027-12-31 Rollover, Reboot, Day of Week, Day of Year 13. 1980-01-01 RTC Reset to zero 14. 1980-01-04 RTC Reset to zero (europe) 15. 9999-99-99 Special value 1.2 Rollover, Reboot, Day of Week Tests The rollover test checks for proper handling of the date transition from 1999 to 2000 without manual intervention. Based on actual tests several different results have been observed as examples of incorrect handling of the transition from 1999 to 2000. Many systems used 2-digit dates and the result may be a rollover to year 100, sometimes the 19 is assumed and the result is the year 19100. For other unknown reasons the years 2001, 2028, and non-printable characters have been observed. The effect of incorrect date calculations may include negative numbers. The reboot test checks for correct date & time storage during power cycles of the system. The system may function correctly when the time is set ahead, but revert to another time and date when the power is cycled. Many PC's revert to 1980 or 1984 when rebooted after the year 2000. The day of the week may be incorrectly calculated. Systems should display the day of the week of January 1, 2000 as Saturday, not Monday which may mean January 1, 1900. 1.2.1 Rollover - 1999 to 2000 - Power on Test: Set the date to 31 Dec. 1999. Set the time to 23:59 (11:59 p.m.). Observe the system date after 00:00 am Expected Result: The system clock advances into the year 2000 and continues normally. 1.2.2 Day of Week Test: The clock is set to 1 Jan 2000. Observe the system day of week display. Expected Result: The system displays the day of week as Saturday. (1 Jan 1900 was a Monday) 1.2.3 Reboot - Date retention Test: Set the date to 1 Jan 2000. Power down the system. Power up the system. Observe the system date Expected Result: The system clock still displays the year 2000 and operates normally. Note: Many personal computers fail either both or the first of these tests and reset themselves to 04 January 1980, or some other past date, whenever they reboot, if the CMOS real time clock says the year is 00. 1.2.4 Rollover - 1999 to 2000 - Power Off The procedure specifies 10 minutes before mid-night, a smaller time may be appropriate, however be sure you can shutdown the system before the rollover occurs. Test: Set the date to 31 Dec. 1999. Set the time to 23:50 (11:50 p.m.). Power down the system before it can roll over to year 2000 Wait until after midnight with the power off. Power up the system. Observe the system date Expected Result: The system clock advances into the year 2000 and operates normally. 1.3 Manual Date Set Test This test checks for correct date & time entry to initialize the system clock. The "set system date" function may operate incorrectly when the time is set ahead, not allow entry over a certain date range, revert to another time and date when set. Some PC's revert to a default date (1980 or 1984) when set to a date in the year 2000. Some systems have multiple date set functions; for a PC the date may be set using the CMOS Setup program at power on, using a DOS date function, using a windows clock or control panel interface. The tests in this section should be executed on all date set functions for the system. The date set function may also be accessed through an application programming interface (API), an example failure of this type was found in an Allen-Bradley PLC "C language" co-processor module. The set date function would not accept a value less than 32, returned an error message and hung. The SCO XENIX operating system "asktime" command only functions correctly for dates from 1970 through 1999, while the "date" function accepts 2000. Certain early versions of DOS do not update the RTC when the date is set using the DOS date command. If the equipment has a battery backed up clock, the date set test may include removing both battery power and external power to completely initialize the system clock and attempting to set the date to 1 Jan 2000. Exercise caution to document all system configuration when attempting this test because the configuration may be lost upon removal of the battery. 1.3.1 Date Set - 1 Jan 2000 Test: Set the date to 1 Jan 2000. Observe the system date Expected Result: The date should be Saturday, 1 Jan 2000. 1.3.2 Date Set - Date retention Check to insure that the date set function sets the real-time clock not just the systems virtual clock. Test: With the Date still in the year 2000 power down the system. Power up the system. Observe the system date Expected Result: The system clock still displays the year 2000 and operates normally. Note: Some PC's which fail the 1.2.3 Reboot - Date retention test will pass the manual date retention test. This is a possible fix for those PC's. 1.3.3 Date Set - 29 Feb. 2000 Test: Set the date to 29 Feb. 2000. Observe the system date Expected Result: The date should be Tuesday 29 Feb. 2000. 1.4 Leap Year Test The leap year test checks the logic which calculates valid dates for leap year. An example of a failure on leap year was published on the Internet which told of 66 industrial controllers in a steel mill all locking up when the date calculation for leap year 1996 occurred. A 2-digit year representation presents a possible divide by zero problem. The year 2000 leap year calculation is more complex because multiple exceptions apply to the calculation leading to greater opportunities for error. It is a leap year ú if the year is divisible by four, ú if the year ends in 00, it is not a leap year, ú if the year is divisible by 400, then it is a leap year, ú if the year is 3600, it is not a leap year. 1.4.1 Leap Year - Rollover 2/28 - Power On Test: Set the date to Monday 28 Feb. 2000. Set the time to 23:59 (11:59 p.m.). Observe the system date after midnight Expected Result: The date should be Tuesday 29 Feb. 2000. 1.4.2 Leap Year - Reboot 2/29 Test: Set the date to 29 Feb. 2000. Power down the system. Power up the system. Observe the system date Expected Result: The date should be Tuesday 29 Feb. 2000. 1.4.3 Leap Year - Rollover 2/29 - Power On Test: Set date to 29 Feb 2000 Set the time to 23:59 (11:59 p.m.). Observe the system date after 00:00 am Expected Result: The date should be Wednesday 1 March 2000. 1.5 Date Window Tests Windowing date systems assume the first 2 digits of a 4-digit year to be 20 for values below the switch value and 19 for values above the switch value. An example switch value of 50 provides for a range of 1951 to 2049. If the 2-digit year is greater than 50 the year is assumed to be 19xx. That is, 84 is greater than the switch value so the year is 1984. If the 2-digit year is less than 50 the year is assumed to be 20xx. That is, 34 is less than the switch value so the year is 2034. When two integrated systems share date information in this format be sure to test the interface at the boundary conditions. Is the behavior specified when the year is the switch value? Do both sides of an interface switch the same way? Systems using date windowing should consider testing: Creation of date data at the switch boundary dates, above and below. Modification of configurable windowing parameters, change the switch value. Re-test the switch boundary dates, above, and below. 1.5.1 Date Window Test - Below Limit Test: Observe the configured date limit. Change the current date to one year below the limit Observe a 4-digit date Expected Result: The date assumes 20xx 1.5.2 Date Window Test - Above Limit Test: Observe the configured date limit. Change the current date to one year above the limit Observe a 4-digit date Expected Result: The date assumes 19xx 1.6 Other Date Representation Tests The following date formats occur often enough that a brief description is included to stimulate thinking about possible date related functions and interface definitions. 1. Julian Date (JD) - is a real number where the integer part represents the number of days since 12:00 Noon 1 January -4712 ( Julian day zero ) and the fraction part is the part of the 24 hour day past Noon. 2. Modified Julian Date (MJD) - is the Julian Date minus 2,400,000.5 shifting the reference date to Midnight 17 November 1858. This reduces the number to 5 digits for the next 150 years. In 1997 the MJD = 50448 + DOY. 3. Gregorian Date in Day of Year formats (DOY) Dates represented using a YYDOY or YYYYDOY format where Y is the Year in 2-digit or 4-digit format and DOY is the day of the year from 001 to 365 (or 366 on a leap year) should be tested for correct operation. The following DOY dates should be checked: 29 February 2000 should be 00060 or 2000060 31 December 2000 should be 00366 or 2000366 Invalid Dates like 98000, 98367, 00000, 2000000 Special Codes 99365 1.6.1 DOY - 29 February 2000 Test: Set the date to 29 February 2000 Observe the DOY date by function call or system display. Expected Result: 29 February 2000 should be 00060 or 2000060 1.6.2 DOY - 31 December 2000 Test: Set the date to 31 December 2000 Observe the date by function call or system display. Expected Result: 31 December 2000 should be 00366 or 2000366 1.6.3 DOY - Invalid Dates Test: Attempt to set the DOY date to 2000000 Observe the DOY date function call return value. Attempt to set the DOY date to 98367 Observe the DOY date function call return value. Expected Result: Error code or message as documented by vendor 1.7 Arithmetic Date Tests If dates are used in any calculations, test for correct operation. The following list is intended to help identify functions which should be checked: Period calculations Financial functions based on a time period Shelf life calculations Time Remaining 1.7.1 Days in 2000 Test: Create a period calculation using 1-Jan-2000 as the start date and 31-Dec-2000 as the end date. Expected Result: The year 2000 has 366 days. 1.7.2 Days across 1999/2000 Boundary Test: Create a period calculation using 1-Dec-1999 as the start date and 31-Jan-2000 as the end date. Expected Result: The period ( (31 January 2000)- (1 December 1999) ) has 61 days. 1.7.3 Days across leap year Test: Create a period calculation using 1-Feb-2000 as the begin date and 1-Mar-2000 as the end date. Expected Result: The month of February has 29 days. 1.8 Upload / Download Tests The upload and download tests check for logic which prevents new files from replacing old files when the date comparison uses only 2 digits. The logic must be reversed when the dates cross the year 2000 boundary. Test to see if a file created in (19)99 is considered older than a file created in (20)00. Systems have failed to download new programs because they were assumed to be older that the current program on the system. Old File Date | New File Date | 2-digit | 4-digit | | difference | difference -------------------------------------------------------- July 4, 1998 | July 4,1999 | +1 | +1 | | | July 4, 1999 | July 4,2000 | -99 | +1 | | | July 4, 2000 | July 4, 2000 | +1 | +1 -------------------------------------------------------- 1.8.1 Upload Test: Set the date of the control system to the date of January 11, 2000. Attempt to upload the test file. Expected Result: Verify that the new file was uploaded. 1.8.2 Download Preparation: Download the existing test file with a date prior to January 1, Year 2000 Create a new version of the test file to download with the file date January 5, 2000. Test: Set the date of the control system under test to a date January 10, 2000. Attempt to download the January 5, 2000 test file. Expected Result: Verify that the new file was downloaded. 1.9 Special Value Test The special value test checks for usage of values in date fields for special purposes that are not dates. An example is special handling of the date September 9, 1999 which may be used as a special code for software license expiration dates, or never expires codes, and/or errors. Systems integrated to higher level systems should include the special value tests. The special values should include the date values 9-9-99, 0-0-00, x-x-9999. This test applies to applications which create records containing the current date as a time or data field, such as database applications or systems which maintain historical data . Test: Set the current date to a special value ( 9-9-99, 9-9-1999, 0-0- 00, 0-0-0000 ) Observe the number of records in a test file at the start of the test Using the application under test, create a new record that contains the current date. Expected Result: Observe that the application was able to create the test record. Observe that the test record is included in displays or reports as applicable. Observe that the end of file continues to function correctly. ( number of records correct? ) Observe that the test record can be deleted from the system. Example: Not terminating an expired software license. Failing to age backup tapes for recycling as scratch tapes. End-of-file markers which use the date field with a value of (9/9/99; 0/0/00; 9999; etc.) function correctly. 1.10 File or Directory Creation Test These tests check for observed problems with file or directory creation when the file name is based on an incorrect year date. These tests apply to any system which can store information collected from the manufacturing process, or allow for editing and creating new files. Errors can occur when the system attempts to create data files after the year 2000 and the date roll over is incorrect. Systems which create file names based on the time and date have been observed to lockup when the file name contained non- printable or illegal characters. Another possibility exists when the file already exists and is being updated. Most user interfaces prompt for verification, "Do you wish to replace file foo.dat 12/30/99 with file foo.dat 01/03/00?", will the newest file replace the older file in both the prompt and the actual replacement? Is the file identifier "FILE00" assumed to be older than "FILE99"? 1.10.1 File - Creation 2000 Test: Set the date of the system under test to a date beyond January 1, 2000. Create an event or choose a time such that the system will attempt to create a file. Expected Result: Verify that the new file was created. Verify that time stamped information is valid inside any history or log files. Verify that any history or log file scan be used by the application or system Examples: Systems which store historical data may create files using a name based on the date. 1.10.2 File - Replacement 1999-2000 Test: Create an old test file in 199X with data identifiable for 199X. Set the date of the system under test to a date beyond January 1, 2000. Create a new test file with the same name and new data. Expected Result: Observe the prompt for the correct order, replace old with new file Verify that the old file was replaced with the new file. Verify that the file contains the new data. Examples: Systems which prompt for confirmation on file update, like the Windows file manager. 1.10.3 File - Replacement 2000-2000 Test: Create an old test file in 2000 with data identifiable for 2000. Set the date of the system under test to a date beyond the creation date. Create a new test file with the same name and new data. Expected Result: Observe the prompt for the correct order, replace old with new file Verify that the old file was replaced with the new file. Verify that the file contains the new data. Examples: Systems which prompt for confirmation on file update, like the Windows file manager. 1.11 Audit Log Test This test checks for problems with audit logging systems. This test applies to systems which include the capability to audit user activity or network transactions. 1.11.1 Audit Log Test: Set the date of the system under test to a date beyond January 1, 2000. Create an event or choose a time such that the system will attempt to create a file. Expected Result: Verify that the new file was created. Verify that time stamped information is valid inside the audit log file. Examples: Systems which incorporate a DBMS with roll back capability. Operating systems which record usage by account. 1.12 Report Tests The report test should include sorting and retrieving, sorting and merging, searching, indexing on either disk file or database table. The report tests apply to manufacturing systems which display data sorted by time and date, examples include alarm fault reports or production reports. The tests should include creating faults or alarms after the year 2000 and observing the alarm display pages or reports for correct ordering of the alarm data. Failure examples include monitoring packages which place new alarms at the end of the list after 2000. One test is to query for all items from now until 12-31-1999 and observe the results, then query for all items from now until 1-2-2000 and observe the results, example failures observed returned no records on the second query. 1.12.1 Report - Query Test: Set the date of the control system under test to a date beyond January 10, 2000. Create new data by forcing a fault or some system event which will create test records. Set the date of the control system under test to a date beyond March 1, 2000. Create a new report containing the Year 2000 data by choosing four time spans: a) November 15, 1999 to December 31, 1999. ( 1999 data ) b) November 15, 1999 to March 1, 2000. ( all data ) c) January 1, 2000 to March 1, 2000. ( 2000 data ) d) February 1, 2000 to March 1, 2000. ( no data ) Expected Result: Verify that the report with all data, b) contained all the data in the report. Verify that the data was ordered correctly in the report. Verify that the report with no data, d) executed correctly and no data was printed in the report. 1.12.2 Report - Sort Test: Set the date of the control system under test to a date beyond January 1, 2000. Create new data by forcing a fault or some system event which will create test records. Create a new report containing the Year 2000 data by choosing a valid time span. Expected Result: Verify that the new data was ordered correctly in the report. 1.12.3 Report - Merge Test: Set the date of the control system under test to a date beyond January 1, 2000. Create new data by forcing a fault or some system event which will create test records. Create a new report containing the Year 2000 data by merging new data. Expected Result: Verify that the new data was merged correctly in the report. 1.12.4 Report - Search Test: Query or Search for an existing record created in the year 1999 with the current time in the year 1999 Query or Search for an existing record created in the year 1999 with the current time in the year 2000 Query or Search for an existing record created in the year 2000 with the current time in the year 2000 Expected Result: Verify that all records are found as expected. 1.13 Log file purge Test The log file purge test applies to manufacturing systems which periodically purge old data to maintain file system space usage by deleting the oldest data. The problem identified occurs when after the year 2000, files with year data lower than other files ( 1998 is less than 1999 ) are removed. Does the system consider data with a year date of 2000 to be less than 1999? If the comparison is only considering 2-digit years, this will happen and newer data can be purged. Test: Verify that log data backups are available. Set the date of the system under test to a date beyond January 10, 2000. Create new data for the log file or rename some existing data. Attempt to purge data from the system older than 7 days. Expected Result: Verify that only data from before the purge date was removed. Examples: VAX/VMS RMS purge command Database products which support a purge function. 1.14 Timer Test This test verifies the creation and operation of event timers in systems or software which provide these capabilities. Test: Set the date of the control system under test prior to 2000. Create new timer to wake up or alarm to trigger at 10:01 AM, January 3, 2000 Set the date of the control system under test to January 2, 2000. Create new timer to wake up or alarm to trigger at 10:02 AM, January 3, 2000 Set the date of the control system under test to January 3, 2000. Set the time to 10:00 AM. Wait for the alarms to trigger Expected Result: Verify that the alarm or timer created before 2000 operates correctly. Verify that the alarm or timer created after 2000 operates correctly. Examples: Unix chron scheduling software. SCADA package timer functions. HVAC Controls for starting and stopping ventilation or cooling equipment. 1.15 Input Data Test The input data test applies to manufacturing systems which read date information from labels or other manufacturing control systems. Test: Set the date of the control system under test to January 2, 2000. Create input labels or simulate input from other systems with a date beyond 1-1-2000. (or in 2-digit year format 00 ) Attempt to read the input data. Expected Result: Verify that the system correctly reads the input data. 1.16 Output Data Test The output data test applies to manufacturing systems which write date information to labels or other manufacturing control systems. Systems which print results to a printer or operator display have been found to lock up or fail to display data after the year 2000. The cause is that from the year date rollover, invalid characters can be produced. Test: Set the date of the control system under test to a date beyond January 1, 2000. Attempt to output data. Expected Result: Verify that the data was output correctly. Examples: Printed labels or product markings Transfer data to other systems 1.17 Activation/Deactivation Tests This test applies to manufacturing systems which contain passwords, accounts, or complex software license systems which contain expiration functions. 1.17.1 Valid access Test: Check that the expiration date extends past January 1, 2000 Set the date of the system under test to a date beyond January 1, 2000. Attempt to execute the software licensed, or use the affected password, account, etc. Expected Result: Verify that the software executes properly after January 1, 2000. 1.17.2 Expired access Test: Check the expiration date. Set the date of the system under test to a date beyond the expiration date in the year 2000 or beyond. Attempt to execute the software licensed, or use the affected password, account, etc. Expected Result: Verify that the software does not execute after the expiration date. 1.18 Display Data Tests The display data test applies to manufacturing systems which display date information on several different pages. The test must include moving the date ahead to the year 2000 and observing every screen which the controller contains. The non-compliance can range from partial display to complete control system lockup. Many industrial controllers have unique software for each display screen, and may behave differently on any screen 1.18.1 Display Data Test Test: Create a list of all the date fields on all the display screens. Set the date of the system under test to a date beyond January 1, 2000. Create new files or fault records. Attempt to display all date fields on all display screens for file dates or fault time stamps. Expected Result: Verify that each date field displays correctly. Examples: SCO XENIX displays file dates using 4 digit years correctly, but the AVAIL software package displays file dates incorrectly; the year field of the file date during a year 2000 test displayed "00:08" as the year. A software application supports 4-digit dates in the 20xx range using a DBMS but only passes 2-digit years to the DBMS which defaults to 19xx and stores the wrong date. 1.19 Indirect Date Usage Tests These tests apply to systems which use date information in an indirect manner. The following list is intended to stimulate questions about a system which could use the date in functions which do not require date information, but may have been implemented using a date function. Test: Identifying functions which use the date indirectly may be very difficult. Expected Result: Where identified, verify correct operation in the year 2000. Examples: Encryption/Decryption algorithms Random Number generators Communications protocols Firmware --------------------------------------- CSSA/PMI Y2000 SIG - WG2