Changes between Version 22 and Version 23 of Building/RunningTests

Timestamp:

08/21/09 02:46:55 (4 years ago)

Author:

simonpj

Comment:

--

Building/RunningTests

@@ -132 +132 @@
     go in parser/, and so on.  
 
-    It's not always possible to find a single best place for a test;
-    in those cases just pick one which seems reasonable.
-
-    Under each main directory may be up to three subdirectories:
-
-       should_compile:
+ It's not always possible to find a single best place for a test;
+ in those cases just pick one which seems reasonable.
+
+ Under each main directory may be up to three subdirectories:
+       '''should_compile''':    
            tests which need to compile only
-
-       should_fail:    
+       '''should_fail''':    
            tests which should fail to compile and generate a particular error message
-
-       should_run:
+       '''should_run''':
            tests which should compile, run with some specific input, and generate a particular output.
     
-    We don't always divide the tests up like this, and it's not
-    essential to do so (the directory names have no meaning as
-    far as the test driver is concerned).        
+ We don't always divide the tests up like this, and it's not
+ essential to do so (the directory names have no meaning as
+ far as the test driver is concerned).        
 
 
  2. Having found a suitable place for the test, give the test a name.
-    For regression tests, we often just name the test after the bug number (e.g. 2047).
+    For regression tests, we often just name the test after the bug number (e.g. T2047).
     Alternatively, follow the convention for the directory in which you place the
     test: for example, in typecheck/should_compile, tests are named
@@ -182 +179 @@
 
 
- 1. Edit all.T in the relevant directory and add a line for the test.  The line is always of the form
+ 3. Edit all.T in the relevant directory and add a line for the test.  The line is always of the form
 {{{
       test(<name>, <setup>, <test-fn>, <args>)
 }}}
-    where
-
-    ''<name>'' is the name of the test, in quotes (' or ").
-
-    ''<setup>''  is a function (i.e. any callable object in Python)
-    which allows the options for this test to be changed.
-    There are many pre-defined functions which can be
-    used in this field:
-
-        '''normal'''                don't change any options from the defaults
-
-        '''skip'''                  skip this test
-
-        '''skip_if_no_ghci'''       skip unless GHCi is available
-
-        '''skip_if_fast'''          skip if "fast" is enabled
-
-        '''omit_ways(ways)'''       skip this test for certain ways
-
-        '''only_ways(ways)'''       do this test certain ways only
-
-        '''extra_ways(ways)'''      add some ways which would normally be disabled
-
-        '''omit_compiler_types(compilers)'''                           skip this test for certain compilers
-
-        '''only_compiler_types(compilers)'''       do this test for certain compilers only
-
-        '''expect_broken(bug)''' this test is a expected not to work due to the indicated trac bug number
-
-        '''expect_broken_for(bug, ways)''' as expect_broken, but only for the indicated ways
-
-        '''if_compiler_type(compiler_type, f)''' Do `f`, but only for the given compiler type
-
-        '''if_platform(plat, f)'''  Do `f`, but only if we are on the specific platform given
-
-        '''if_tag(tag, f)'''        do `f` if the compiler has a given tag
-
-        '''unless_tag(tag, f)'''    do `f` unless the compiler has a given tag
-
-        '''set_stdin(file)'''       use a different file for stdin
-
-        '''no_stdin'''              use no stdin at all (otherwise use `/dev/null`)
-
-        '''exit_code(n)'''          expect an exit code of 'n' from the prog
-
-        '''extra_run_opts(opts)'''  pass some extra opts to the prog
-
-        '''no_clean'''              don't clean up after this test
-
-        '''extra_clean(files)'''    extra files to clean after the test has completed
-
-        '''reqlib(P)'''             requires package P
-
-        '''req_profiling'''         requires profiling
-
-        '''ignore_output'''         don't try to compare output
-
-        '''alone'''                 don't run this test in parallel with anything else
-
-        '''literate'''              look for a `.lhs` file instead of a `.hs` file
-
-        '''c_src'''                 look for a `.c` file
-
-        '''cmd_prefix(string)'''    prefix this string to the command when run
-
-        '''normalise_slashes'''     convert backslashes to forward slashes before comparing the output
-
-    To use more than one modifier on a test, just put them in a list.
-    For example, to expect an exit code of 3 and omit way 'opt', we could use
-{{{
-      [ omit_ways(['opt']), exit_code(3) ]
-}}}
-      as the `<setup>` argument.
-
-    The following should normally not be used; instead, use the `expect_broken*`
-    functions above so that the problem doesn't get forgotten about, and when we
-    come back to look at the test later we know whether current behaviour is why
-    we marked it as expected to fail:
-
-        '''expect_fail'''           this test is an expected failure, i.e. there is a known bug in the compiler, but we don't want to fix it.
-
-        '''expect_fail_for(ways)''' expect failure for certain ways 
-
-    ''<test-fn>''
-    is a function which describes how the test should be
-    run, and determines the form of <args>.  The possible
-    values are:
-
-      compile
-            Just compile the program, the compilation should succeed.
-
-      compile_fail
-            Just compile the program, the
-            compilation should fail (error
-            messages will be in T.stderr).
-            This kind of failure is mandated by the language definition - it does '''not''' indicate any bug in the compiler.
-
-      compile_and_run 
-            Compile the program and run it,
-            comparing the output against the 
-            relevant files.
-
-      multimod_compile
-            Compile a multi-module program
-            (more about multi-module programs
-            below).
-
-      multimod_compile_fail
-            Compile a multi-module program,
-            and expect the compilation to fail
-            with error messages in T.stderr.  This kind of failure does '''not''' indicate a bug in the compiler.
-
-      multimod_compile_and_run
-            Compile and run a multi-module
-            program.
-
-      compile_and_run_with_prefix
-            Same as compile_and_run, but with command to use to run the execution of the result binary.
-
-      multimod_compile_and_run_with_prefix
-            Same as multimod_compile_and_run, but with command to use to run the execution of the result binary.
-
-      run_command
-            Just run an arbitrary command.  The output is checked
-            against `T.stdout` and `T.stderr` (unless `ignore_output`
-            is used), and the stdin and expected exit code can be
-            changed in the same way as for compile_and_run.
-
-      ghci_script
-            Runs the current compiler, passing
-            --interactive and using the specified
-            script as standard input.
-
-    ''<args>''
-    is a list of arguments to be passed to <test-fn>.
-
-    For compile, compile_fail and compile_and_run, <args>
-    is a list with a single string which contains extra
-    compiler options with which to run the test.  eg.
-{{{                
-    test('tc001', normal, compile, ['-fglasgow-exts'])
-}}}
-    would pass the flag -fglasgow-exts to the compiler
-    when compiling tc001.
-
-    The multimod_ versions of compile and compile_and_run
-    expect an extra argument on the front of the list: the
-    name of the top module in the program to be compiled
-    (usually this will be 'Main').
-
+The format of these fields is described in the [wiki:Building/RunningTests#Formatofthetestentries next section].
+
+ 
 
 A multi-module test is straightforward.  It usually goes in a
@@ -350 +200 @@
 
 For some examples, take a look in tests/ghc-regress/programs.
+
+= Format of the test entries =
+
+Each test in a `test.T` file is specified by a line the form
+{{{
+      test(<name>, <setup>, <test-fn>, <args>)
+}}}
+
+== The <name> field ==
+
+''<name>'' is the name of the test, in quotes (' or ").
+
+== The <setup> field ==
+
+''<setup>''  is a function (i.e. any callable object in Python)
+which allows the options for this test to be changed.
+There are many pre-defined functions which can be
+used in this field:
+
+ * '''normal'''                don't change any options from the defaults
+ * '''skip'''                  skip this test
+ * '''skip_if_no_ghci'''       skip unless GHCi is available
+
+ * '''skip_if_fast'''          skip if "fast" is enabled
+
+ * '''omit_ways(ways)'''       skip this test for certain ways
+
+ * '''only_ways(ways)'''       do this test certain ways only
+
+ * '''extra_ways(ways)'''      add some ways which would normally be disabled
+
+ * '''omit_compiler_types(compilers)'''                           skip this test for certain compilers
+
+ * '''only_compiler_types(compilers)'''       do this test for certain compilers only
+
+ * '''expect_broken(bug)''' this test is a expected not to work due to the indicated trac bug number
+
+ * '''expect_broken_for(bug, ways)''' as expect_broken, but only for the indicated ways
+
+ * '''if_compiler_type(compiler_type, f)''' Do `f`, but only for the given compiler type
+
+ * '''if_platform(plat, f)'''  Do `f`, but only if we are on the specific platform given
+
+ * '''if_tag(tag, f)'''        do `f` if the compiler has a given tag
+
+ * '''unless_tag(tag, f)'''    do `f` unless the compiler has a given tag
+
+ * '''set_stdin(file)'''       use a different file for stdin
+
+ * '''no_stdin'''              use no stdin at all (otherwise use `/dev/null`)
+
+ * '''exit_code(n)'''          expect an exit code of 'n' from the prog
+
+ * '''extra_run_opts(opts)'''  pass some extra opts to the prog
+
+ * '''no_clean'''              don't clean up after this test
+
+ * '''extra_clean(files)'''    extra files to clean after the test has completed
+
+ * '''reqlib(P)'''             requires package P
+
+ * '''req_profiling'''         requires profiling
+
+ * '''ignore_output'''         don't try to compare output
+
+ * '''alone'''                 don't run this test in parallel with anything else
+
+ * '''literate'''              look for a `.lhs` file instead of a `.hs` file
+
+ * '''c_src'''                 look for a `.c` file
+
+ * '''cmd_prefix(string)'''    prefix this string to the command when run
+
+ * '''normalise_slashes'''     convert backslashes to forward slashes before comparing the output
+
+The following should normally not be used; instead, use the `expect_broken*`
+functions above so that the problem doesn't get forgotten about, and when we
+come back to look at the test later we know whether current behaviour is why
+we marked it as expected to fail:
+
+ * '''expect_fail'''           this test is an expected failure, i.e. there is a known bug in the compiler, but we don't want to fix it.
+
+ * '''expect_fail_for(ways)''' expect failure for certain ways 
+
+To use more than one modifier on a test, just put them in a list.
+For example, to expect an exit code of 3 and omit way 'opt', we could use
+{{{
+      [ omit_ways(['opt']), exit_code(3) ]
+}}}
+as the `<setup>` argument.
+
+== The <test-fn> field ==
+
+''<test-fn>''
+is a function which describes how the test should be
+run, and determines the form of <args>.  The possible
+values are:
+
+ * '''compile'''  Just compile the program, the compilation should succeed.
+
+ * '''compile_fail'''
+   Just compile the program, the
+   compilation should fail (error
+   messages will be in T.stderr).
+   This kind of failure is mandated by the language definition - it does '''not''' indicate any bug in the compiler.
+
+ * '''compile_and_run'''
+   Compile the program and run it,
+   comparing the output against the 
+   relevant files.
+
+ * '''multimod_compile'''
+   Compile a multi-module program
+   (more about multi-module programs
+   below).
+
+ * '''multimod_compile_fail'''
+   Compile a multi-module program,
+   and expect the compilation to fail
+   with error messages in T.stderr.  This kind of failure does '''not''' indicate a bug in the compiler.
+
+ * '''multimod_compile_and_run'''
+   Compile and run a multi-module
+   program.
+
+ * '''compile_and_run_with_prefix'''
+   Same as compile_and_run, but with command to use to run the execution of the result binary.
+
+ * '''multimod_compile_and_run_with_prefix'''
+   Same as multimod_compile_and_run, but with command to use to run the execution of the result binary.
+
+ * '''run_command'''
+   Just run an arbitrary command.  The output is checked
+   against `T.stdout` and `T.stderr` (unless `ignore_output`
+   is used), and the stdin and expected exit code can be
+   changed in the same way as for compile_and_run.
+
+ * '''ghci_script'''
+   Runs the current compiler, passing
+   --interactive and using the specified
+   script as standard input.
+
+== The <args> field ==
+
+''<args>'' is a list of arguments to be passed to <test-fn>.
+
+For compile, compile_fail and compile_and_run, <args>
+is a list with a single string which contains extra
+compiler options with which to run the test.  eg.
+{{{                
+    test('tc001', normal, compile, ['-fglasgow-exts'])
+}}}
+would pass the flag -fglasgow-exts to the compiler
+when compiling tc001.
+
+The multimod_ versions of compile and compile_and_run
+expect an extra argument on the front of the list: the
+name of the top module in the program to be compiled
+(usually this will be 'Main').
+
+
 
 = Sample output files =