Trace to grade configuration

Main configuration file

Trace-to-grade takes a YAML configuration file as input. This file is used to tell the tool how to compute the student grades based on their traces. Therefore it must define all the assignment to grade and how to distribute points for each of them.

This file must respect the following structure:

Assignments

The main Assignments structure is defined as follows:

Field	Type	Description
assignments	List<Assignment>	List of all assignments to process.

Assignment

The Assignment structure is defined as follows:

Field	Type	Description
name	String	Name of the assignment in outputs.
assignment	String	Name of the assignment in input. See the tool section for more information.
value	Double	Max number of points attributed to this assignment.
scale	Scale	Scale configuration of the assignment. See the section below for more information.

Scale configuration

To tell the tool how to distribute points for an assignment you need to specify the rules related to each testcase of the student trace. For instance you may want to make a test mandatory so that the student will get a 0 if it do not pass the test.

To have a better overall configuration, we recommend using separate files to define the scales of each Assignment. It is more maintainable and organized than putting every scales with the rest of the configuration.

To do this you can import a scale files with the following YAML line:

assignments:
  # ...
  scale: !include path/to/scale.yml

caution

If you are using the above approach, make sure that the path is either absolute or relative to where you type trace-to-grade commands.

Scale

The Scale structure is defined as follows:

Field	Type	Description
bonus	Optional<Double>	Max number of bonus points attributed by bonus tests.
categories	List<Category>	List of computation rules to apply for matched testcases.

Category

The Category structure define computation rules for all testcases matched by a given pattern. It is defined as follows:

Field	Type	Description
name	String	Name of the category in outputs.
pattern	String	Regex pattern for matching testcases. The matching is done on the classname property of XML traces.
value	Int	Number of points attributed if all matched testcases are validated.
mandatory	Optional<Boolean>	Specifies if all matched testcase must be validated to avoid getting 0 for the whole assignment.
bonus	Optional<Boolean>	Specifies if all matched testcase are bonus tests.
ratio	Optional<Boolean>	Specifies if all matched must be validated to have the category points.
malus_ratio	Optional<Int>	Ratio to apply on the final grade of the Assignment if any matched testcases is not validated.

note

As some testcase can appear in the trace only if they failed, if no tests are found for a mandatory Category, then it is considered validated.

note

As a max value is defined in the Assignment, the value field in each scales act more as weights than real points. This confusing notation is kept for backward compatibility.

Also points obtained with bonus tests are truncated to the max value defined in the Scale structure.

Final

The final configuration file should look like the following:

assignments:
  - name: exercise_1
    assignment: exercise_1
    value: 10
    scale: !include path/to/exercise_1-scale.yml
  - name: exercise_2
    assignment: exercise_2
    value: 10
    scale: !include path/to/exercise_2-scale.yml

And your scale files should look like the following:

bonus: 1.0
categories:
  - name: make
    pattern: exercise_1.build.make
    value: 0
    mandatory: true

  - name: forbidden
    pattern: exercise_1.Forbidden.symbols
    value: 0
    mandatory: true

  - name: clang-format
    pattern: exercise_1.codestyle-clang-format
    value: 0
    malus_ratio: 75

  - name: asan
    pattern: exercise_1.asan.*
    value: 0
    malus_ratio: 75

  - name: tests
    pattern: exercise_1.tests
    value: 1
    ratio: false

  - name: bonus
    pattern: exercise_1.bonus
    value: 1
    ratio: false

A trace corresponding to the exercise_1 of this configuration would look like this:

<?xml version='1.0' encoding='UTF-8'?>
<testsuites>
    <testsuite>
        <testcase name="file.c" classname="exercise_1.build.make"/>
        <testcase name="file.c" classname="exercise_1.clang-format">
            <failure>/student/file.c:2:2: error: code should be clang-formatted [-Wclang-format-violations]
{
 ^
            </failure>
        <testcase name="test1" classname="exercise_1.tests" time="0.000113" zluor_idx="0"/>
        <testcase name="test2" classname="exercise_1.tests" time="0.000151" zluor_idx="0"/>
    </testsuite>
</testsuites>