Set up tests

In order to run tests efficiently, it is essential to prepare a testsuite and use a Docker environment. This approach ensures a standardized testing process, leveraging the benefits of containerization to achieve test uniformity, isolate test environments, and mitigate potential side effects or security issues.

By encapsulating tests within Docker containers, it becomes possible to create reproducible and controlled environments, reducing the risk of interference from external factors and enhancing the overall reliability and security of the testing process. This practice not only streamlines the testing workflow but also contributes to a more robust and consistent evaluation of code performance.

Prerequisite

Testsuite

To evaluate a student's code, you need to create a testsuite, ie. a set of tests aligned with the activity's definition. To execute and analyze these tests efficiently, it's recommended to use a testing library that produces XML JUnit-formatted output.

Wikipedia has a list of frameworks with a column specifying if they are able to emit results in JUnit XML format.

tip

MaaS logs contain the exact Docker command that was ran. You can use this to debug your testsuite.

Docker images

caution

MaaS makes heavy use of Docker and this documentation expects a decent level of understanding on what Docker is and how docker run is used. If you are not familiar with Docker, you should check out Docker Curriculum, which provides a good introduction.

MaaS can accept any kind x86-64 Docker image. However, MaaS will only pull images from a specific registry in order to avoid running untrusted, uncontrolled images.

You cannot define said registry in your workflow on your own. You must contact us if you wish to have the allowed registry URL(s) changed.

You must also take care to match the version of software students use. For example, if they are expected to work on the school computers, you should ensure that the version of the language, compilers and tools in your Docker image match those used on the PIE.

Repository structure

Here's the basic structure of a repo including all the elements required for automated correction of an exercise.

.
├── exercices
│   └── exo1
│       ├── testsuite
│       │   └── test.py
│       ├── maasWorkflow.yml
│       └── subject.html
├── .gitlab-ci.yml
├── Dockerfile
├── README.md
└── activity.yml

activity.yml

For each assignment, you need to specify a maas workflow (and a testsuite) in the activity.yml file

assignments:
  - maxLearners: 1
    name: Exercice 1
    slug: exo1
    submissions:
      - slug: exo1-submission
        handoffType: GIT_TAG
        tagPrefix: exo1-
        limit: 100
        limitType: SUBMISSION
        maxInFlight: 1
        pick: LAST
        autoPublish: true
    subscriptions:
      logins:
        - xavier.login
    documents:
      - path: exercises/exo1/maasWorkflow.yml
        name: maasWorkflow.yml
        filename: maasWorkflow.yml
        type: WORKFLOW
      - path: exercises/exo1/subject.html
        name: subject.html
        filename: subject.html
        type: EMBEDDED
      - path: exercises/exo1/testsuite.tar
        name: testsuite.tar
        filename: testsuite.tar
        type: TESTSUITE

.gitlab-cy.yml

Here's .gitlab-ci.yml file example that use a docker image from the Forge registry in a GitLab CI stage using the access token previously generated

stages:
  - build
  - deploy

docker-build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - mkdir -p /kaniko/.docker
    - echo
      "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}"
      > /kaniko/.docker/config.json
    - >-
      /kaniko/executor --context "${CI_PROJECT_DIR}" --dockerfile
      "${CI_PROJECT_DIR}/Dockerfile" --destination
      "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}" --destination
      "${CI_REGISTRY_IMAGE}:latest"
  rules:
    - if: $CI_COMMIT_TAG

deploy:
  stage: deploy
  artifacts:
    paths:
      - projects.yml
  only:
    - main
  script:
    - echo "update" | tee output.txt
    - ls -lah