Poulpy

Poulpy is an auto-updated and customizable Python tool for working and operating with Forge services.

Installation

Poulpy is available as a Python package on the Forge GitLab. You can use pip to install it:

pip install poulpy --index-url https://gitlab.cri.epita.fr/api/v4/projects/8706/packages/pypi/simple

You can then run Poulpy commands simply by running poulpy:

$ poulpy --help

Available commands

Authentication

You can use Poulpy to retrieve JWT tokens that you can then use to authenticate with Forge services.

JWTs are confidential

Note that JWTs are sensitive credentials that should not be shared with anyone.

You can use the following command to print a JWT token on stdout:

poulpy auth

Note that you may be prompted to go to a URL for logging in (the messages themselves are sent on stderr to keep your stdout clean with only the JWT in it).

You can then provide this token in the Authorization header of your requests:

Authorization: Bearer YOUR_TOKEN_HERE

For example, with curl:

curl --fail -H "Authorization: Bearer $(poulpy auth)"

In case you want to directly put the JWT in your clipboard, you can use the following commands on Linux:

• poulpy auth | wl-copy for Wayland (requires wl-clipboard)
• poulpy auth | xsel -b for X11

Tokens are stored in ~/.cri-tokens, which means that you can also retrieve a stored tokens via tools like jq, e.g. cat ~/.cri-tokens | jq -r .idToken. Note that tokens issued by the CRI are usually pretty short-lived, so you should keep calling poulpy auth to get a known-valid token.

Intranet deployment

The intranet has several inputs for operators, teachers and assistants alike to deploy their projects. The 2 main types of inputs available to you are likely:

• Tenant
• Activities

These 2 inputs are in their own git repository. The Forge Team has already setup the repository for you but the tool they are using in the background to validate and send these inputs is Poulpy.

This section describes how you can manually use these commands to manually validate and deploy these inputs.

CI/CD Errors

The logs you get on gitlab are often truncated and not colored. Manually running the same command you see in the logs is the best way to start debugging your input.

Deployment

The deployment of these files is made using tokens. These tokens are not available to you directly if you do not have Maintainer rights on your repository. This is the best way for us to make sure you do not spam the services which deploy your inputs.

Tenant

The tenant file documentation can be found here You should check this page first if Poulpy or your CI/CD fails. The two available commands are tenant-validate and tenant-deploy. Example:

poulpy intranet tenant-validate --metadata tenant.yml

Here are the options available:

Option	Required	Default	Environment Variable	Description
--metadata	yes			Path the the tenant file
--base-url	no	https://repo-tenant.api.forge.epita.fr	TENANT_ACTIVITY_BASE_URL	Url on which to deploy and validate
--token	no		TENANT_TOKEN	Token to deploy the tenant

Activity

The activity file documentation can be found here You should check this page first if Poulpy or your CI/CD fails. The two available commands are pacv2-validate and pacv2-deploy. Example:

poulpy intranet pacv2-validate --metadata tenant.yml

Here are the options available:

Option	Required	Default	Environment Variable	Description
--metadata	yes			Path the the activity file
--base-url	no	https://repo-activity.api.forge.epita.fr	REPO_ACTIVITY_BASE_URL	Url on which to deploy and validate
--token	no		ACTIVITY_TOKEN	Token to deploy the activity

Calling APIs

As mentioned earlier, Poulpy's main goal is to be an auto-updated CLI equivalent for Swagger. It means that it allows to call Forge endpoints in CLI, it also auto-update itself when changes on Forge services are detected.

To do this, Poulpy uses an external tool called Restish, we use the aforementioned script to handle authentication. You can use the -p/--profile option to use a different authentication profile (e.g. if you need to authenticate against the development or staging services).

You can call poulpy --help to get a list of available commands. Those that start with "Rest Client" correspond to API services that can be called; Use poulpy NAME --help to get the list of available endpoints for a given service.

Poulpy uses a default configuration file to specify available Forge services. This file can be modified manually if needed, it is stored in ~/.poulpy/restish.json.

Configuration files

Note that the configuration folder and files will be automatically created when calling Poulpy if they are not present.

To add a new API to your configuration, you can directly edit the file. For more information see Restish's documentation.

How to update the configuration

Using restish command to update the configuration will only modify ~/.restish/apis.json, which is overriden by Poulpy. Therefore you need to do the modification by hand on the configuration file mentioned above.

Restish syntax

Even if Restish gives you understandable examples, you may encounter difficulties with the syntax if you are just starting to use this tool. Don't forget to check the documentation if you have any doubt with it.

Running complex pipelines

In multiple cases you will need to call multiple Forge endpoints to achieve your goal. To facilitate those operations, Poulpy includes a pipelining system.

This system allows you to easily chain calls to Poulpy while manipulating endpoints data.

To do this, Poulpy uses another external tool called Nushell. It is a shell where everything is considered as data (somewhat like PowerShell). It allows you to chain Poulpy commands and manipulate results with builtins.

The pipeline system can be used with poulpy pipe 'command ...'.

To call Forge services inside a pipeline, you need to write the service name directly, it allows us to properly inject authentication options on the pipeline. For instance a call like this:

poulpy operator command ...

Would be written like this in a pipeline:

poulpy pipe 'operator command ...'

For a real example, downloading student traces for a given submission could be done like this:

poulpy pipe 'operator get-picked-submission <submission_uri> -f body | from json | each {|job| fetch $job.tracePresignedUrl | save -r $"($job.groupSlug).xml" }'

For more information on the syntax and/or the available builtins, see Nushell's documentation.

Preset system

As you can see, even with the pipeline system, commands are still tedious to type. It is even more annoying when only one parameter differs from the last time you typed your command. To overcome this problem, Poulpy also includes presets.

The preset system allows you to generate Poulpy commands that are aliases of long and painful Poulpy pipelines. To do this, we provide another configuration file that is stored in ~/.poulpy/presets.json that can be modified if needed.

Configuration file

Just like other configuration files, a default one is generated when calling Poulpy if one do not exists already.

For instance the following configuration:

{
  "download_traces": {
    "args": ["submission_definition_uri"],
    "cmd": "operator get-picked-submission {submission_definition_uri} -f body | from json | each {{|job| fetch $job.tracePresignedUrl | save -r $\"($job.groupSlug).xml\"}}",
    "description": "Download students traces for a given submission"
  }
}

Allows you to directly download traces of a given submission with the following command:

poulpy download_traces --submission_definition_uri <submission_definition_uri>

It allows you to download traces for another submission without having to modify the pipeline by hand every time.

Configuration

To add a new preset you can add an entry directly in the configuration file:

{
  "new_command": {
    "cmd": "my command"
  }
}

The args and description fields are optionals.

To add an arg to your preset, you must:

• Add the arg into the args field of your preset.
• Put the arg at the desired place in the cmd field between brackets (The same way you would use F-strings in Python).

To really use brackets in the cmd field, you must escape them with another brackets (ex: {{...}}).

For instance, the following preset:

{
  "new_command": {
    "args": ["arg"],
    "cmd": "^echo {{{arg}}}"
  }
}

would be used like this:

$ poulpy new_command --arg "aled"
{aled}

However keep in mind that the command will throw an error if it fails to format every unescaped pairs of bracket.

Furethermore for complex commands, you can use nu scripts instead of simple commands, to do this you just have to write your script in a separate file and launch it in the cmd field, like this:

{
  "new_command": {
    "args": ["arg"],
    "cmd": "nu /path/to/script.nu {arg}"
  }
}

To write proper nu scripts check Nu documentation and the default scripts that are used with the default presets, they are located in the following folder: ~/.poulpy/nu-scripts/.

Refresh the default configuration

As mentioned above, Poulpy use multiple configuration files. Each one of them come with default configuration and then can be modified by the user to fit everyone needs.

In order to update the defauts configuration without losing your personal added configuration you can use the refresh command. You can specify which configuration to update if you only want one part of the configuration to be updated.

For instance, you can update only the presets configuration with:

poulpy refresh -t "PRESETS"

Using Poulpy as a library

caution

This section of the documentation is not available yet. Please check back later. In the mean time, if you need help, please contact us.