3. Developer guide

This developer guide is meant for software developers who would like to understand REANA-Job-Controller source code and contribute to it.

3.1. Compute backends

REANA-Job-Controller offers an abstract interface to submit jobs to different compute backends. Currently it is only implemented for Kubernetes.

Job Manager.

class reana_job_controller.job_manager.JobManager(docker_img='', cmd=[], env_vars={}, job_id=None, workflow_uuid=None, workflow_workspace=None, job_name=None)

Job management interface.

after_execution()

After job submission hook.

before_execution()

Before job submission hook.

cache_job()

Cache a job.

create_job_in_db(backend_job_id)

Create job in db.

execution_hook(fn)

Add before execution hooks and DB operations.

get_logs()

Get job log.

Returns:stderr, stdout of a job. Return type:dict

get_status()

Get job status.

Returns:job status. Return type:str

stop()

Stop a job.

update_job_status()

Update job status in DB.

3.1.1. Kubernetes

Kubernetes jobs management is done via KubernetesJobManager class which implements the previously mentioned JobManager interface.

Execute method creates Kubernetes job specification and submits it. This method uses @JobManager.execution_hook decorator to execute specific operations defined in before_execution and necessary DB transactions in the right order.

Stop static function is responsible for stoping/deleting successfully finished or failed jobs.

3.1.2. HTCondor

To build REANA-Job-Controller Docker image with HTCondor dependencies use build argument COMPUTE_BACKENDS=kubernetes,htcondorcern.

$ reana-dev docker-build -c reana-job-controller -b COMPUTE_BACKENDS=kubernetes,htcondorcern

The users should then upload their HTCondor username and keytab secrets using:

$ reana-client secrets-add --env HTCONDORCERN_USERNAME=johndoe
                           --env HTCONDORCERN_KEYTAB=.keytab
                           --file ~/.keytab

see the reana-client’s documentation on secrets.

The users will then be able to specify compute backend htcondorcern in their workflow specification files to provide hints to the workflow execution system to run certain workflow steps on the HTCondorCERN backend. How this is done concretely depends on the specific workflow engine (CWL, Serial, Yadage).