arc.job.local

A module for running jobs on the local machine. When transitioning to Python 3, use subprocess.run()

arc.job.local.change_mode(mode: str, file_name: str, recursive: bool = False, path: str = '') → None[source]

Change the mode of a file or a directory.

Parameters:

  • mode (str) – The mode change to be applied, can be either octal or symbolic.

  • file_name (str) – The path to the file or the directory to be changed.

  • recursive (bool, optional) – Whether to recursively change the mode to all files under a directory. True for recursively change.

  • path (str, optional) – The directory path at which the command will be executed.

arc.job.local.check_job_status(job_id: int) → str[source]

Possible status values: before_submission, running, errored on node xx, done Status line formats:

OGE:

540420 0.45326 xq1340b    user_name       r     10/26/2018 11:08:30 long1@node18.cluster

Slurm:

14428     debug xq1371m2   user_name  R 50-04:04:46      1 node06
PBS (taken from zeldo.dow.com)::

Req’d Req’d Elap

Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time ———————– ———– ——– ————— —— —– —— ——— ——— - ——— 2016614.zeldo.local u780444 workq scan.pbs 75380 1 10 – 730:00:00 R 00:00:20 2016616.zeldo.local u780444 workq scan.pbs 75380 1 10 – 730:00:00 R 00:00:20

HTCondor (using ARC’s modified condor_q command):

3261.0 R 10 28161 a2719 56
3263.0 R 10 28161 a2721 23
3268.0 R 10 28161 a2726 18
3269.0 R 10 28161 a2727 17
3270.0 P 10 28161 a2728 23
arc.job.local.check_running_jobs_ids() → List[str][source]

Check which jobs are still running on the server for this user.

Returns:

List of job IDs.

Return type:

List[str]

arc.job.local.delete_all_local_arc_jobs(jobs: Optional[List[Union[str, int]]] = None) → None[source]

Delete all ARC-spawned jobs (with job name starting with a and a digit) from the local server. Make sure you know what you’re doing, so unrelated jobs won’t be deleted… Useful when terminating ARC while some (ghost) jobs are still running.

Parameters:

jobs (List[Union[str, int]], optional) – Specific ARC job IDs to delete.

arc.job.local.delete_job(job_id: Union[int, str])[source]

Deletes a running job.

arc.job.local.execute_command(command: Union[str, List[str]], shell: bool = True, no_fail: bool = False, executable: Optional[str] = None) → Tuple[Optional[list], Optional[list]][source]

Execute a command.

Notes

If no_fail is True, then a warning is logged and False is returned so that the calling function can debug the situation.

Parameters:

  • command (Union[str, List[str]]) – An array of string commands to send.

  • shell (bool, optional) – Specifies whether the command should be executed using bash instead of Python.

  • no_fail (bool, optional) – If True then ARC will not crash if an error is encountered.

  • executable (str, optional) – Select a specific shell to run with, e.g., ‘/bin/bash’. Default shell of the subprocess command is ‘/bin/sh’.

Returns: Tuple[list, list]:

  • A list of lines of standard output stream.

  • A list of lines of the standard error stream.

arc.job.local.get_last_modified_time(file_path_1: str, file_path_2: Optional[str] = None) → Optional[datetime][source]

Returns the last modified time of file_path_1 if the file exists, else returns the last modified time of file_path_2 if the file exists.

Parameters:

  • file_path_1 (str) – The path to file 1.

  • file_path_2 (str, optional) – The path to file 2.

arc.job.local.parse_running_jobs_ids(stdout: List[str], cluster_soft: Optional[str] = None) → List[str][source]

A helper function for parsing job IDs from the stdout of a job status command.

Parameters:

  • stdout (List[str]) – The stdout of a job status command.

  • cluster_soft (Optional[str]) – The cluster software.

Returns:

List of job IDs.

Return type:

List(str)

arc.job.local.rename_output(local_file_path: str, software: str) → None[source]

Rename the output file to “output.out” for consistency between software.

Parameters:

  • local_file_path (str) – The full path to the output.out file.

  • software (str) – The software used for the job by which the original output file name was determined.

arc.job.local.submit_job(path: str, cluster_soft: Optional[str] = None, submit_cmd: Optional[str] = None, submit_filename: Optional[str] = None, recursion: bool = False) → Tuple[Optional[str], Optional[str]][source]

Submit a job.

Parameters:

  • path (str) – The job’s folder path, where the submit script is located (just the folder path, w/o the filename).

  • cluster_soft (str, optional) – The server cluster software.

  • submit_cmd (str, optional) – The submit command.

  • submit_filename (str, optional) – The submit script file name.

  • recursion (bool, optional) – Whether this call is within a recursion.

Returns:

job_status, job_id

Return type:

Tuple[Optional[str], Optional[str]]

arc.job.local.write_file(file_path: str, file_string: str) → None[source]

Write file_string as the file’s content in file_path.

Parameters:

  • file_path (str) – The file path.

  • file_string (str) – The content to be written into the file.