arc.job.ssh

A module for SSHing into servers. Used for giving commands, uploading, and downloading files.

exception arc.job.ssh.HostKeyMismatchError[source]

Raised when the host key a server presents contradicts the one stored in known_hosts.

Told apart from UnknownHostKeyError, which is the absence of a stored key, because the two mean different things: an absent key is a host never connected to before, while a contradicted key is either a re-keyed server or an interception, and only the second of those is a security event. Being an ARC ServerError keeps it inside ARC’s server error handling, and being a paramiko.SSHException keeps it catchable alongside the exception it is raised from.

class arc.job.ssh.LogAndAcceptHostKeyPolicy[source]

A missing host key policy that reports the unknown key through ARC’s logger, then accepts it.

paramiko’s WarningPolicy emits through warnings.warn(), which ARC’s arc.common.initialize_log() filters out for the paramiko module, so nothing of it reaches the log file or the terminal. This policy logs the host and the key’s fingerprint at the warning level, and connects.

missing_host_key(client, hostname, key)[source]

Log the unknown host key and accept it.

Parameters:

client (paramiko.SSHClient) – The client the key was presented to.
hostname (str) – The address of the server that presented the key.
key (paramiko.PKey) – The host key that is not in known_hosts.

class arc.job.ssh.RejectUnknownHostKeyPolicy[source]

A missing host key policy that refuses the connection, raising UnknownHostKeyError.

missing_host_key(client, hostname, key)[source]

Refuse the unknown host key.

Parameters:

client (paramiko.SSHClient) – The client the key was presented to.
hostname (str) – The address of the server that presented the key.
key (paramiko.PKey) – The host key that is not in known_hosts.

Raises:

UnknownHostKeyError – Always.

class arc.job.ssh.SSHClient(server='', connection_attempts=1440)[source]

This is a class for communicating with remote servers via SSH.

Parameters:

server (str) – The server name as specified in ARCs’s settings file under servers as a key.
connection_attempts (int, optional) – The number of times to try connecting to the server, waiting a minute between attempts. The default keeps trying for 24 hours, which is appropriate while jobs are running. Pass a low number where blocking is worse than giving up. A permanent failure raises on the first attempt whatever this is set to, see connect().

server

The server name as specified in ARCs’s settings file under servers as a key.

Type:: str

address

The server’s address.

Type:: str

un

The username to use on the server.

Type:: str

key

A path to a file containing the SSH private key to the server. Optional: when it is not set (or set to an empty string), no explicit identity is offered and paramiko falls back to a running ssh-agent and then to the default key paths (~/.ssh/id_rsa, ~/.ssh/id_ecdsa, ~/.ssh/id_ed25519), which is how agent-based setups authenticate.

Type:: str | None

connection_attempts

The number of times to try connecting to the server.

Type:: int

_ssh

A high-level representation of a session with an SSH server.

Type:: paramiko.SSHClient

_sftp

SFTP client used to perform remote file operations.

Type:: paramiko.sftp_client.SFTPClient

_keepalive_interval

The keepalive interval this client was asked for, which connect() re-applies to every transport it opens. None until arc.job.ssh_pool.set_keepalive() sets it, since a client that is not pooled is not held open long enough to be dropped while idle.

Type:: int | None

change_mode(mode, file_name, recursive=False, remote_path='')[source]

Change the mode of a file or a directory.

Parameters:

mode (str) – The mode change to be applied, can be either octal or symbolic.
file_name (str) – The path to the file or the directory to be changed.
recursive (bool, optional) – Whether to recursively change the mode to all files under a directory.``True`` for recursively change.
remote_path (str, optional) – The directory path at which the command will be executed.

check_job_status(job_id)[source]

Check job’s status.

Parameters:: job_id (int) – The job’s ID.

Returns: str: Possible statuses: before_submission, running, errored on node xx, done, and errored: …

check_running_jobs_ids()[source]

Check all jobs submitted by the user on a server.

Returns: list: A list of job IDs.

close()[source]: Close the connection to paramiko SSHClient and SFTPClient

connect()[source]

A modulator function for _connect(). Connect to the server.

Failures that retrying cannot resolve – a rejected authentication, a host key that does not match known_hosts, and an unknown host key on a server that sets strict_host_key_checking (PERMANENT_CONNECTION_ERRORS) – raise a ServerError on the first attempt, carrying the paramiko exception as its cause – this holds whatever connection_attempts is set to, since no number of retries can resolve them. A configured key file that does not exist is permanent for the same reason, and is told apart from the transport-level OSError``s by :meth:`_is_a_missing_key_file`. Every other failure is transport-level, and is retried once a minute until ``connection_attempts attempts have been made (24 hours by default). No interval is waited out after the last attempt, so a client asked for a single attempt fails at once.

A contradicted host key raises the HostKeyMismatchError subclass of ServerError rather than a plain one, and is reported at the error level naming both fingerprints, so it is told apart from a wrong password in the log rather than reading as one more failed connection.

Raises:

HostKeyMismatchError – The server’s host key contradicts the one in known_hosts.
ServerError – Cannot connect to the server with maximum times to try, or the failure is permanent.

delete_job(job_id)[source]

Deletes a running job.

Parameters:: job_id (int | str) – The job’s ID.

delete_jobs(jobs=None)[source]

Delete all of the jobs on a specific server.

Parameters:: jobs (list[str | int], optional) – Specific ARC job IDs to delete.

delete_remote_check_files(remote_path)[source]

Delete ESS checkfiles under a remote directory (recursively). They usually take up lots of space and are not needed after ARC terminates. Pass True to the keep_checks flag in ARC to avoid deleting check files. The local counterpart of this method is arc.common.delete_check_files().

Unlike remove_dir(), this keeps the remote directory and everything in it that is not a checkfile, so a project’s outputs remain on the server after the cleanup. A failure is logged rather than raised, see delete_check_files_on_servers().

Parameters:: remote_path (str) – The remote directory path under which checkfiles will be deleted.

find_package(package_name)[source]

Find the path to the package.

Parameters:: package_name (str) – The name of the package to search for.

list_available_nodes()[source]

List available nodes on the server.

Returns:: lines of the node hostnames.
Return type:: list

list_dir(remote_path='')[source]

List directory contents.

Parameters:: remote_path (str, optional) – The directory path at which the command will be executed.

remove_dir(remote_path)[source]

Remove a directory, and everything under it, on the server.

This is the remote-cleanup primitive. ARC’s own job flow does not call it: no job removes its remote work directory today, and the caller that will is added separately. It is reached through arc.job.adapter.JobAdapter.remove_remote_files(), which supplies the job’s remote path.

Parameters:: remote_path (str) – The path to the directory to remove on the remote server.
Raises:: ServerError – If the directory could not be removed.

submit_job(remote_path, recursion=False)[source]

Submit a job to the server.

Parameters:

remote_path (str) – The remote path contains the input file and the submission script.
recursion (bool, optional) – Whether this call is within a recursion.

Returns: tuple[str, int]

A string indicate the status of job submission. Either errored or submitted.
The job ID of the submitted job.

exception arc.job.ssh.UnknownHostKeyError[source]

Raised when a server’s host key is absent from known_hosts and the server sets strict_host_key_checking.

An ARC ServerError, so ARC’s server error handling covers it, and a paramiko.SSHException, which is what a missing host key policy is expected to raise. Being a distinct type, a refused host key is told apart from a transport failure without matching on paramiko’s message text.

arc.job.ssh.check_connections(function)[source]

A decorator designned for SSHClient``to check SSH connections before calling a method. It first checks if ``self._ssh is available in a SSHClient instance and then checks if you can send ls and get response to make sure your connection still alive. If connection is bad, this decorator will reconnect the SSH channel, to avoid connection related error when executing the method.

connect() assigns self._sftp and self._ssh itself and returns nothing, so its result is not unpacked into them. Unpacking it raised TypeError: cannot unpack non-iterable NoneType object for any client that had not connected yet, which is the one case this branch exists to serve.

arc.job.ssh.check_job_status_in_stdout(job_id, stdout, server)[source]

A helper function for checking job status.

Parameters:

job_id (int) – the job ID recognized by the server.
stdout (list | str) – The output of a queue status check.
server (str) – The server name.

Returns:

The job status on the server (‘running’, ‘done’, or ‘errored’).

Return type:

str

arc.job.ssh.check_servers_known_hosts(server_dict=None, known_hosts_path=None)[source]

Report configured servers whose host keys need attention before any job is submitted.

Two offline conditions are reported, at two levels. A server with no host key at all is a warning: ARC connects to an unknown host anyway (see SSHClient._connect()), so without this the first sign of an unseeded known_hosts is a per-connection warning buried in a running job’s log, or – for a server with strict_host_key_checking – a run that appears to hang while every connection is refused. A server with contradictory entries (get_servers_with_conflicting_host_keys()) is an error: the file records two different keys as that server’s, only one of them is consulted, and which one is trusted is decided by line order rather than by anything the reader chose.

A stored key that no longer matches the key the server presents is not reported here and cannot be, since the comparison needs the server. paramiko makes it while connecting, and it surfaces as HostKeyMismatchError.

Parameters:

server_dict (dict, optional) – The servers to check. Defaults to the configured servers.
known_hosts_path (str, optional) – The known_hosts file to read. Defaults to KNOWN_HOSTS_PATH.

Returns: dict[str, str]: The address of each server that has no host key, keyed by server name.

arc.job.ssh.delete_all_arc_jobs(server_list, jobs=None)[source]

Delete all ARC-spawned jobs (with job name starting with a and a digit) from :list:servers (servers could also be a string of one server name) Make sure you know what you’re doing, so unrelated jobs won’t be deleted… Useful when terminating ARC while some (ghost) jobs are still running.

Parameters:

server_list (list) – List of servers to delete ARC jobs from.
jobs (list[str] | None) – Specific ARC job IDs to delete.

arc.job.ssh.delete_check_files_on_servers(remote_project_paths)[source]

Delete ESS checkfiles from an ARC project’s directory on all servers it ran jobs on. The local counterpart of this function is arc.common.delete_check_files(). Errors are only logged and never raised: this runs once ARC is done with the science, an unreachable server at that point is an inconvenience, not a reason to lose a run.

Each server is reached through its own single-attempt client rather than through the connection pool (arc.job.ssh_pool), for the same reason: both the pool’s factory and the fallback in borrow_ssh_client() build a client with the default 24-hour retry, so borrowing here would let a server that has gone away hold up the end of a run indefinitely. A cleanup that cannot reach a server must give up, not wait.

Parameters:: remote_project_paths (dict) – Keys are server names, values are the respective remote paths of the project’s directory on that server.

arc.job.ssh.get_host_key_fingerprint(key)[source]

Return the OpenSSH-style SHA256 fingerprint of a host key.

Parameters:: key (paramiko.PKey) – The host key to fingerprint.

Returns: str: The fingerprint, formatted as SHA256:<unpadded base64>, as reported by ssh-keygen -lf and by OpenSSH when it prompts about an unknown host.

arc.job.ssh.get_servers_missing_host_keys(server_dict=None, known_hosts_path=None)[source]

Determine which of the configured servers have no host key on this machine.

The lookup is local and offline: the known_hosts file is read, nothing is resolved and no connection is opened. paramiko’s HostKeys performs the lookup, so hashed entries (ssh-keyscan -H) and [host]:port entries are matched as OpenSSH matches them.

This reports an absent key only. Whether a stored key still matches the one a server presents is not knowable from this machine, since only the server can present it; that comparison is made by paramiko while connecting, and raises HostKeyMismatchError. What can be checked offline alongside an absent key is a known_hosts file that contradicts itself, which is get_servers_with_conflicting_host_keys().

Parameters:

server_dict (dict, optional) – The servers to check. Defaults to the configured servers.
known_hosts_path (str, optional) – The known_hosts file to read. Defaults to KNOWN_HOSTS_PATH.

Returns: dict[str, str]: The address of each server that has no host key, keyed by server name.

arc.job.ssh.get_servers_with_conflicting_host_keys(server_dict=None, known_hosts_path=None)[source]

Determine which of the configured servers have contradictory host keys on this machine.

A server legitimately has one host key per key type, and ssh-keyscan writes one line per type. More than one entry of the same type for one address means the file disagrees with itself about what that server’s key is, which is what a stale entry left behind by a rebuilt server looks like, and equally what an entry prepended to shadow the real key looks like. Only the first matching entry is ever consulted – by OpenSSH, and by the HostKeys.lookup paramiko authenticates with – so a shadowed key is trusted silently while the server’s real key is reported as a mismatch.

The check is local and offline: the known_hosts file is read, nothing is resolved and no connection is opened. It therefore cannot say which of the recorded keys is the server’s; answering that requires the key the server presents, which is compared while connecting and raises HostKeyMismatchError.

Parameters:

server_dict (dict, optional) – The servers to check. Defaults to the configured servers.
known_hosts_path (str, optional) – The known_hosts file to read. Defaults to KNOWN_HOSTS_PATH.

Returns: dict[str, list[str]]: The key types recorded more than once, sorted, keyed by server name. Servers whose entries do not contradict each other are absent.