| Title: | Scaffold and Submit Computational Jobs to CHTC Schedulers |
|---|---|
| Description: | Provides scaffolding tools to help researchers prepare and submit computational jobs to high-throughput computing (HTC) schedulers. Generates the files required to run containerized R analyses on 'HTCondor', including submit files and executable scripts, and wraps the system commands needed to stage files, submit jobs, monitor status, and retrieve results from a CHTC submit node. A submission state file stored alongside the project configuration tracks each submission across the workflow, so that input files can be uploaded and result files downloaded automatically, without constructing file lists by hand and without losing the job when the R session ends. Provides htc_start() for session management and htc_config() for connection details and SSH connection reuse guidance. Works naturally alongside 'containr' for container image management and 'toolero' for dataset splitting and project scaffolding. |
| Authors: | Erwin Lares [aut, cre] (ORCID: <https://orcid.org/0000-0002-3284-828X>) |
| Maintainer: | Erwin Lares <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.1.0.9000 |
| Built: | 2026-10-31 12:11:18 UTC |
| Source: | https://github.com/erwinlares/submitr |
htc_cancel() connects to an HTC submit node via SSH and runs
condor_rm to remove jobs from the queue – a mistaken submission (the
wrong resource request, the wrong image, 500 jobs instead of 5) can be
stopped from R rather than requiring a manual SSH session (S-G2).
htc_cancel( cluster_id = NULL, reason = NULL, config = NULL, dry_run = FALSE, verbose = FALSE, path = "." )htc_cancel( cluster_id = NULL, reason = NULL, config = NULL, dry_run = FALSE, verbose = FALSE, path = "." )
cluster_id |
An integer, character string, or |
reason |
A character string or |
config |
A named list as returned by |
dry_run |
Logical. If |
verbose |
Logical. If |
path |
A character string. Directory holding the submission state
( |
The condor_rm output as a character vector, returned invisibly.
cfg <- htc_config() job <- htc_submit(config = cfg) # Realized the resource request was wrong -- stop it before it runs htc_cancel(cluster_id = job, config = cfg, reason = "wrong request_memory")
htc_release() to un-pause a held job instead of removing it,
and htc_status() to see which jobs are running, idle, or held.
# Preview the SSH command without connecting to CHTC cfg <- list(username = "netid", server = "ap2002.chtc.wisc.edu") htc_cancel(cluster_id = 6302860, config = cfg, dry_run = TRUE) ## Not run: cfg <- htc_config() htc_cancel(cluster_id = 6302860, config = cfg) htc_cancel(cluster_id = 6302860, config = cfg, reason = "submitted by mistake") ## End(Not run)# Preview the SSH command without connecting to CHTC cfg <- list(username = "netid", server = "ap2002.chtc.wisc.edu") htc_cancel(cluster_id = 6302860, config = cfg, dry_run = TRUE) ## Not run: cfg <- htc_config() htc_cancel(cluster_id = 6302860, config = cfg) htc_cancel(cluster_id = 6302860, config = cfg, reason = "submitted by mistake") ## End(Not run)
htc_check() verifies, locally and in seconds, several things that would
otherwise only surface an hour later as a held or failed job on the
cluster (S-G4): that every file the job depends on actually exists on
this machine, that a "multiple"-mode job's subset files still match
what subdatasets.csv expects, that the resource request looks
plausible, and – best effort, when a container tool is available
locally – that the container image looks reachable.
htc_check( container_image = NULL, input_files = NULL, data_files = NULL, resources = NULL, check_image = TRUE, verbose = TRUE, path = "." )htc_check( container_image = NULL, input_files = NULL, data_files = NULL, resources = NULL, check_image = TRUE, verbose = TRUE, path = "." )
container_image |
A character string or |
input_files |
A character vector or |
data_files |
A character vector or |
resources |
A named list with |
check_image |
Logical. If |
verbose |
Logical. If |
path |
A character string. Directory holding the submission state
( |
Every argument can be resolved from the submission state, so the common case
is htc_check() with no arguments at all, run right after
htc_gen_submit() and htc_gen_executable() and before htc_upload().
A tibble with one row per issue found, columns check,
severity ("error" or "warning"), and message. Zero rows means
nothing was found. Returned invisibly; call print() explicitly or
assign it to inspect the detail behind the printed summary.
A missing input, data, or subset file is an "error": the job will not
run without it, full stop. A resource request outside typical bounds, a
container_image tagged latest, or an image that could not be
confirmed pullable are "warning"s – each might be exactly what you
intend, and none of them is something htc_check() can be certain about
from the local machine alone. htc_upload()'s own check argument only
blocks the upload on "error"s.
htc_check() only attempts to confirm container_image is pullable when
podman or docker is found on the local PATH, via
<tool> manifest inspect. A failure there is reported as a warning, not
an error, and is not conclusive either way: it may mean the image
genuinely does not exist, or simply that you are not logged in to the
registry from this machine, or that the tool timed out. When neither
tool is found, or check_image = FALSE, the probe is skipped and
reported as such.
htc_upload()'s check argument, which runs this
automatically and aborts before uploading if an "error"-level issue
is found.
tmp <- withr::local_tempdir() htc_gen_submit( container_image = "registry.doit.wisc.edu/netid/myimage:latest", resources = "small", output = tmp, path = tmp ) # check_image = FALSE skips asking podman or docker about the image, # which would otherwise contact the registry. htc_check(path = tmp, check_image = FALSE)tmp <- withr::local_tempdir() htc_gen_submit( container_image = "registry.doit.wisc.edu/netid/myimage:latest", resources = "small", output = tmp, path = tmp ) # check_image = FALSE skips asking podman or docker about the image, # which would otherwise contact the registry. htc_check(path = tmp, check_image = FALSE)
htc_collect() is the step after htc_download(). It extracts every
job's results tarball into its own folder and returns the job index: a
tibble with one row per job saying whether its results came back, where
they landed, what files they hold, and where that job's HTCondor log
files are. In "multiple" mode that is one row per subset, the HTC-side
counterpart to the tibble toolero::run_by_group() returns for a local
run.
htc_collect( tarballs = NULL, local_path = ".", extract_dir = NULL, overwrite = FALSE, path = "." )htc_collect( tarballs = NULL, local_path = ".", extract_dir = NULL, overwrite = FALSE, path = "." )
tarballs |
A named character vector or |
local_path |
A character string. The directory |
extract_dir |
A character string or |
overwrite |
Logical. If |
path |
A character string. Directory holding the submission state
( |
It works for any analysis, whatever the script wrote into its results
folder, and never opens the files themselves: their types are whatever
each job chose to save, and no single reader is right for all of them.
Read them with whatever wrote them, using output_dir and files (see
the Workflow section).
A job whose tarball is missing, or will not extract, still gets a row,
with extracted = FALSE, and the collection carries on with the other
jobs. A job whose R script fails still sends its tarball back (see
htc_gen_executable()), so a missing tarball means the job stopped
before it got that far: the container did not start, a file it needed
was not transferred, or the job was removed. Its .err and .log
files, in the err and log columns when they were downloaded, are the
place to look.
The job index: a tibble with one row per job and these columns.
group_id – the subset stem in "multiple" mode ("adelie" for
adelie.csv), the script stem in "single" mode, or the name given
in tarballs.
proc_id – the job's HTCondor process number, from its position in
the submission state; NA when it cannot be matched.
cluster_id – from the submission state; NA when absent.
extracted – whether the tarball was found and extracted.
n_files – how many files the results folder holds; NA when not
extracted.
has_record – whether the results folder holds an output record
(project-manifest.json, written by toolero::generate_manifest()).
The file is only checked for, never read. NA when not extracted.
output_dir – where the job's results folder landed; NA when not
extracted.
files – a list column of paths relative to output_dir.
log, err, out – local paths to the job's three HTCondor files,
found beside its tarball; NA when not there.
container_image – from the submission state; NA when absent.
htc_download(local_path = "downloads/")
index <- htc_collect(local_path = "downloads/")
# Which jobs came back?
index[, c("group_id", "extracted", "n_files")]
# Why did a job fail? Its .err file usually says.
readLines(index$err[!index$extracted][1])
# Full paths to every file each job produced
paths <- Map(file.path, index$output_dir, index$files)
models <- lapply(unlist(paths[index$extracted]), readRDS)
# Build a single-job tarball by hand to show what htc_collect() does with # one -- normally this tarball would have come from htc_download() after # a real job finished. job_dir <- tempfile("job") dir.create(file.path(job_dir, "output"), recursive = TRUE) saveRDS(mtcars, file.path(job_dir, "output", "mtcars.rds")) local_path <- tempfile("downloads") dir.create(local_path) tarball <- file.path(local_path, "analysis-results.tar.gz") old <- setwd(job_dir) utils::tar(tarball, "output", compression = "gzip", tar = "internal") setwd(old) index <- htc_collect( tarballs = c(analysis = tarball), extract_dir = tempfile("collected") ) index index$files[[1]]# Build a single-job tarball by hand to show what htc_collect() does with # one -- normally this tarball would have come from htc_download() after # a real job finished. job_dir <- tempfile("job") dir.create(file.path(job_dir, "output"), recursive = TRUE) saveRDS(mtcars, file.path(job_dir, "output", "mtcars.rds")) local_path <- tempfile("downloads") dir.create(local_path) tarball <- file.path(local_path, "analysis-results.tar.gz") old <- setwd(job_dir) utils::tar(tarball, "output", compression = "gzip", tar = "internal") setwd(old) index <- htc_collect( tarballs = c(analysis = tarball), extract_dir = tempfile("collected") ) index index$files[[1]]
htc_config() creates or reads an htc.cfg file that stores the
connection details needed by htc_upload(), htc_submit(),
htc_status(), and htc_download(). On first use it prompts
interactively for your username and server address, writes htc.cfg
to path, and adds it to .gitignore. Subsequent calls read the
existing file.
htc_config( username = NULL, server = NULL, path = ".", overwrite = FALSE, project_config = NULL, check_server = getOption("submitr.check_server", default = TRUE) )htc_config( username = NULL, server = NULL, path = ".", overwrite = FALSE, project_config = NULL, check_server = getOption("submitr.check_server", default = TRUE) )
username |
A character string. Your HTC username (NetID), e.g.
|
server |
A character string. The HTC submit server hostname.
Defaults to |
path |
A character string. Directory where |
overwrite |
Logical. If |
project_config |
A character string or |
check_server |
Logical. If |
A named list with elements username and server, plus a
project element when project_config is supplied, returned
invisibly.
Two options adjust how much htc_config() does on your behalf. Both
default to TRUE, and both are most useful set once for a whole session
or test suite rather than per call.
submitr.verbose controls the progress messages ("Reading HTC config
from ...", "Checking connectivity to ..."). Setting it to FALSE leaves
warnings and errors intact.
submitr.check_server controls the reachability probe described under
check_server above. The argument takes precedence when supplied, so the
option sets the default and a call can still override it.
Each call to htc_upload(), htc_submit(), htc_status(), or
htc_download() opens a new SSH connection to the submit server,
which triggers a Duo MFA prompt each time. You can avoid this by
configuring SSH connection reuse (ControlMaster) in your
~/.ssh/config file. Add the following block:
Host *.chtc.wisc.edu ControlMaster auto ControlPersist 2h ControlPath ~/.ssh/connections/%r@%h:%p
Then create the connections directory:
mkdir -p ~/.ssh/connections
After this, only the first connection in a two-hour window will require Duo authentication. Full documentation: https://chtc.cs.wisc.edu/uw-research-computing/configure-ssh
project_config is how submitr learns the layout a toolero project
already settled on, instead of retyping it. toolero::init_project()
writes _toolero.yml to a project's root once it has resolved the
folder set, and that file records two things: folders, the full list
of folders the project uses, and conventions, the names the family
resolves rather than assumes (output_dir, script_dir, split_dir).
Passing project_config = "_toolero.yml" reads that file once and
returns it under config$project:
cfg <- htc_config(project_config = "_toolero.yml") cfg$project$conventions$output_dir #> [1] "output"
As of this release (S-G5), passing the returned config's project
element on to htc_gen_executable() and htc_gen_submit() via their own
config argument lets results_folder and queue_from default from
conventions$output_dir and conventions$split_dir, the way
containr::generate_dockerfile() already reads the same file for its own
purposes:
cfg <- htc_config(project_config = "_toolero.yml") htc_gen_executable(r_script = "R/analysis.R", config = cfg) htc_gen_submit(mode = "multiple", config = cfg)
r_script itself is not defaulted from conventions$script_dir: the
convention names a directory, not a file, and the script's own filename
is project-specific information _toolero.yml has no way to record.
htc.cfg contains your username and server address. Neither is
sensitive on its own, but htc_config() adds htc.cfg to
.gitignore on creation to avoid accidentally committing
institutional account details to a public repository.
# Preview what htc_config() would return without writing any files cfg <- list(username = "netid", server = "ap2002.chtc.wisc.edu") str(cfg) ## Not run: # Interactive first-time setup cfg <- htc_config() # Non-interactive setup (for scripts) cfg <- htc_config( username = "erwin.lares", server = "ap2002.chtc.wisc.edu" ) # Force recreation of htc.cfg cfg <- htc_config(overwrite = TRUE) # Read the config without probing the server, e.g. in a script or on CI cfg <- htc_config(check_server = FALSE) # Or turn the probe off for a whole session options(submitr.check_server = FALSE) # Fold in a toolero project's own folder layout and conventions cfg <- htc_config(project_config = "_toolero.yml") cfg$project$conventions$output_dir # Use in other functions htc_upload(files = c("job.sub", "job.sh"), config = cfg) ## End(Not run)# Preview what htc_config() would return without writing any files cfg <- list(username = "netid", server = "ap2002.chtc.wisc.edu") str(cfg) ## Not run: # Interactive first-time setup cfg <- htc_config() # Non-interactive setup (for scripts) cfg <- htc_config( username = "erwin.lares", server = "ap2002.chtc.wisc.edu" ) # Force recreation of htc.cfg cfg <- htc_config(overwrite = TRUE) # Read the config without probing the server, e.g. in a script or on CI cfg <- htc_config(check_server = FALSE) # Or turn the probe off for a whole session options(submitr.check_server = FALSE) # Fold in a toolero project's own folder layout and conventions cfg <- htc_config(project_config = "_toolero.yml") cfg$project$conventions$output_dir # Use in other functions htc_upload(files = c("job.sub", "job.sh"), config = cfg) ## End(Not run)
htc_download() copies one or more files from a directory on an HTC
submit node to a local directory via scp. It is the final step in the
job submission workflow – called after htc_status() confirms all jobs
have completed.
htc_download( files = NULL, cluster_id = NULL, remote_path = NULL, local_path = ".", config = NULL, dry_run = FALSE, verbose = FALSE, path = "." )htc_download( files = NULL, cluster_id = NULL, remote_path = NULL, local_path = ".", config = NULL, dry_run = FALSE, verbose = FALSE, path = "." )
files |
A character vector or |
cluster_id |
A character string or |
remote_path |
A character string or |
local_path |
A character string. The local directory where downloaded
files will be saved. Defaults to |
config |
A named list as returned by |
dry_run |
Logical. If |
verbose |
Logical. If |
path |
A character string. Directory holding the submission state
( |
When cluster_id is supplied without files, the function uses the
submission state built up by htc_gen_submit(), htc_gen_executable(),
and htc_submit() to determine which files to download. For single-mode
jobs, this includes the results tarball and the log, error, and output
files. For multiple-mode jobs, the function reads the subset names from
the submission state and constructs per-job tarball names and per-process log
file patterns.
Glob patterns such as "*.tar.gz" are supported when using the files
argument and are evaluated on the remote server, not locally.
Called for its side effects. Returns invisible(NULL).
When files is NULL, the function resolves the file list from the
submission state. The submission state is built automatically as you call
htc_gen_submit(), htc_gen_executable(), and htc_submit() during
the normal workflow. No extra steps are needed.
For a single-mode job:
The results tarball (e.g. "analysis-results.tar.gz")
Log files: "{cluster_id}-0-job.log", ".err", ".out"
For a multiple-mode job:
Per-subset tarballs (e.g. "analysis-adelie-results.tar.gz")
Log files for each process: "{cluster_id}-{0,1,...}-job.log", etc.
htc_download() is the final system-facing step in the submitr workflow.
Call it after htc_status() confirms all jobs have completed.
# Automatic: uses the submission state to determine what to download
htc_start()
htc_gen_submit(...)
htc_gen_executable(...)
htc_upload(...)
job <- htc_submit("analysis.sub")
htc_status(cluster_id = job, watch = TRUE)
htc_download()
When using files directly, glob patterns are passed to the remote
shell for evaluation so they match files on the submit node, not on
your local machine. The pattern is single-quoted in the scp command
to prevent local shell expansion.
Each call to htc_download() opens a new SSH connection. If you have
not configured ControlMaster in your ~/.ssh/config, this will trigger
a Duo MFA prompt. Run htc_config() for setup guidance.
# Preview the scp command without connecting to CHTC cfg <- list(username = "netid", server = "ap2002.chtc.wisc.edu") htc_download(files = "*.tar.gz", config = cfg, dry_run = TRUE) ## Not run: # All remaining examples require a live CHTC connection # Automatic download after a workflow htc_start() htc_gen_submit(...) htc_gen_executable(...) htc_upload(...) job <- htc_submit("job.sub") htc_status(cluster_id = job, watch = TRUE) htc_download() # Download by cluster ID htc_download(cluster_id = "6590895") # Download specific files using globs htc_download(files = "*.tar.gz", local_path = "downloads/") # Download logs only htc_download(files = c("*.log", "*.err", "*.out"), local_path = "logs/") ## End(Not run)# Preview the scp command without connecting to CHTC cfg <- list(username = "netid", server = "ap2002.chtc.wisc.edu") htc_download(files = "*.tar.gz", config = cfg, dry_run = TRUE) ## Not run: # All remaining examples require a live CHTC connection # Automatic download after a workflow htc_start() htc_gen_submit(...) htc_gen_executable(...) htc_upload(...) job <- htc_submit("job.sub") htc_status(cluster_id = job, watch = TRUE) htc_download() # Download by cluster ID htc_download(cluster_id = "6590895") # Download specific files using globs htc_download(files = "*.tar.gz", local_path = "downloads/") # Download logs only htc_download(files = c("*.log", "*.err", "*.out"), local_path = "logs/") ## End(Not run)
htc_gen_executable() writes a ready-to-use bash script (.sh) that
HTCondor runs inside the container for each job. The script changes to
HTCondor's writable scratch directory, creates an output folder, runs
the R script via Rscript, and compresses the results into a tarball
for transfer back to the submit node. The tarball is built whether or
not the R script succeeds (see the section on failed jobs below).
htc_gen_executable( output_file = NULL, r_script = NULL, data_files = NULL, results_folder = NULL, home_dir = "/home", mode = "single", set_executable = TRUE, verbose = FALSE, comments = FALSE, output = ".", config = NULL, path = "." )htc_gen_executable( output_file = NULL, r_script = NULL, data_files = NULL, results_folder = NULL, home_dir = "/home", mode = "single", set_executable = TRUE, verbose = FALSE, comments = FALSE, output = ".", config = NULL, path = "." )
output_file |
A character string or |
r_script |
A character string. Name of the R script that HTCondor
will run, e.g. |
data_files |
A character vector or |
results_folder |
A character string or |
home_dir |
A character string. The working directory inside the
container where baked-in |
mode |
A character string. Execution mode. |
set_executable |
Logical. If |
verbose |
Logical. If |
comments |
Logical. If |
output |
A character string. Directory where the shell script will
be written. Defaults to |
config |
A named list as returned by |
path |
A character string. Directory where the submission state
( |
Called for its side effects. Writes a bash script to
file.path(output, output_file) and sets executable permissions when
set_executable = TRUE. Returns invisible(NULL).
The generated script draws on two different sources for its inputs, and reads each one a different way:
The R script – r_script is not baked into the container image.
It travels to the execute node as an uploaded job input file, listed in
transfer_input_files by htc_gen_submit(). HTCondor's file transfer
does not preserve subdirectories, so whatever you pass as r_script
lands flat, by basename, in the scratch directory alongside the
executable script itself. The Rscript line therefore refers to it by
a bare relative name (e.g. Rscript analysis.R), not an absolute,
home_dir-prefixed path. This is deliberate: editing the analysis
script only requires re-uploading it and resubmitting the job, with no
container rebuild or registry push in between.
Data files – data_files are the opposite case: they are baked
into the container at build time by containr::generate_dockerfile()
and live under home_dir (default "/home"). The Rscript line
passes these as absolute paths (e.g. /home/data-raw/sample.csv) so
they are found regardless of the working directory. Baking data in
rather than uploading it keeps large or unchanging reference data out
of every job's file transfer.
Writing – the script changes to HTCondor's scratch directory
(_CONDOR_SCRATCH_DIR) before creating the output folder. This
directory is writable and is where HTCondor looks for
transfer_output_files. The R script writes outputs to "output/"
using a relative path, which resolves to the scratch directory.
This separation means the R script stays portable – "output/" works
in RStudio, in quarto render, and on HTCondor – while the .sh
script handles the HTCondor-specific directory setup.
The script runs under set -euo pipefail, so any failing command stops
it – except the Rscript line, which is bracketed by set +e and
set -e so that its exit status is recorded rather than fatal. The
results folder is then compressed as usual, and the script exits with
the status R returned. A failed job therefore still sends back its
tarball, holding whatever the analysis wrote before it stopped (and,
with toolero::save_output(), the accumulator row recording the failed
write), while HTCondor still sees a non-zero exit code and reports the
job as failed rather than completed. Without this, a failed job would
leave no tarball behind, and HTCondor would hold the job for a missing
transfer_output_files entry instead of letting it finish.
The executable script generated by htc_gen_executable() is the file
referenced by the executable argument in htc_gen_submit(). The two
functions should always use the same mode. In "multiple" mode,
HTCondor passes each subset filename to the script as ${1}, which is
forwarded to the R script as a positional argument. The R script must
be written to accept this argument – the recommended approach is
toolero::detect_execution_context():
context <- toolero::detect_execution_context() input_file <- switch(context, interactive = "data-raw/sample.csv", quarto = params$input_file, rscript = commandArgs(trailingOnly = TRUE)[1] )
# output writes the generated .sh file; path is where the submission state # (htc-manifest.yml) gets read from and written to. The two are # independent arguments (see @param path), so both must point at the # same scratch directory here to keep the submission state out of the current # working directory. tmp <- tempdir() # Single-job executable script with baked-in data htc_gen_executable( r_script = "R/analysis.R", data_files = "data-raw/sample.csv", output = tmp, path = tmp ) # Multiple-job executable script htc_gen_executable( r_script = "R/analysis.R", mode = "multiple", output = tmp, path = tmp ) # Custom names with annotations htc_gen_executable( output_file = "run.sh", r_script = "R/run-analysis.R", data_files = c("data-raw/train.csv", "data-raw/test.csv"), comments = TRUE, verbose = TRUE, output = tmp, path = tmp )# output writes the generated .sh file; path is where the submission state # (htc-manifest.yml) gets read from and written to. The two are # independent arguments (see @param path), so both must point at the # same scratch directory here to keep the submission state out of the current # working directory. tmp <- tempdir() # Single-job executable script with baked-in data htc_gen_executable( r_script = "R/analysis.R", data_files = "data-raw/sample.csv", output = tmp, path = tmp ) # Multiple-job executable script htc_gen_executable( r_script = "R/analysis.R", mode = "multiple", output = tmp, path = tmp ) # Custom names with annotations htc_gen_executable( output_file = "run.sh", r_script = "R/run-analysis.R", data_files = c("data-raw/train.csv", "data-raw/test.csv"), comments = TRUE, verbose = TRUE, output = tmp, path = tmp )
htc_gen_submit() writes a ready-to-use HTCondor submit file (.sub)
for running a containerized R job on an HTC cluster such as CHTC. It
supports both single-job and multiple-job submission modes.
htc_gen_submit( output_file = "job.sub", container_image = NULL, executable = NULL, r_script = NULL, input_files = NULL, output_files = NULL, mode = "single", queue = 1L, queue_from = NULL, resources = "small", custom_resources = NULL, gpu = FALSE, gpu_options = NULL, verbose = FALSE, comments = FALSE, output = ".", config = NULL, path = "." )htc_gen_submit( output_file = "job.sub", container_image = NULL, executable = NULL, r_script = NULL, input_files = NULL, output_files = NULL, mode = "single", queue = 1L, queue_from = NULL, resources = "small", custom_resources = NULL, gpu = FALSE, gpu_options = NULL, verbose = FALSE, comments = FALSE, output = ".", config = NULL, path = "." )
output_file |
A character string. Name of the submit file to write.
Must end in |
container_image |
A character string. The container image to use,
e.g. |
executable |
A character string or |
r_script |
A character string. The R script the job runs, e.g.
|
input_files |
A character vector. Files to transfer to the job's
working directory before execution, given as paths on this machine,
e.g. |
output_files |
A character vector. Files to transfer back from the
job's working directory after execution. When not supplied, it is
derived from |
mode |
A character string. Submission mode. |
queue |
A positive integer. Number of identical jobs to submit.
Only used when |
queue_from |
A character string or |
resources |
A character string. Compute resource preset. One of
|
custom_resources |
A named list. Required when |
gpu |
Logical. If |
gpu_options |
A named list or |
verbose |
Logical. If |
comments |
Logical. If |
output |
A character string. Directory where the submit file (and,
in |
config |
A named list as returned by |
path |
A character string. Directory where the submission state
( |
Called for its side effects. Writes an HTCondor submit file to
file.path(output, output_file). In "multiple" mode also writes
subdatasets.csv to output. Returns invisible(NULL).
When mode = "multiple", HTCondor passes each subset filename to the
executable as a positional argument via arguments = $(file). Your R
script must be written to accept and use this argument. The recommended
approach is to use toolero::detect_execution_context() in your analysis
script, which resolves the input file path correctly across interactive,
Quarto, and Rscript execution contexts:
context <- toolero::detect_execution_context() input_file <- switch(context, interactive = "data/penguins.csv", quarto = params$input_file, rscript = commandArgs(trailingOnly = TRUE)[1] ) data <- readr::read_csv(input_file)
The typical workflow is:
Write and develop your analysis in analysis.qmd using
toolero::detect_execution_context() for data loading.
Split your dataset with toolero::write_by_group(manifest = TRUE) to
produce subset CSV files and a manifest.csv.
Strip analysis.qmd to R/analysis.R with knitr::purl().
Call htc_gen_submit(mode = "multiple", queue_from = "manifest.csv")
to produce the submit file and subdatasets.csv.
Copy R/analysis.R, the subset data files, analysis.sub,
analysis.sh, and subdatasets.csv to CHTC and submit.
Resource presets are loaded at runtime from inst/extdata/htc-resources.yml.
To customize presets for a specific project, copy that file to your project
directory as htc-resources.yml and edit the values. htc_gen_submit()
checks for a local htc-resources.yml in the working directory first,
falling back to the package default if none is found.
submitr 0.1.0 named the file htc-resources.yaml. A local file by that
name is still read, with a warning asking for it to be renamed, for this
release only; the family writes YAML files with the .yml extension.
When both names are present, htc-resources.yml is used and the old
file is ignored, also with a warning.
# output writes the generated .sub file; path is where the submission state # (htc-manifest.yml) gets read from and written to. The two are # independent arguments (see @param path), so both must point at the # same scratch directory here to keep the submission state out of the current # working directory. tmp <- tempdir() # Single-job submit file with default resource preset htc_gen_submit(output = tmp, path = tmp) # Single-job submit file with medium resources and file transfer htc_gen_submit( output_file = "analysis.sub", container_image = "docker://registry.doit.wisc.edu/netid/myimage", executable = "analysis.sh", r_script = "R/analysis.R", input_files = "R/analysis.R", resources = "medium", output = tmp, path = tmp ) # Annotated submit file useful for learning HTCondor syntax htc_gen_submit( output_file = "annotated.sub", comments = TRUE, verbose = TRUE, output = tmp, path = tmp ) # Custom resource request htc_gen_submit( resources = "custom", custom_resources = list(cpus = 2, memory = "8GB", disk = "4GB"), output = tmp, path = tmp ) ## Not run: # Multiple-job submit file driven by a write_by_group() job manifest htc_gen_submit( output_file = "analysis.sub", container_image = "docker://registry.doit.wisc.edu/netid/myimage", executable = "analysis.sh", input_files = "analysis.R", mode = "multiple", queue_from = "data/manifest.csv", resources = "medium", output = "." ) ## End(Not run)# output writes the generated .sub file; path is where the submission state # (htc-manifest.yml) gets read from and written to. The two are # independent arguments (see @param path), so both must point at the # same scratch directory here to keep the submission state out of the current # working directory. tmp <- tempdir() # Single-job submit file with default resource preset htc_gen_submit(output = tmp, path = tmp) # Single-job submit file with medium resources and file transfer htc_gen_submit( output_file = "analysis.sub", container_image = "docker://registry.doit.wisc.edu/netid/myimage", executable = "analysis.sh", r_script = "R/analysis.R", input_files = "R/analysis.R", resources = "medium", output = tmp, path = tmp ) # Annotated submit file useful for learning HTCondor syntax htc_gen_submit( output_file = "annotated.sub", comments = TRUE, verbose = TRUE, output = tmp, path = tmp ) # Custom resource request htc_gen_submit( resources = "custom", custom_resources = list(cpus = 2, memory = "8GB", disk = "4GB"), output = tmp, path = tmp ) ## Not run: # Multiple-job submit file driven by a write_by_group() job manifest htc_gen_submit( output_file = "analysis.sub", container_image = "docker://registry.doit.wisc.edu/netid/myimage", executable = "analysis.sh", input_files = "analysis.R", mode = "multiple", queue_from = "data/manifest.csv", resources = "medium", output = "." ) ## End(Not run)
htc_release() connects to an HTC submit node via SSH and runs
condor_release to un-pause jobs that HTCondor has held (S-G2) –
typically after fixing whatever caused the hold, such as raising a
resource request reported by htc_status()'s hold-reason output.
htc_release( cluster_id = NULL, config = NULL, dry_run = FALSE, verbose = FALSE, path = "." )htc_release( cluster_id = NULL, config = NULL, dry_run = FALSE, verbose = FALSE, path = "." )
cluster_id |
An integer, character string, or |
config |
A named list as returned by |
dry_run |
Logical. If |
verbose |
Logical. If |
path |
A character string. Directory holding the submission state
( |
The condor_release output as a character vector, returned
invisibly.
cfg <- htc_config() htc_status(cluster_id = 6302860, config = cfg) # ... shows job 6302860.0 held; hold reason printed below the table ... # ... fix the resource request that caused the hold, then: htc_release(cluster_id = 6302860, config = cfg)
htc_cancel() to remove a job instead of releasing it, and
htc_status() to see hold reasons.
# Preview the SSH command without connecting to CHTC cfg <- list(username = "netid", server = "ap2002.chtc.wisc.edu") htc_release(cluster_id = 6302860, config = cfg, dry_run = TRUE) ## Not run: cfg <- htc_config() htc_release(cluster_id = 6302860, config = cfg) ## End(Not run)# Preview the SSH command without connecting to CHTC cfg <- list(username = "netid", server = "ap2002.chtc.wisc.edu") htc_release(cluster_id = 6302860, config = cfg, dry_run = TRUE) ## Not run: cfg <- htc_config() htc_release(cluster_id = 6302860, config = cfg) ## End(Not run)
htc_ssh_setup() writes the ControlMaster block described in
htc_config()'s own setup guidance to ~/.ssh/config and creates the
connections directory it references, so that reuse can be set up without
leaving R (S-G3). Every function in this package that opens an SSH
connection – htc_upload(), htc_submit(), htc_status(),
htc_download(), htc_cancel(), and htc_release() – benefits, since
each currently opens a fresh connection per call and triggers a separate
Duo MFA prompt without this.
htc_ssh_setup( ssh_config_path = "~/.ssh/config", connections_dir = "~/.ssh/connections", host_pattern = "*.chtc.wisc.edu", dry_run = FALSE )htc_ssh_setup( ssh_config_path = "~/.ssh/config", connections_dir = "~/.ssh/connections", host_pattern = "*.chtc.wisc.edu", dry_run = FALSE )
ssh_config_path |
A character string. Path to the SSH client config
file to append to. Defaults to |
connections_dir |
A character string. Path to the directory
|
host_pattern |
A character string. The |
dry_run |
Logical. If |
Called for its side effects. Returns invisible(NULL).
If a Host block matching host_pattern already exists in
ssh_config_path, htc_ssh_setup() leaves the file untouched and says
so – it never edits or replaces an existing block, since the existing
one may have been customized deliberately. Remove or edit it by hand
first if you want htc_ssh_setup() to write a fresh one.
ControlPath relies on Unix domain sockets, which native Windows OpenSSH
(as shipped with Windows 10/11) has historically supported inconsistently
across versions. htc_ssh_setup() still writes the block on Windows –
it may simply work, particularly on a current OpenSSH release – but if
ssh subsequently errors mentioning ControlPath or a socket, connection
reuse is not supported on this system: remove the added block and expect
a Duo MFA prompt on every call instead. OpenSSH bundled with WSL or with
Git Bash is more likely to support this than the native Windows client.
htc_config(), whose own first-run message describes this same
block for manual setup.
# Preview without touching any files htc_ssh_setup(dry_run = TRUE) ## Not run: htc_ssh_setup() # Apply only to one specific submit node htc_ssh_setup(host_pattern = "ap2002.chtc.wisc.edu") ## End(Not run)# Preview without touching any files htc_ssh_setup(dry_run = TRUE) ## Not run: htc_ssh_setup() # Apply only to one specific submit node htc_ssh_setup(host_pattern = "ap2002.chtc.wisc.edu") ## End(Not run)
htc_start() calls htc_config() to read or create the connection
configuration, then stores the result as a session-level option so
that subsequent htc_*() functions can use it without requiring an
explicit config argument on every call.
htc_start(...)htc_start(...)
... |
Arguments passed to |
After calling htc_start(), functions like htc_upload(),
htc_submit(), htc_status(), and htc_download() will
automatically use the stored configuration when config = NULL
(the default). You can still pass config explicitly to any
function to override the session config.
The session config is stored via options(submitr.config = ...) and
is cleared automatically when the R session ends. To clear it
manually, call options(submitr.config = NULL).
The submission state built up by htc_gen_submit(), htc_gen_executable(),
and htc_submit() is persisted to disk separately (see
htc_upload()/htc_download()), so calling htc_start() again –
for example, in a new R session – does not discard it.
Invisibly returns the config list (same as htc_config()).
## Not run: # Start a session -- all subsequent htc_*() calls use this config htc_start() # Now these work without config = cfg htc_upload(files = c("job.sub", "job.sh")) htc_submit(submit_file = "job.sub") htc_status(cluster_id = 6351616) htc_download(files = "*.tar.gz") # You can still override for a specific call other_cfg <- htc_config(path = "other-project/") htc_upload(files = "job.sub", config = other_cfg) # Clear the session config manually options(submitr.config = NULL) ## End(Not run)## Not run: # Start a session -- all subsequent htc_*() calls use this config htc_start() # Now these work without config = cfg htc_upload(files = c("job.sub", "job.sh")) htc_submit(submit_file = "job.sub") htc_status(cluster_id = 6351616) htc_download(files = "*.tar.gz") # You can still override for a specific call other_cfg <- htc_config(path = "other-project/") htc_upload(files = "job.sub", config = other_cfg) # Clear the session config manually options(submitr.config = NULL) ## End(Not run)
htc_status() connects to an HTC submit node via SSH and runs
condor_q to report the status of jobs in the queue. By default it
shows all of your jobs. Optionally filter by cluster ID to monitor a
specific submission.
htc_status( cluster_id = NULL, config = NULL, watch = FALSE, interval = 60L, dry_run = FALSE, verbose = FALSE, path = ".", show_hold_reason = TRUE )htc_status( cluster_id = NULL, config = NULL, watch = FALSE, interval = 60L, dry_run = FALSE, verbose = FALSE, path = ".", show_hold_reason = TRUE )
cluster_id |
An integer, character string, or |
config |
A named list as returned by |
watch |
Logical. If |
interval |
A positive integer. Number of seconds to wait between
polls when |
dry_run |
Logical. If |
verbose |
Logical. If |
path |
A character string. Directory holding the submission state
( |
show_hold_reason |
Logical. If |
When watch = TRUE, htc_status() polls the queue repeatedly at a
fixed interval until all jobs in the cluster have completed, printing
a timestamped snapshot after each poll.
Called for its side effects. Prints the condor_q output (and,
when applicable, hold reasons) to the console. Returns the most recent
condor_q output invisibly as a character vector.
HTCondor reports each job's status with a single letter:
| Code | Meaning |
| I | Idle -- waiting for a matching execute node |
| R | Running -- currently executing |
| H | Held -- paused, usually due to an error |
| C | Completed -- finished successfully |
| X | Removed -- cancelled |
| S | Suspended |
Jobs disappear from condor_q once they complete and their output has
been transferred back to the submit node. Use htc_download() to retrieve
completed job output.
A held (H) job is not a lost cause: htc_status() surfaces the hold
reason automatically (see show_hold_reason), and once the underlying
problem is fixed – most often the resource request – htc_release()
resumes it without resubmitting. To abandon a job entirely instead, use
htc_cancel().
cfg <- htc_config() # One-shot status check htc_status(config = cfg) # Monitor a specific cluster until completion htc_status(cluster_id = 6302860, config = cfg, watch = TRUE)
Each poll in watch mode opens a new SSH connection. Configuring
ControlMaster in your ~/.ssh/config (see htc_config()) is strongly
recommended when using watch = TRUE to avoid repeated Duo MFA prompts.
htc_cancel() to remove a job, and htc_release() to resume a
held one after fixing what caused the hold.
# Preview the SSH command without connecting to CHTC cfg <- list(username = "netid", server = "ap2002.chtc.wisc.edu") htc_status(config = cfg, dry_run = TRUE) # Preview with a specific cluster ID htc_status(cluster_id = 6302860, config = cfg, dry_run = TRUE) ## Not run: # All remaining examples require a live CHTC connection cfg <- htc_config() # Check all your jobs htc_status(config = cfg) # Check a specific cluster htc_status(cluster_id = 6302860, config = cfg) # Watch a cluster until all jobs complete (polls every 60 seconds) htc_status(cluster_id = 6302860, config = cfg, watch = TRUE) # Watch with a shorter polling interval htc_status(cluster_id = 6302860, config = cfg, watch = TRUE, interval = 30) ## End(Not run)# Preview the SSH command without connecting to CHTC cfg <- list(username = "netid", server = "ap2002.chtc.wisc.edu") htc_status(config = cfg, dry_run = TRUE) # Preview with a specific cluster ID htc_status(cluster_id = 6302860, config = cfg, dry_run = TRUE) ## Not run: # All remaining examples require a live CHTC connection cfg <- htc_config() # Check all your jobs htc_status(config = cfg) # Check a specific cluster htc_status(cluster_id = 6302860, config = cfg) # Watch a cluster until all jobs complete (polls every 60 seconds) htc_status(cluster_id = 6302860, config = cfg, watch = TRUE) # Watch with a shorter polling interval htc_status(cluster_id = 6302860, config = cfg, watch = TRUE, interval = 30) ## End(Not run)
htc_submit() connects to an HTC submit node via SSH and runs
condor_submit on a submit file that has already been uploaded with
htc_upload(). It changes into the remote directory before submitting
so that relative paths in the submit file resolve correctly.
htc_submit( submit_file = NULL, remote_path = NULL, config = NULL, dry_run = FALSE, verbose = FALSE, path = "." )htc_submit( submit_file = NULL, remote_path = NULL, config = NULL, dry_run = FALSE, verbose = FALSE, path = "." )
submit_file |
A character string or |
remote_path |
A character string or |
config |
A named list as returned by |
dry_run |
Logical. If |
verbose |
Logical. If |
path |
A character string. Directory holding the submission state
( |
The cluster ID assigned by HTCondor as a character string,
returned invisibly. Pass it directly to htc_status() to monitor
job progress. Returns invisible(NULL) if the cluster ID cannot be
parsed from the condor_submit output.
htc_submit() is the second system-facing step in the submitr workflow.
Call it after uploading your files with htc_upload(). The returned
cluster ID can be passed directly to htc_status().
cfg <- htc_config()
htc_upload(
files = c("job.sub", "job.sh", "analysis.R"),
config = cfg
)
cluster_id <- htc_submit(submit_file = "job.sub", config = cfg)
htc_status(cluster_id = cluster_id, config = cfg, watch = TRUE)
remote_path must match htc_upload()
htc_submit() runs cd remote_path && condor_submit submit_file on the
submit node. HTCondor resolves all paths in the submit file relative to
the directory where condor_submit is called. If remote_path does not
match the directory where files were uploaded, HTCondor will not find the
executable, input files, or output destinations and the job will fail.
Each call to htc_submit() opens a new SSH connection. If you have not
configured ControlMaster in your ~/.ssh/config, this will trigger a
Duo MFA prompt. Run htc_config() for setup guidance.
# Preview the SSH command without connecting to CHTC cfg <- list(username = "netid", server = "ap2002.chtc.wisc.edu") htc_submit(submit_file = "job.sub", config = cfg, dry_run = TRUE) ## Not run: # All remaining examples require a live CHTC connection cfg <- htc_config() # Submit using default remote path htc_submit(submit_file = "job.sub", config = cfg) # Submit from a specific remote directory htc_submit( submit_file = "analysis.sub", remote_path = "~/projects/penguins/", config = cfg ) # Submit with verbose output to see condor_submit response htc_submit( submit_file = "job.sub", config = cfg, verbose = TRUE ) ## End(Not run)# Preview the SSH command without connecting to CHTC cfg <- list(username = "netid", server = "ap2002.chtc.wisc.edu") htc_submit(submit_file = "job.sub", config = cfg, dry_run = TRUE) ## Not run: # All remaining examples require a live CHTC connection cfg <- htc_config() # Submit using default remote path htc_submit(submit_file = "job.sub", config = cfg) # Submit from a specific remote directory htc_submit( submit_file = "analysis.sub", remote_path = "~/projects/penguins/", config = cfg ) # Submit with verbose output to see condor_submit response htc_submit( submit_file = "job.sub", config = cfg, verbose = TRUE ) ## End(Not run)
htc_upload() copies one or more local files or directories to a
directory on an HTC submit node via scp. It is the first step in the
job submission workflow – files must be present on the submit node before
htc_submit() can run condor_submit.
htc_upload( files = NULL, remote_path = NULL, config = NULL, dry_run = FALSE, verbose = FALSE, path = ".", check = FALSE )htc_upload( files = NULL, remote_path = NULL, config = NULL, dry_run = FALSE, verbose = FALSE, path = ".", check = FALSE )
files |
A character vector or |
remote_path |
A character string or |
config |
A named list as returned by |
dry_run |
Logical. If |
verbose |
Logical. If |
path |
A character string. Directory holding the submission state
( |
check |
Logical. If |
Called for its side effects. Returns invisible(NULL).
htc_upload() is the first system-facing step in the submitr workflow.
Call it after generating your submit file and executable script with
htc_gen_submit() and htc_gen_executable(), and before calling
htc_submit().
The typical sequence relies on automatic resolution from the submission
state, so files can usually be omitted:
cfg <- htc_config()
htc_gen_submit(executable = "job.sh", r_script = "R/analysis.R",
input_files = "R/analysis.R")
htc_gen_executable(r_script = "R/analysis.R")
htc_upload(config = cfg)
htc_submit(submit_file = "job.sub", config = cfg)
Pass files explicitly to upload a specific set of files instead:
htc_upload(
files = c("job.sub", "job.sh", "analysis.R", "data.csv"),
config = cfg
)
Each call to htc_upload() opens a new SSH connection. If you have not
configured ControlMaster in your ~/.ssh/config, this will trigger a
Duo MFA prompt. Run htc_config() for setup guidance.
# Preview the scp command without connecting to CHTC cfg <- list(username = "netid", server = "ap2002.chtc.wisc.edu") tmp <- tempfile(fileext = ".sub") writeLines("queue 1", tmp) htc_upload(files = tmp, config = cfg, dry_run = TRUE) ## Not run: # All remaining examples require a live CHTC connection cfg <- htc_config() # Resolve files automatically from the submission state htc_gen_submit(executable = "job.sh", r_script = "R/analysis.R", input_files = "R/analysis.R") htc_gen_executable(r_script = "R/analysis.R") htc_upload(config = cfg) # Upload a single file htc_upload(files = "job.sub", config = cfg) # Upload multiple files htc_upload( files = c("job.sub", "job.sh", "analysis.R"), config = cfg ) # Upload a directory htc_upload(files = "jobs/", config = cfg) # Upload to a specific remote directory htc_upload( files = c("job.sub", "job.sh"), remote_path = "~/projects/penguins/", config = cfg ) ## End(Not run)# Preview the scp command without connecting to CHTC cfg <- list(username = "netid", server = "ap2002.chtc.wisc.edu") tmp <- tempfile(fileext = ".sub") writeLines("queue 1", tmp) htc_upload(files = tmp, config = cfg, dry_run = TRUE) ## Not run: # All remaining examples require a live CHTC connection cfg <- htc_config() # Resolve files automatically from the submission state htc_gen_submit(executable = "job.sh", r_script = "R/analysis.R", input_files = "R/analysis.R") htc_gen_executable(r_script = "R/analysis.R") htc_upload(config = cfg) # Upload a single file htc_upload(files = "job.sub", config = cfg) # Upload multiple files htc_upload( files = c("job.sub", "job.sh", "analysis.R"), config = cfg ) # Upload a directory htc_upload(files = "jobs/", config = cfg) # Upload to a specific remote directory htc_upload( files = c("job.sub", "job.sh"), remote_path = "~/projects/penguins/", config = cfg ) ## End(Not run)