The results folder written by htc_gen_executable() is now output/
rather than results/, matching the folder convention used across
toolero and containr. Analysis scripts that write to results/ will
produce an empty tarball until they are updated; toolero::save_output()
handles this for you.
Results tarballs are named <script stem>[-<subset stem>]-results.tar.gz,
with directories and extensions stripped from both stems. A single job
running analysis.R still produces analysis-results.tar.gz, and
r_script = "R/analysis.R" now does too, but a multiple-mode job over
adelie.csv produces analysis-adelie-results.tar.gz where it previously
produced adelie.csv-results.tar.gz. This is a clean break: a job
submitted with 0.1.0 and collected with this version will have
htc_download() looking for names that do not exist on the submit node.
Download those results before upgrading, or pass files explicitly. See
the README section "A note on the results naming change" for the
rationale.
htc_gen_submit() gains an r_script argument, positioned after
executable. It is used to derive the default output_files name, which
has to match the tarball htc_gen_executable() tells the job to build,
and to check that the script is among the uploaded inputs (see below).
The function never reads the executable script or the Dockerfile, so the
script's name cannot be inferred and has to be supplied. In multiple
mode, omitting it warns. Code calling htc_gen_submit() with positional
arguments past executable will need updating.
Single-mode submit files now carry a transfer_output_files line derived
from r_script, where previously they carried only a placeholder comment
unless output_files was supplied. This is what makes htc_download()
bring back the results tarball in single mode rather than log files
alone.
The analysis script now travels to the execute node as an uploaded job
input rather than being baked into the container image.
htc_gen_executable() runs it by bare name (e.g. Rscript analysis.R)
rather than by an absolute, home_dir-prefixed path, because HTCondor's
file transfer does not preserve subdirectories: a script listed in
transfer_input_files always lands flat in the scratch directory. List
the script in input_files when calling htc_gen_submit(), and
htc_upload() sends it along with everything else; htc_gen_submit()
warns when r_script is known but its basename is missing from
input_files. data_files are unaffected: they remain baked
into the image under home_dir and are still read by absolute path.
The practical benefit is that editing the analysis script now requires
only re-uploading and resubmitting, not a container rebuild and registry
push. A Dockerfile built with
containr::generate_dockerfile(code_file = r_script) should drop that
argument and add the script to input_files instead; see the updated
vignettes.
The option controlling htc_config()'s progress messages is renamed from
htc_config_verbose to submitr.verbose, matching the namespacing of
submitr.config and submitr.check_server. It was undocumented, so this
is unlikely to affect anyone; both options are now documented under
?htc_config.
The resource presets file is now htc-resources.yml, following the
family-wide rule (toolero's CONVENTIONS.md, section 8) that every YAML
file a family package names for itself ends in .yml. The copy shipped in
inst/extdata/ is renamed. A project's own htc-resources.yaml, as
submitr 0.1.0 documented it, is still read for this release, with a
warning asking for it to be renamed; the old name will stop being read in
the next release. When both names are present, htc-resources.yml is
used and the old file is ignored, also with a warning.
htc_start() starts an HTC session by reading the project's htc.cfg
and storing it for the rest of the R session. htc_upload(),
htc_submit(), htc_status(), and htc_download() then use the stored
config automatically when config = NULL, so config = cfg no longer
has to be passed on every call. Each of them checks for an explicit
config first, falls back to the session, and errors with instructions
if neither is available. Call options(submitr.config = NULL) to clear
the session manually, or let it expire when R restarts.
htc_config() gains a project_config argument. Pass the path to a
_toolero.yml file (the project config toolero::init_project() writes
to a project's root) and its folders and conventions sections are
parsed once and folded into the returned list as config$project$folders
and config$project$conventions, kept separate from the connection
details config has always held and never written into htc.cfg.
containr::generate_dockerfile() reads the same file through its own
config argument.
htc_config() gains a check_server argument controlling whether it
opens an SSH connection to report the server's reachability. It
previously did so unconditionally, which meant that merely reading a
config file required a network, and that scripts, test suites, and
R CMD check all paid for a probe with no one to read it. The argument
defaults to the new submitr.check_server option, so it can be set once
for a session or a CI job rather than passed at every call, and
htc_start() accepts it through ....
htc_ssh_setup() writes the ControlMaster block described in
htc_config()'s setup guidance to ~/.ssh/config, and creates the
connections directory it references, without leaving R. It leaves the
file untouched if a matching Host block already exists, and supports
dry_run = TRUE to preview the change first.
submitr now keeps a submission state file, htc-manifest.yml, in the
project root beside htc.cfg. htc_gen_submit(), htc_gen_executable(),
htc_upload(), and htc_submit() record what they know as they run (mode,
submit and executable file names, input and output files, subset names,
remote path, cluster ID), and every later step reads it back for its
defaults, so values typed once flow through the rest of the workflow. The
file lives on disk rather than in a session option, so it survives an R
restart: a long job can be submitted in one session and collected in
another. Every function that reads or writes it takes a path argument
naming the directory that holds it, defaulting to ".". The generators'
path is independent of their output argument, so generated job files
can be written to a subfolder while the submission state stays in the
project root; to keep it elsewhere, pass the same path to every call.
Development versions before 0.2.0 wrote the file as htc-manifest.yaml.
It is now htc-manifest.yml, following the family's .yml rule, with no
fallback to the old name since no release ever wrote it: in a project set
up with a development version, rename the file or regenerate the job
files.
htc_gen_submit() and htc_gen_executable() gain a config argument (a
named list as returned by htc_config()). When config was built with
project_config, htc_gen_submit()'s queue_from defaults to
file.path(config$project$conventions$split_dir, "manifest.csv") and
htc_gen_executable()'s results_folder defaults to
config$project$conventions$output_dir. Both remain fully optional:
everything can still be passed explicitly.
htc_gen_submit()'s executable argument and htc_gen_executable()'s
output_file argument default to the executable name the other generator
already recorded in the submission state, so the name only has to be
typed once, whichever generator runs first. If an explicit value
disagrees with the recorded one, both functions warn rather than silently
preferring one over the other; the explicit value is still used.
htc_check() is a preflight check that catches, locally and in seconds,
problems that would otherwise surface an hour later as a held or failed
job on the cluster: a missing input or data file, a "multiple"-mode job
whose subset files no longer match subdatasets.csv, an implausible
resource request, and a container_image tagged latest (or carrying no
tag at all). When podman or docker is available locally, it also
makes a best-effort attempt to confirm the image is pullable;
check_image = FALSE skips that probe, which contacts the registry.
Every other argument resolves from the submission state, so the common
case is htc_check() with no arguments, run after the generators and
before htc_upload(). Returns a tibble of issues (possibly zero rows), each
tagged "error" or "warning".
htc_upload() gains a files = NULL default. When files is omitted,
the upload list is resolved from the submission state: the submit file,
the executable script, any shared input files, and, in "multiple" mode,
subdatasets.csv together with the individual subset files.
remote_path also defaults to NULL, resolving from an explicit
argument, then the value already recorded, then "~/", and is recorded
after a successful (non-dry_run) upload.
htc_upload() gains a check argument (default FALSE). When TRUE, it
runs htc_check() before uploading and aborts if any "error"-level
issue is found; "warning"-level issues are reported but do not block
the upload.
htc_submit() reads the submission state for its own defaults.
submit_file and remote_path both default to NULL and resolve, in
order, from an explicit argument, the recorded value, and finally the old
hardcoded literal, so uploading to a non-default remote_path no longer
breaks the submit step. The cluster ID it prints is recorded for the
functions that follow.
htc_status() gains a path argument and resolves cluster_id from the
submission state when omitted, so the cluster ID htc_submit() just
printed no longer has to be retyped.
htc_status() gains a show_hold_reason argument (default TRUE). When
any held jobs are present, it automatically runs a follow-up
condor_q -hold query and prints the hold reason, so diagnosing a held
job no longer means leaving R. Set show_hold_reason = FALSE to skip the
extra query.
htc_cancel() and htc_release() cancel a submitted cluster with
condor_rm, or release jobs HTCondor has held back into the queue with
condor_release. Both resolve cluster_id from the submission state
when not supplied but, unlike htc_status(), refuse to proceed when no
cluster_id can be resolved rather than acting on every job in the
queue: canceling or releasing everything is a much more consequential
default than merely showing everything. htc_cancel() also accepts a
reason recorded against the removed jobs. Both support dry_run and
verbose.
htc_download() gains a cluster_id argument and a files = NULL
default. When files is omitted, it builds the download list from what
htc_gen_submit(), htc_gen_executable(), and htc_submit() recorded,
for both single and multiple mode. remote_path defaults to NULL,
resolving from an explicit argument, then the value recorded in the
submission state, then "~/"; previously it assumed "~/" even when the
job had been submitted from somewhere else.
htc_collect() unpacks the results tarballs htc_download() brought back,
one subfolder per job, and returns the job index: a tibble with one row
per job giving its group, HTCondor process and cluster numbers, whether
its tarball was extracted, how many files its results folder holds and
what they are (a list column of paths relative to output_dir), whether
it includes an output record, the paths to its .log, .err, and .out
files, and the container image it ran in. It resolves the tarballs from
the submission state, or accepts an explicit named tarballs vector. It
works for any analysis, whatever the script wrote, and never opens the
files themselves: the output record (project-manifest.json) is only
checked for, not read, since interpreting it is toolero's job. A job
whose tarball is missing or will not extract still gets a row, with
extracted = FALSE, and one warning covers all such jobs, so a failed job
no longer stops the collection. An existing extraction folder is an error
raised before anything is extracted, unless overwrite = TRUE.
htc_upload() and htc_download() print a success confirmation after
every successful transfer, rather than only when verbose = TRUE.
Remote commands in htc_submit() and htc_status() are assembled with a
single POSIX quoting idiom rather than shQuote(). This does not fix a
known failure: shQuote() does escape a dollar sign when it switches to
double quotes, so the previous arrangement held. But it relied on a
choice shQuote() makes from its own input, and on a dialect that follows
the local platform, neither of which suits a command bound for the POSIX
shell at the far end of an SSH connection and quoted twice on the way.
Quoting is now the same whatever the input and whatever the local
platform.
htc_gen_submit() lists input files by basename in
transfer_input_files. htc_upload() sends every file flat into one
directory on the submit node, so input_files = "R/analysis.R" put a
path in the submit file that did not exist there, and the job failed
before it started. The submission state still records the paths as
given, which is how htc_upload() finds the files on this machine. Two
input files that share a basename (R/utils.R and scripts/utils.R)
would overwrite each other on the submit node, so they are now an error,
raised before anything is written.
The script htc_gen_executable() writes now packs the results folder
even when the R script fails, and then exits with R's own exit status.
Previously set -euo pipefail stopped the script at the failing
Rscript line, so no tarball was built: whatever the analysis had
written was lost, and HTCondor held the job for a missing
transfer_output_files entry rather than letting it finish. A failed job
now returns its partial results and is reported by HTCondor as failed,
not held.
htc_gen_executable() always writes #!/bin/bash as the first line of
the generated script. With comments = TRUE, the shebang section's
explanatory comment was written ahead of it, putting #!/bin/bash on
line two, where the kernel does not look for it, so the script ran under
whatever shell happened to invoke it. The shebang now carries no comment
of its own, and the comment sits with set -euo pipefail, which is what
it was describing all along.
htc_gen_executable() includes set -euo pipefail after the shebang, so
the script exits on the first error instead of silently continuing.
htc_gen_executable() changes to HTCondor's scratch directory
(cd "${_CONDOR_SCRATCH_DIR:-$PWD}") before any file operations, and
reads baked-in data files by absolute path under home_dir (default
/home), where containr::generate_dockerfile() put them. Outputs are
therefore written where HTCondor looks for transfer_output_files, while
data is read from where the image actually holds it.
htc_gen_submit() prepends docker:// to container_image when the
prefix is missing. Previously, omitting it caused HTCondor to treat the
image as a local file.
htc_gen_submit() includes should_transfer_files = YES and
when_to_transfer_output = ON_EXIT in the transfer section. HTCondor
requires both for its file transfer mechanism to work.
htc_status(watch = TRUE) decides when to stop polling by reading the job
count from condor_q's own Total for query: line, falling back to
matching the cluster ID against the JOB_IDS column. It previously
searched the whole report for the cluster ID as a substring, which could
match the address or port in the schedd header, or a count in the
Total for all users: line. Because that test drives the loop's exit, a
false match did not give a wrong answer; it left the loop polling
forever.
htc_submit() quotes remote_path and submit_file for the remote shell
rather than typing quote characters around the assembled command. A
filename or path containing a space or an apostrophe previously truncated
the command at that character, producing a shell syntax error or, in the
worst case, running the remainder as separate commands. A leading ~ is
held outside the quoting so the remote shell still expands it.
htc_submit() prints condor_submit output with cat() rather than
passing it to cli, which treats braces as inline markup and would try
to evaluate anything an HTCondor message happened to wrap in them. Error
details from htc_submit() and htc_status() are escaped for the same
reason. This matches how htc_status() already printed condor_q
output.
CONVENTIONS.md). htc-manifest.yml
is the submission state and manifest.csv from
toolero::write_by_group() is the job manifest. Earlier development
versions called the submission state the "job manifest", which collided
with the toolero file of the same name. Messages across the package now
say "submission state" wherever they mean htc-manifest.yml and "job
manifest" only when they mean manifest.csv. Function names and
arguments are unchanged.tests/testthat/test-readme-workflow.R, a documentation-regression
suite rather than a code-correctness one. It reproduces the documented
code blocks in README.md (the first workflow, scaling to many jobs)
with their literal argument values in a temporary directory, and checks
the result against claims made elsewhere in the README: the submission
state's example YAML, the resource preset table, the results-naming
table, and the quick function reference. It exists because the README
has already gone stale once, after the output folder and the tarball
naming convention changed.submitr is the third package in the From the Notebook to the Cluster
family, alongside toolero and containr. It provides a workflow for
submitting containerized R analyses to the UW-Madison Center for High
Throughput Computing (CHTC) from inside R.
htc_config() -- create or read a project-level htc.cfg configuration
file. On first use, prompts interactively for username and server,
displays ControlMaster SSH setup guidance to reduce Duo MFA prompts,
writes htc.cfg, and adds it to .gitignore. Subsequent calls read the
existing file and validate server reachability. Returns a named list with
username and server. Errors informatively when username or server
are supplied as empty strings.htc_gen_submit() -- generate an HTCondor .sub submit file from
project parameters. Supports single-job and multiple-job modes. Multiple
mode reads the job manifest from toolero::write_by_group(manifest = TRUE),
extracts filenames, writes subdatasets.csv, and emits
queue file from subdatasets.csv. Resource presets (small, medium,
large, custom) are loaded at runtime from
inst/extdata/htc-resources.yaml; a local ./htc-resources.yaml takes
precedence over the package default. GPU support via gpu = TRUE and
gpu_options. comments = TRUE annotates each section of the generated
file with explanatory text.
htc_gen_executable() -- generate the .sh executable script that
HTCondor runs inside the container. Produces a four-element script:
shebang, mkdir, Rscript, and tar. In multiple-job mode, passes
${1} as a positional argument to the R script. r_script must be
supplied explicitly -- there is no default. set_executable = TRUE
(default) sets executable permissions via Sys.chmod().
htc_upload() -- copy files to the CHTC submit node via scp. Accepts
single files, vectors of files, directories (transferred recursively),
and glob patterns. remote_path defaults to "~/". dry_run = TRUE
previews the command without executing it.
htc_submit() -- run condor_submit on the remote submit node via SSH
from the remote directory where files were uploaded. Returns the cluster
ID invisibly for use with htc_status(). Supports dry_run = TRUE.
htc_status() -- check job progress via condor_q. Optionally filters
by cluster ID. watch = TRUE polls at interval seconds (default 60)
until the cluster ID leaves the queue. Returns condor_q output
invisibly as a character vector. Supports dry_run = TRUE.
htc_download() -- copy result files back from the submit node via
scp. Supports single filenames, vectors of filenames, and glob patterns
("*.tar.gz", "job.*"). Glob patterns are single-quoted to prevent
local shell expansion. local_path defaults to ".". Supports
dry_run = TRUE.
inst/extdata/htc-resources.yaml ships with the package and provides
default resource presets for htc_gen_submit().
inst/extdata/hello-world.sub and inst/extdata/hello-world.sh are
included as test files for end-to-end workflow verification.
inst/extdata/sample.R is included as a sample R script for use in
examples.
dry_run = TRUE and mocked bindings. Layer 3 integration tests are
opt-in via Sys.setenv(CHTC_USERNAME = "your.netid") and never run on
CRAN or CI. 153 tests passing across seven test files.