toolero toolero package logo

The problem with starting from scratch

Every research coding project begins with a blank slate and a set of early decisions: where to put the data, how to name the scripts, whether to track dependencies, and whether to use version control. These decisions feel low-stakes in the moment. They rarely are. The cost usually appears later, when the project needs to be shared, reviewed, rerun, containerized, or moved to a larger computing system.

A project that starts with a flat folder, no dependency tracking, and scripts that mix data loading, cleaning, modeling, and reporting is not impossible to rescue later, but it is genuinely hard. Collaborators cannot reproduce results because the package versions are unknown. The analysis breaks when moved to a different machine. The manuscript references outputs that no longer exist in the file system.

These are not exotic failure modes. They are the ordinary cost of skipping setup decisions that feel optional at the start of a project. The From the Notebook to the Cluster package family exists to make those decisions easier to get right the first time. toolero is the first step in that family.

toolero is a small, opinionated set of tools designed to make good research workflow decisions easier to adopt. It does not impose a rigid framework. It provides practical defaults for common research projects and gets out of the way when you need to customize.

If you are new to research computing, toolero gives you a solid starting point without requiring you to know in advance why each piece matters. If you are experienced, it automates the setup work you would otherwise do by hand at the start of every project.

When to use toolero

Use toolero when you are:

toolero is useful on its own. You do not need to containerize your project or submit work to a cluster to benefit from better project structure, cleaner inputs, literate analysis documents, and repeatable workflows. That said, starting with toolero means your project is already prepared for the next step when the time comes.

From the Notebook to the Cluster

toolero is the first package in From the Notebook to the Cluster, a three-package family for reproducible research workflows. The family covers the full arc from local project setup to high-throughput computing:

toolero     organize, scaffold, split, apply, record
  └─ containr   freeze the software environment in a container
       └─ submitr    send the analysis to CHTC and retrieve results

The organizing idea behind the family is that good practices at each stage make the next stage easier. A project structured with toolero, with dependency tracking, a clean folder layout, and data split into independent pieces, is already most of the way to being containerizable. A containerized project is already most of the way to being submittable to a high-throughput computing cluster. The family does not require you to commit to the full arc upfront. Each package is useful on its own, and you can adopt them one at a time as your project's needs grow.

toolero does not require containr, and containr does not require submitr. The dependencies run in one direction only: each package prepares cleanly for the next, but none reaches backward.

What the later packages read, rather than guess, is recorded in the project config: init_project() writes _toolero.yml at the project root, describing the folder set it resolved and the naming conventions in force. Commit that file. It describes the project, not the machine it was created on, and it is how a customized project stays legible to check_project(), containr, and submitr without being handed the same configuration again.

Installation

Install from CRAN:

install.packages("toolero")

Install the development version from GitHub:

# install.packages("pak")
# pak::pak("erwinlares/toolero")
install.packages("toolero", repos = c('https://erwinlares.r-universe.dev', 'https://cloud.r-project.org'))

A first workflow

The functions below cover a common path from project creation to recorded outputs. It runs end to end against the sample data create_qmd() copies in, and uses a temporary directory so you can try it without writing to your Documents folder. The only extra requirement is dplyr, for the analysis function in step 8.

library(toolero)

# tempfile() rather than a fixed name under tempdir(): it hands back a
# path that does not exist yet, so running this block twice in one
# session scaffolds a second project instead of colliding with the first.
project_dir <- tempfile("my-analysis-")

# 1. Create a project with sensible defaults.
#    renv and git are on by default and are what you want in a real
#    project. They are off here so the example runs quickly and without
#    prompting.
init_project(path = project_dir, use_renv = FALSE, use_git = FALSE)

# 2. Audit the project structure
check_project(path = project_dir)

# 3. Scaffold a reproducible Quarto analysis document. This also copies
#    the bundled sample data into data-raw/.
create_qmd(path = project_dir, filename = "analysis.qmd")

# 4. Extract the R code from the document into a standalone script.
#    R/ is where toolero expects derived scripts to live.
qmd_to_r(
  input  = file.path(project_dir, "analysis.qmd"),
  output = file.path(project_dir, "R", "analysis.R")
)

# 5. Read and clean the sample data
penguins <- read_clean_csv(
  file.path(project_dir, "data-raw", "sample.csv"),
  na      = c("", "NA", "N/A", "."),
  summary = TRUE
)

# 6. Write the cleaned data
write_clean_csv(penguins, file.path(project_dir, "data", "clean.csv"))

# 7. Split the data into per-group subsets, one file per species
write_by_group(
  penguins,
  group_col  = "species",
  output_dir = file.path(project_dir, "data", "jobs"),
  manifest   = TRUE
)

# 8. Define an analysis function and apply it to each subset
summarise_species <- function(data) {
  dplyr::summarise(
    data,
    n            = dplyr::n(),
    mean_mass    = mean(body_mass_g, na.rm = TRUE),
    mean_flipper = mean(flipper_length_mm, na.rm = TRUE)
  )
}

results <- run_by_group(
  manifest = file.path(project_dir, "data", "jobs", "manifest.csv"),
  .f       = summarise_species
)

# 9. Save outputs and record each write in the project accumulator.
#    output_dir is where the accumulator goes. It defaults to output/
#    under the project root, and the working directory here is not inside
#    project_dir, so it is given explicitly.
save_output(
  results,
  file.path(project_dir, "output", "results.rds"),
  .f         = saveRDS,
  output_dir = file.path(project_dir, "output")
)

# 10. Write the output record, summarizing what was produced
generate_manifest(output_dir = file.path(project_dir, "output"))

# 11. Read it back as a table, one row per artifact
read_output_records(file.path(project_dir, "output"))

In a real project, replace project_dir with the path where you want the project to live. The important idea is that toolero helps you start with a structure that can grow: local analysis first, reproducible execution later, and scalable computing when needed.

Quick reference

Function What it does
init_project() Creates a new R project with a standard research-oriented folder structure and records that structure in _toolero.yml at the project root. Can set up renv, initialize git, customize folders via custom_folders, load a config file, optionally copy branding assets into assets/ via the branding argument (TRUE for generic placeholders, "uw-madison" for RCI branding), create a README via use_readme (README.md by default, "plain" for README.txt, or FALSE to skip it), and, via use_rprofile, keep loading your own ~/.Rprofile customizations in a project that renv would otherwise shadow them in.
generate_project_config() Writes a skeleton YAML project configuration file pre-filled with the standard toolero folder structure and conventions. Edit to define a custom layout and pass to init_project() via config. Same schema, template and writer as the _toolero.yml a project carries.
check_project() Audits an existing project for common reproducibility scaffolding: the expected folders, an .Rproj file, renv.lock and whether it actually records anything, .renvignore entries that would hide your source from renv, git, README, .gitignore, the project config (_toolero.yml), stale purled .R scripts, and hidden files such as .RData or .Rhistory. Audits against the project's own _toolero.yml when it has one.
create_qmd() Scaffolds a Quarto document. Can create a full worked example or a blank skeleton, pre-populate YAML metadata via header_defaults, wire in custom styling from a standardized assets/ folder, and (opt-in via use_purl, FALSE by default) stamp purl: true/false into the document's header and set up a post-render purl hook, merged into an existing _quarto.yml where possible and skipped with a warning for website, book and manuscript projects.
generate_profile() Writes a YAML skeleton of author information and document formatting preferences, meant to be edited once and reused across projects via create_qmd(header_defaults = ).
qmd_to_r() Extracts R code chunks from a Quarto document into a standalone .R script, creating the output directory if it does not exist. Useful when the .qmd is the source of truth but a script is needed for batch execution or sharing.
read_clean_csv() Reads a CSV file, cleans column names, handles missing values, optionally drops incomplete rows, and can print a short ingest summary.
write_clean_csv() Writes a data frame to CSV with clean column names and command-line feedback. Reinforces the pattern of keeping raw inputs in data-raw/ and analysis-ready outputs in data/.
write_by_group() Splits a data frame by one or more grouping columns and writes one CSV per group, optionally prefixed via prefix. Can also create a job manifest for parallel or high-throughput workflows. output_dir can be resolved from a project's _toolero.yml via config.
run_by_group() Applies a function to each group subset and collects the results. Accepts a job manifest from write_by_group() or a named list of data frames. Supports parallel execution and returns a flat tibble or a nested tibble depending on what the function returns.
save_output() Writes an object to disk via a user-supplied function and appends a row to the project accumulator recording the path, class, function used, and whether the write succeeded. output_dir can be resolved from a project's _toolero.yml via config.
generate_manifest() Reads the project accumulator, deduplicates by path, and writes the output record, project-manifest.json, describing every artifact the analysis produced, along with the git commit checked out at the time (when available).
read_output_records() Reads one or more output records back as a tibble, one row per artifact, labeled by folder. Falls back to the accumulator, with a warning, when a folder has no usable record. Experimental.
detect_execution_context() Returns "interactive", "quarto", or "rscript" so one codebase can adapt to local exploration, document rendering, or batch execution.
resolve_input_path() Resolves where the input data lives for the current execution context, and says what to fix when it cannot. The companion to detect_execution_context() for the specific case of finding your data.
generate_kb_xml() Converts a rendered Quarto HTML document into UW-Madison Knowledge Base importable XML with embedded resources and metadata derived from the source document.
generate_citation() Writes a CITATION.cff skeleton, optionally pulling author information from a profile written by generate_profile().
generate_license() Writes a plain-text LICENSE file from one of three common templates ("MIT", "CC0", "GPL-3"), with the copyright holder and year filled in. "GPL-3" links to the canonical full text rather than reproducing it.
generate_data_doc() Writes a Markdown documentation stub for a single dataset -- source, date obtained, license and usage terms, collection method, a variables table, and known issues -- with the dataset's file name and today's date pre-filled.
arborize() Renders syntactic trees as PNG images using Quarto's Typst engine. Can also write a provenance YAML file so the tree image can be reproduced or modified later.

Paths start at the project root

Most of the confusion about file paths in R comes from the working directory, which depends on how the code is run. A chunk run by hand in RStudio, a document rendered by Quarto from reports/, and a script dispatched to a cluster each start somewhere different, so a path such as "output/results.csv" can name three different files. toolero removes that ambiguity with one rule: a relative path given to any toolero function is read from the project root, not from the working directory.

In practice, save_output(fit, "output/fit.rds", saveRDS) writes into the project's own output/ folder whether you call it from the project root, from a document under reports/, or from a job on an HTCondor execute node. You do not need to wrap the path in here::here(), although doing so still works, because here::here() returns an absolute path and absolute paths are used as given. So are paths starting with ~ and URLs.

The root is found afresh on every call, in this order. First comes the job's scratch directory, when HTCondor names one in _CONDOR_SCRATCH_DIR (HTCondor sets that variable itself, inside every job). Next comes the nearest folder above the working directory that carries a project marker: the .here file init_project() writes, an .Rproj file, a package DESCRIPTION, or a version control folder such as .git. If neither applies, the working directory is the root.

Reading is forgiving. A relative path that does not exist at the root but does exist relative to the working directory is read from there, and a message names both locations, so documents written before this rule keep working. Writing never guesses, since a file written in the wrong place is harder to notice than one that cannot be found.

A few things are left alone on purpose. init_project() reads its path, config and use_rprofile from the working directory, because the project does not exist yet. Under Rscript, resolve_input_path() takes the command line argument as given, so run Rscript from the project root or pass an absolute path. Arguments passed on through ... reach the other function untouched. There are also two limits to keep in mind. A project nested inside another project is not supported, since the nearest marker wins, and on an execute node ~ and $HOME cannot be trusted, so relative paths are the safer choice in job scripts. The same rule is written out once in ?"project-paths", and every function that takes a path shows it in a "Paths" section of its help page.

Core workflow functions

init_project()

Creates a new R project with a standard folder structure suited for research workflows. Optionally sets up renv for dependency management and git for version control, both on by default, because both matter.

The default structure follows conventions established by The Carpentries and UW-Madison Libraries workshops: data-raw/, data/, R/, scripts/, output/figures/, output/tables/, and reports/. R/ holds the .R script derived from your .qmd source, whether that derivation comes from qmd_to_r() or from the post-render hook create_qmd(use_purl = TRUE) scaffolds. scripts/ holds hand-written scripts you maintain yourself. The distinction matters because downstream packages resolve the derived script by convention.

The custom_folders argument lets you add folders or suppress defaults without changing the standard set for other projects. A "-" prefix removes a folder from the set that will be created; bare names add new ones.

# Standard project
init_project(path = "~/Documents/my-project")

# Add a folder and suppress one from the standard set
init_project(
  path           = "~/Documents/my-project",
  custom_folders = c("models", "-output/figures")
)

# Drive the folder structure entirely from a config file
init_project(
  path   = "~/Documents/my-project",
  config = "~/linguistics-project.yml"
)

One folder cannot be suppressed, by a config or by custom_folders: R/. usethis::create_project() creates it unconditionally, so it is present in every project init_project() makes. A structure that leaves it out is honored everywhere else, so R/ is absent from _toolero.yml, gets no placeholder file, and is not audited by check_project(). Only the directory itself is unavoidable.

For projects where the standard structure does not fit, generate_project_config() writes a skeleton YAML config pre-filled with the standard folders and conventions. Edit the file to define your own layout and store it in your home directory so it is easy to reuse across projects.

# Write a config skeleton to your home directory
generate_project_config("linguistics-project.yml", path = "~")

Every project init_project() creates carries a _toolero.yml at its root recording the folder set it resolved and the naming conventions in force. It records the resolved structure, never the inputs that produced it, so a project built from a config, one built with custom_folders, and one built from the defaults all produce the same shape of file and nobody has to replay anything to learn what the project looks like. Commit it.

init_project() also writes an empty .here file at the project root. Commit that too. It marks the project root for toolero's own functions, which read a relative path from there (see "Paths start at the project root" above), and for here::here() in any code of yours that still calls it. An .Rproj file or a .git folder would do the same job, but a project created outside RStudio and without git has neither.

Neither file is there only for the benefit of containr or submitr, and a project that never leaves your laptop still needs both. toolero itself uses them: check_project() audits against _toolero.yml, save_output() and its relatives read it through config, and every path built from the project root depends on .here. They also serve a longer game. Together with renv.lock and the output record generate_manifest() writes, they describe what a project is, what it needs, and what it produced. That description is what the family's planned fourth package, encapsulr, is being designed to assemble into a self-describing research object: the analysis, its data, its software environment, and the provenance that ties them together, packaged to be archived with a paper, handed to a reviewer, or deposited in a repository. None of that requires a cluster, and a project started with init_project() will be ready for it without being rebuilt.

Every folder init_project() creates that is still empty when the call finishes also receives a zero-byte .gitkeep. git tracks files rather than directories, so without this a scaffolded structure survives nothing: the opening commit carries the files at the project root and none of the layout, and a collaborator cloning the repository gets a project with no folders in it.

# Generic placeholder branding
init_project(path = "~/Documents/my-project", branding = TRUE)

# UW-Madison RCI branding
init_project(path = "~/Documents/my-project", branding = "uw-madison")

The branding argument controls whether an assets/ folder is created and populated. branding = TRUE copies in generic placeholder files (logo.png, favicon.png, header.html, footer.html, styles.css). branding = "uw-madison" copies UW-Madison RCI branding under the same standardized names. Both modes produce identically named files, so create_qmd(use_style = TRUE) works the same way regardless of which branding mode was used. branding = "none" (the default) creates no assets/ folder.

The use_readme argument controls whether a README file is created at the project root. use_readme = TRUE (the default) creates README.md from toolero's generalist README template, a short guide covering what a README is, general best practices, and a recommended section structure for documenting software, data, or both, with pointers to the Cornell Data Services guides for further detail. use_readme = "plain" creates README.txt with identical content; only the extension differs. use_readme = FALSE skips the README entirely. If a README already exists at the destination, in any capitalization and with any extension, init_project() stops with an informative error rather than overwriting it.

# Skip the README
init_project(path = "~/Documents/my-project", use_readme = FALSE)

# Plain-text README instead of Markdown, same content, different extension
init_project(path = "~/Documents/my-project", use_readme = "plain")

When use_renv = TRUE, init_project() calls renv::scaffold() rather than renv::init(). init() loads the newly created project into the calling R session, repointing .libPaths() at a library that is empty apart from renv itself, so every package you had available disappears until you restart R. scaffold() builds the same infrastructure and leaves your session alone.

Nothing is snapshotted at creation time, because a project that has just been created has no code in it and there is nothing to discover. Take a snapshot yourself once the analysis exists, and before containerizing:

renv::snapshot()

The renv lockfile is what containr::generate_dockerfile() reads to containerize the project later. Starting with init_project() means that step is already prepared, even if you never need it. If the project uses branding assets and is later containerized, pass misc_file = "assets/" to containr::generate_dockerfile() so the styling files are copied into the image alongside the .qmd.

Setting up renv has a side effect worth knowing about: R reads exactly one .Rprofile per session, and renv::scaffold()'s own .Rprofile (the one that activates the project) becomes that one, so your personal ~/.Rprofile -- aliases, options, helper functions -- silently stops loading the moment a project is under renv. use_rprofile = TRUE appends a guarded block to the project's .Rprofile, after renv's own activation line, that sources ~/.Rprofile if it exists:

init_project(path = "~/Documents/my-project", use_rprofile = TRUE)

The check happens every time a session starts, not just once at creation, so a ~/.Rprofile you write or edit later is still picked up. It defaults to FALSE because it works against renv's own point: a project that automatically re-sources your personal environment is no longer fully isolated from it. The block is also written generically -- it sources whichever file is at ~/.Rprofile for whoever opens the project, not one person's file baked in -- so a collaborator who clones the project gets the same behavior, for their own home directory, that the project's author chose.

check_project()

Audits an existing project directory and reports whether it follows toolero conventions. Useful both for projects initialized with init_project() and for any existing R project you want to evaluate.

The report checks for the expected folder structure, an .Rproj file, renv.lock, a git repository, a README, and a .gitignore. It also notes the presence of hidden files like .RData and .Rhistory that are common sources of reproducibility problems.

Two of the checks are about renv specifically, and they are paired for a reason. A lockfile that records nothing but renv itself is reported only when the project also has .R or .qmd source files: a freshly scaffolded project legitimately has an empty lockfile, but a project with code in it and nothing in its lockfile is the state that produces a container image which builds cleanly and then cannot run the analysis. Separately, a .renvignore that excludes .qmd files is reported, because in a project whose library() calls live in its Quarto source, which is the arrangement this package recommends, that entry hides the dependencies from the snapshot. Versions of init_project() before v0.5.0 wrote such a file, so projects created by those versions still carry one.

Which folders are audited depends on what the project declares. Precedence is an explicit config argument first, then the project's own _toolero.yml, then the built-in standard set. A folder declared by either of the first two and missing from disk is reported as a failure rather than a warning: a declaration that is not met is a conformance failure, whereas the standard set is a suggestion nobody signed up for.

Naming conventions are reported only when they differ from the defaults, so a conventions row in the output always means something in this project resolves differently from every other one.

check_project() also reports on stale purled scripts. Every .qmd whose header declares purl: true (see create_qmd()'s use_purl argument below) gets its own row, comparing it against the .R script R/purl.R is expected to have derived from it. Where that script is expected depends on the project type (flat R/<name>.R, or mirrored under R/ for a website or book). If the script is missing but one exists at the other layout's path, which is what a project scaffolded with an older R/purl.R looks like, the row says so and shows how to refresh R/purl.R. A script that is missing entirely, or older than the .qmd it was purled from, is a "warn": the .qmd is the source of truth, so an out-of-date R/ script means an edit was made and not yet re-rendered, and a container or cluster job that bakes in that script would run the old analysis without any error to say so. A document never opted into purl produces no row.

README detection is case-insensitive and extension-agnostic: any file whose stem matches readme, in any capitalization, counts regardless of extension or the absence of one. README.md, readme, Readme.pdf, and README.tex all pass.

# Audit the current project, against its own _toolero.yml if it has one
check_project()

# Audit against a specific config instead
check_project(config = "~/linguistics-project.yml")

# Access results programmatically
out <- check_project()

create_qmd()

Scaffolds a new Quarto document from a reproducible template with optional sample data, custom styling, YAML pre-population, and, opt-in via use_purl, a post-render hook that extracts R code from the rendered document into a companion .R file automatically.

The function has two main motivations. First, it reduces repetitive setup work. If you regularly create Quarto documents with the same author information, institutional metadata, or preferred format settings, the header_defaults argument lets you pre-populate the YAML header from a personal configuration file instead of rebuilding the same header by hand. generate_profile() is the natural way to create that file: it scaffolds a personal configuration template in your home directory, ready to fill in once and reuse across every document you create.

Second, it helps reduce code drift, when you opt in. In a literate programming workflow, the .qmd document can serve as the source of truth: prose, code, results, and interpretation live together. With use_purl = TRUE, the post-render hook derives the standalone .R script from the document automatically, so you do not have to maintain a separate script by hand. This pattern is discussed in more detail in the post From the Notebook to the Cluster. Part 1: Start with the Document.

Every edit create_qmd() makes to a document's YAML header is made line by line rather than by parsing the header and writing it back out. Keys the edit does not touch keep the template's own quoting, indentation, comments and ordering, so the document you open is the template as shipped plus the keys you asked for.

Arguments:

# Blank skeleton, no examples, no styling (use_purl = FALSE is the default)
create_qmd(path = "my-project", filename = "analysis.qmd",
           include_examples = FALSE)

# Full worked example with sample data and placeholder logo (default)
create_qmd(path = "my-project", filename = "analysis.qmd")

# Opt this document into purl: stamps purl: true, scaffolds R/purl.R, and
# wires up the _quarto.yml post-render hook (merged if the file already
# exists and is not a website, book or manuscript project)
create_qmd(path = "my-project", filename = "analysis.qmd", use_purl = TRUE)

# Blank document wired to branding assets in assets/
create_qmd(path = "my-project", filename = "report.qmd",
           include_examples = FALSE, use_style = TRUE)

# Blank document with custom branding from another directory
create_qmd(path = "my-project", filename = "report.qmd",
           include_examples = FALSE, use_style = "my-branding/")

# Pre-populate YAML from a personal config file
create_qmd(path = "my-project", filename = "analysis.qmd",
           header_defaults = "my-config.yml")

A document can live below the project root, the usual arrangement being reports/:

create_qmd(path = "my-project", filename = "reports/analysis.qmd",
           use_style = TRUE)

The folder is created if needed; the sample data, the logo, R/purl.R, and _quarto.yml still go to the project root. Code in the document keeps paths that start at the project root, data-raw/sample.csv in params: and output/results.csv for results, so it runs unchanged at the root, in reports/, and on a cluster. Header paths are the exception, because Quarto resolves them relative to the document: use_style writes css: ../assets/styles.css for a document in reports/, and adds the project root to the document's resource-path so the logo in header.html is found.

If use_purl = TRUE is used inside an existing website, book, or manuscript project, _quarto.yml is left untouched and a warning names what to add by hand: a post-render: R/purl.R entry under the project: key. The document's purl: true header stamp is written either way, and R/purl.R is scaffolded either way. Only the automatic _quarto.yml edit is skipped.

Both bundled templates set embed-resources: true. A non-self-contained HTML depends on the _files/ sidecar directory rendered beside it, and nothing that moves these documents around knows about sidecars: an archived cluster run brings back a folder, an emailed report is one file, and a rendered document committed next to its analysis quietly depends on a directory nobody thinks to copy. The cost of true is a larger file; the cost of false is an artifact that works only on the machine that made it.

generate_profile()

Scaffolds a personal configuration template -- author name, affiliation, ORCID, email, and a handful of common document format settings -- meant to be filled in once and reused as the header_defaults argument to create_qmd() or the profile argument to generate_citation(), rather than retyping the same author block into every new document or citation file.

generate_profile(filename = "toolero-profile.yml")

Arguments:

The written file has two sections: a personal information block (author:, with name:, affiliation:, orcid:, and email: sub-keys) and a document settings block (format: and related keys) for the format defaults you find yourself setting the same way across documents. Neither a phone number nor a mailing address placeholder is included -- those are not things create_qmd() or generate_citation() make use of, so the template does not ask for them.

Nothing about the filename or location is enforced beyond what you pass in: you can keep more than one profile (a personal.yml and a work.yml, say) and point different projects at whichever one applies.

qmd_to_r()

Extracts R code chunks from any .qmd file into a standalone .R script. This is the direct counterpart to the purl hook in create_qmd(), and it works on any Quarto document regardless of how it was created.

The output path defaults to the same directory as the input with the .qmd extension replaced by .R. Its parent directory is created if it does not already exist, so writing into R/ works whether or not the project was scaffolded by init_project(). The documentation argument controls how much context is preserved in the extracted script: chunk labels only (1, the default), full roxygen blocks (2), or pure code with no comments (0).

# Default output: same directory, .R extension
qmd_to_r(input = "analysis.qmd")

# Explicit output path. R/ is the documented home for derived scripts.
qmd_to_r(
  input  = "analysis.qmd",
  output = "R/analysis.R"
)

qmd_to_r() needs the knitr package, which is a suggested rather than a required dependency. Install it if you have not already.

read_clean_csv()

Reads a CSV file into a tibble and cleans the column names in one step. Column names become lowercase, spaces become underscores, and special characters are removed. Beyond name cleaning, the function supports explicit missing-value handling, selective row dropping, and an optional ingest summary that surfaces common data problems immediately.

# Basic usage
data <- read_clean_csv("data-raw/input.csv")

# Explicit missing-value codes and ingest summary
data <- read_clean_csv(
  "data-raw/input.csv",
  na      = c("", "NA", "N/A", ".", "-999", "unknown"),
  summary = TRUE
)

# Drop rows missing in specific columns
data <- read_clean_csv(
  "data-raw/input.csv",
  drop_na = c("participant_id", "response_score")
)

write_clean_csv()

Writes a cleaned data frame to a CSV file with cli feedback. The natural counterpart to read_clean_csv(), reinforcing the convention that data-raw/ holds original inputs and data/ holds analysis-ready outputs.

If the data frame's column names are not already clean, write_clean_csv() applies janitor::clean_names() before writing and warns you about the affected columns, so the output file always has consistent names regardless of what was passed in.

data <- read_clean_csv("data-raw/input.csv")

write_clean_csv(data, "data/clean.csv")

# Overwrite an existing file
write_clean_csv(data, "data/clean.csv", overwrite = TRUE)

write_by_group() and run_by_group()

These two functions form the split-apply pair at the heart of toolero's workflow support. The idea is simple: split the data once, then apply an analysis function to each piece and collect the results. The split and the apply are deliberately separate steps so you can iterate on the analysis function without re-splitting the data each time.

write_by_group() handles the split. It partitions a data frame by one or more grouping columns, writes one CSV per group with sanitized filenames, and optionally produces a manifest.csv recording each group's value, file path, and row count. That job manifest is the input to run_by_group(). Rows with a missing value in any grouping column are dropped by default (drop_na = TRUE), with a message reporting how many were dropped; set drop_na = FALSE to instead treat missing values as their own group.

output_dir can be supplied explicitly, or resolved from a project's own _toolero.yml by passing its path as config: write_by_group() reads the config's split_dir convention (the folder init_project() set aside for split-by-group output) and uses that when output_dir is not supplied directly. An explicit output_dir always wins over config; config only fills in what you did not already say. This is opt-in -- a project never scaffolded by init_project(), or a call that supplies output_dir directly, behaves exactly as it always has.

Groups are written, and job manifest rows recorded, in order of first appearance in the data rather than in sort order. This is more than cosmetic: submitr writes its subdatasets.csv in job manifest order, HTCondor assigns ProcId in that order, and log filenames are reconstructed from position, so job manifest row order is the mapping from a job number back to a group.

run_by_group() handles the apply. It reads each subset from the job manifest, calls your function on each one, and assembles the results into a single tibble. If your function returns a data frame, the output is automatically unnested into a flat tibble with a group ID column prepended. If it returns anything else, a model, a plot, a file path, the results come back as a nested tibble with a list-column.

sample_path <- system.file("templates", "sample.csv", package = "toolero")
penguins    <- read_clean_csv(sample_path)

# Split to disk
write_by_group(
  penguins,
  group_col  = "species",
  output_dir = "data/jobs",
  manifest   = TRUE
)

# Define an analysis function
summarise_species <- function(data) {
  dplyr::summarise(data,
    n            = dplyr::n(),
    mean_mass    = mean(body_mass_g, na.rm = TRUE),
    mean_flipper = mean(flipper_length_mm, na.rm = TRUE)
  )
}

# Apply from disk via the job manifest, returning a flat tibble
results <- run_by_group(
  manifest = "data/jobs/manifest.csv",
  .f       = summarise_species
)

# Apply from memory via named list, same result, no disk reads
subsets <- split(penguins, penguins$species)

results <- run_by_group(
  groups = subsets,
  .f     = summarise_species
)

group_col also accepts more than one column name. Grouping by c("species", "sex") writes one file per combination that actually appears in the data, adelie--female.csv, adelie--male.csv, and so on, rather than the full cross-product of possible values. The -- separator is load bearing: because sanitizing collapses any run of non-alphanumeric characters to exactly one dash, a sanitized value can contain a single - but never two consecutive ones, which is what leaves -- free to mark a column boundary unambiguously.

The job manifest has one schema regardless of how many grouping columns were supplied: one column per grouping variable holding the raw value, then group_value (the raw values joined with " | "), n_rows, and file_path. Grouping on a single column produces the same shape, with the grouping column's value repeating group_value exactly. That redundancy is deliberate, since one schema with a varying column count is easier to read, validate and rely on than two schemas selected by how many columns you happened to group on.

write_by_group(
  penguins,
  group_col  = c("species", "sex"),
  output_dir = "data/jobs",
  manifest   = TRUE
)

The prefix argument prepends a namespace to every output filename, so a.csv becomes data-a.csv and a--female.csv becomes data-a--female.csv. It is worth using before a high-throughput run: submitr::htc_gen_submit() reduces the job manifest's paths to basename(), so subsets from two datasets split on the same short column would otherwise land in one flat directory on the access point and overwrite each other.

write_by_group(
  penguins,
  group_col  = "species",
  output_dir = "data/jobs",
  prefix     = "penguins",
  manifest   = TRUE
)

Anything else you pass to run_by_group() is forwarded to your analysis function on every call. Ordinary values -- a number, a string, a logical, a file path -- need nothing special. The one case that does is a bare column name, such as x = flipper_length_mm, since a symbol like that has no meaning until it meets the data. A function accepting one has to capture it with {{ }} rather than evaluate it, which is a property of how that function is written rather than anything run_by_group() asks for: calling it directly has the same requirement. Passing the column name as a string instead, and indexing with .data[[x]] inside the function, avoids the question entirely.

Bare column names do not survive workers > 1, because parallel execution has to serialize every argument to send it to a worker session and a symbol that only means something inside the data has nothing to send. run_by_group() says so directly if you try. Moving the column name inside the analysis function, with a lambda, works in both modes.

For analyses that are slow or computationally independent across groups, run_by_group() supports parallel execution via furrr. The workers argument controls how many R sessions to use. The ceiling is max(1L, parallelly::availableCores() - 1L), which is environment-aware and reserves one core for the main session. workers = 1L (the default) or workers = NULL runs sequentially.

# Parallel execution using available cores
workers <- max(1L, parallelly::availableCores() - 1L)

results <- run_by_group(
  manifest = "data/jobs/manifest.csv",
  .f       = summarise_species,
  workers  = workers
)

The job manifest produced by write_by_group() is also the input to submitr::htc_gen_submit() in multiple-job mode, making this split-apply pattern the natural on-ramp to high-throughput computing when local parallelism is not enough.

save_output() and generate_manifest()

These two functions form the record half of the toolero workflow. Once an analysis has produced its results, save_output() writes each object to disk and appends a row to a project-level accumulator tracking what was saved, how, and whether the write succeeded. generate_manifest() reads that accumulator at the end of the analysis and writes a project-manifest.json describing every artifact the project produced.

This is the output record, a record of outputs from a computation that has already happened. It is a different document from the job manifest write_by_group() produces, which lists inputs to a computation about to happen. The file is named project-manifest.json, and the function that writes it generate_manifest(), for compatibility; the family calls it the output record so that the word "manifest" is not doing two jobs. The vocabulary section of CONVENTIONS.md lists all four terms.

save_output() wraps any write function behind a narrowly-scoped tryCatch(). A failed write is recorded with status = "failure" and the caught error message before the original condition is rethrown unmodified, so the output record captures what went wrong on an unattended run even if nothing else does.

# Save a model and record it
save_output(
  model,
  "output/model.rds",
  .f   = saveRDS,
  note = "Final model, trained on full dataset."
)

# Save a plot. ggsave() takes the filename first and the plot second,
# which is the reverse of the order save_output() passes them in, so it
# needs a wrapper. Writers that take the object first -- saveRDS(),
# write.csv(), write_clean_csv() -- can be passed directly.
save_output(
  my_plot,
  "output/figures/coefficients.png",
  .f     = \(object, file_path, ...) ggplot2::ggsave(file_path, object, ...),
  width  = 8,
  height = 5
)

# Write the output record at the end of the analysis
generate_manifest()

A document under reports/ runs with reports/ as its working directory when rendered, which is why a bare "output/model.rds" used to put its results in reports/output/. toolero now reads that relative path from the project root instead, so you write "output/model.rds" and leave here::here() out. On a cluster the root is the job's scratch directory, which is where the results folder is. save_output() and generate_manifest() follow the same rule for their own default: output_dir is output/ under the project root. And save_output() records each file_path relative to the project root (output/model.rds), so the output record never carries your home directory.

This is also the reason to save through save_output() when the writer is not a toolero function. saveRDS() and ggsave() do not know about the project root, so given "output/model.rds" they would read it from the working directory. save_output() hands the writer a path that has already been resolved, as the call to ggsave() above shows.

save_output() calls the writer as .f(object, file_path, ...), so a writer whose own signature puts the destination first needs a wrapper, as ggsave() does above. The cost is that the accumulator records the function_used column as anonymous function: ... rather than a clean name, which is a small loss of provenance in exchange for the write working at all.

The accumulator at output/accumulator.csv is append-only and written incrementally throughout the analysis. The output record at output/project-manifest.json is the deduplicated, end-of-run summary: execution_context and generated_at recorded once at the top level, followed by an artifacts array with one entry per output file. When an analysis re-runs within a session and overwrites an earlier output, the output record keeps only the most recent write per path.

Like write_by_group(), both functions accept a config argument: a path to a project's _toolero.yml, from which output_dir is resolved (via the config's output_dir convention) when not supplied directly. An explicit output_dir still wins, and a project without a config behaves exactly as before.

generate_manifest() also accepts a git_root argument (default ".", the project root) and records a commit field in the output record -- the git commit checked out in git_root at the moment the record was written, recorded once at the top level alongside execution_context and generated_at. This is the one piece of provenance package versions cannot supply: renv.lock already records which package versions were in play, but nothing else records which revision of the analysis script itself produced a given set of outputs. commit is null when git_root is not a git repository, has no commits yet, or git is not installed, so its absence is informative rather than a failure.

A missing accumulator when generate_manifest() runs is an error, since no save_output() calls were ever recorded. An accumulator with no rows produces an empty output record with a warning, since that is a truthful result rather than a setup mistake.

For unattended execution on CHTC where nobody is watching the job log in real time, the recommended pattern is:

tryCatch(
  {
    # ... analysis code ...
    save_output(results, "output/results.rds", .f = saveRDS)
  },
  finally = try(generate_manifest(), silent = TRUE)
)

The try() inside finally ensures that a crash before the first save_output() call, which leaves no accumulator on disk, does not replace the original error with generate_manifest()'s missing-accumulator error in the job log.

What holds still. The accumulator's column set is the contract between these two functions and anything downstream that reads what an analysis produced, encapsulr::describe() included. Those columns are file_path, r_class, timestamp, function_used, status, error_message and note, and a change to them gets a NEWS.md entry and a deprecation cycle rather than a straight swap. save_output() checks the header of an existing accumulator against that schema before appending and aborts on a mismatch rather than writing misaligned rows. The functions themselves are newer and may still move; the schema is what to build against.

read_output_records()

read_output_records() is the reader for the output record, and it is experimental: its columns may still move before it settles. It returns a tibble with one row per artifact. The seven artifact fields come first, exactly as recorded; after them come read_from, schema_version, and the run-level facts (execution_context, generated_at, commit), repeated on each of that run's rows; and a source column at the front says which folder each row came from.

# The project's own output/, found the same way save_output() finds it
records <- read_output_records()

# What failed?
records[records$status == "failure", c("file_path", "error_message")]

Given several folders, it reads each one and stacks the results. That is the shape a multi-job run comes back in. submitr::htc_collect() (submitr 0.2.0 or later) extracts each job's results folder and returns an index of where they landed; read_output_records() then reads what those folders say about themselves. Name the folders, and the names become source:

jobs    <- submitr::htc_collect()
jobs    <- jobs[!is.na(jobs$output_dir), ]  # drop jobs whose results never arrived
records <- read_output_records(setNames(jobs$output_dir, jobs$group_id))

The two calls split the work the way the family does. submitr knows where the results landed and whether each job brought a record back; toolero wrote the record, so toolero is the package that interprets it (see section 7 of CONVENTIONS.md).

A folder with no usable output record, such as a job that crashed before generate_manifest() ran, is read from its accumulator instead, keeping the latest row per file, with a warning; its run-level columns are NA. A folder with neither contributes no rows, also with a warning, so one incomplete job never stops the rest from being read. A record written by toolero 0.5.x, which has no schema_version, reads as version 1; a record from a newer toolero reads as far as possible, with a warning.

detect_execution_context() and resolve_input_path()

An analysis written in a .qmd runs in three places over its life: in your console while you develop it, under quarto render when you produce the document, and under Rscript when a cluster runs the derived script. Each one finds its input data somewhere different.

detect_execution_context() identifies which of the three you are in and returns "interactive", "quarto", or "rscript". Use it directly for anything that varies by context but is not about data.

# Progress output is useful at a console and only clutters a job log
if (detect_execution_context() == "rscript") {
  options(cli.progress_show_after = Inf)
}

For the data path specifically, use resolve_input_path(). It picks the branch, checks that the result is usable, and says what to fix when it is not.

input_file <- resolve_input_path(
  interactive = "data-raw/sample.csv",
  quarto      = params$input_file,
  rscript     = commandArgs(trailingOnly = TRUE)[1]
)

Only the branch matching the current context is evaluated, so the params reference is safe under Rscript, where params does not exist at all.

The checking is the point. Each of those branches can quietly produce something that is not a path. commandArgs(trailingOnly = TRUE)[1] is NA_character_ when no argument was passed, which is what you get from a submitr single-mode job with no data_files set. params exists only if the YAML header declares it, and params$input_file is NULL if the block exists without that key. Without the check, all of these surface one call later as a message about NA or NULL from whatever tried to read the file, naming neither the context nor the fix, from a job log on a machine you are not sitting at.

You can also leave the arguments off. The rscript branch defaults to the first command line argument, and the other two fall back to the document's own params$input_file, which makes the YAML header the single place the path is written rather than one place in the header and another in a chunk that has to be kept in step with it.

input_file <- resolve_input_path()

In the interactive and Quarto contexts, a relative path is read from the project root, so a document in reports/ can declare input_file: data-raw/sample.csv and find the file whether it is rendered or run chunk by chunk. The rscript context is left alone: on a cluster the path arrives as a command line argument that already points at the file.

Set must_exist = FALSE when the resolved value is a URL or anything else that is not a local file.

Documentation and communication utilities

generate_kb_xml()

Produces a UW-Madison Knowledge Base importable XML file from a rendered Quarto document. Write and maintain the guide in Quarto, then generate the KB-ready XML from that source. The Quarto document remains the maintained version and the XML becomes a derived artifact, reducing documentation drift.

generate_kb_xml(
  html_path  = "docs/analysis.html",
  output_dir = "exports"
)

When importing the resulting XML into the KB, check the Decode HTML entity in body content option.

generate_citation()

Writes a CITATION.cff skeleton -- the Citation File Format GitHub and other tools use to render a "Cite this repository" button and machine-readable citation metadata -- so a project's citation file does not have to be built by hand.

generate_citation(profile = "~/toolero-profile.yml")

Arguments:

title, version, repository-code, url, and license are facts about the project, not the person, so a profile has no way to supply them and they are left as placeholders (some commented out) regardless of whether profile is used. date-released is filled in with today's date.

The given-names/family-names split is done by breaking a profile author's name on its last space -- right for the ordinary two-word case, and wrong for some real names: multi-word family names, single-word names, and family-name-first orderings all defeat it. Review the generated file's given-names and family-names fields before relying on them, especially for names that do not follow that pattern.

If a supplied profile has no author field at all, generate_citation() warns and falls back to the same placeholder author it would have written with no profile supplied.

generate_license()

Writes a plain-text LICENSE file at a project's root from one of a small set of common license templates, with the copyright holder and year filled in.

generate_license(license = "MIT", holder = "Jane Researcher")

Arguments:

"MIT" and "CC0" are both short enough to reproduce in full. "GPL-3"'s full text runs to several hundred lines; rather than risk an inaccurate transcription, license = "GPL-3" writes the notice the Free Software Foundation itself recommends attaching to a program, together with a link to the canonical full text.

This is different from how an R package normally carries a license -- a package's DESCRIPTION declares License: MIT + file LICENSE and pairs a two-line CRAN stub with the full text in LICENSE.md. The projects init_project() scaffolds are research compendia, not packages, so generate_license() instead writes one self-contained file carrying the license text directly, the way a plain GitHub repository does.

generate_data_doc()

Writes a Markdown documentation stub for a single dataset, with the sections a data-management plan typically asks for: source, date obtained, license and usage terms, collection method, a variables table, and known issues. The dataset's file name and today's date are pre-filled; everything else is left as a placeholder to complete by hand -- the same skeleton-you-complete-yourself approach generate_citation() uses for CITATION.cff.

generate_data_doc("survey_responses.csv")

Arguments:

arborize()

Renders a syntactic tree as a standalone PNG image using Quarto's Typst engine. Accepts bracket notation for simple trees or structured notation for trees requiring movement arrows and per-node styling. A provenance .yaml file is written alongside the PNG by default, recording the tree string and render settings so the image can be reproduced or modified later.

# Simple bracket notation
arborize(
  "[NP [Det the] [N cat]]",
  output    = "figures/np-tree.png",
  papersize = "a6"
)

The papersize argument controls how tightly the image is cropped around the tree. Use "a6" or "a7" for small trees, "a5" (the default) for medium trees, and "a4" or "a3" for wide or deep trees. Requires Quarto 1.4+ with Typst support and the pdftools package.

toolero inside a container

An analysis scaffolded by create_qmd() loads toolero in its setup chunk, and the script purled out of it inherits that call. So a containerized toolero analysis needs toolero inside the image. That is the design, and the alternative is worse: researchers hand-rolling context detection and output recording on an execute node. But it has two consequences worth stating rather than discovering.

First, toolero has to be in renv.lock, which is the concrete reason check_project() reports a lockfile that records nothing. Take a snapshot after the analysis exists and before containerizing.

Second, it puts toolero on the critical path of every job. Four functions run on the execute node, detect_execution_context(), resolve_input_path(), save_output() and generate_manifest(), and a change to their return values or file schemas changes the behaviour of analyses inside images built against an older version. They are treated as a stability surface accordingly.

Dependencies

toolero builds on a focused set of R packages for project setup, file handling, data import, documentation, and workflow automation:

cli, fs, glue, here, janitor, jsonlite, lifecycle, parallelly, purrr,
quarto, readr, renv, rlang, rprojroot, rvest, tibble, tidyr, usethis, utils,
withr, xml2, yaml

Some functions need a package that is suggested rather than required, so that installing toolero does not drag in more than most users need:

knitr      required by qmd_to_r()
furrr      required by run_by_group() when workers > 1
future     required by run_by_group() when workers > 1
pdftools   required by arborize()
dplyr      used in the examples throughout this README

Related packages

toolero is the first step in a family of packages for reproducible research workflows:

Each package can be used independently. The shared design goal is to make good research-computing practices easier to adopt before a project becomes difficult to change.

The folder names, path conventions, and shared vocabulary used consistently across all three packages are collected in one place, CONVENTIONS.md, which lives in this repository since toolero is where those conventions are authored. containr and submitr link back to it rather than repeating it.

Citation

citation("toolero")

License

MIT © Erwin Lares