Package 'SpaDES.project'

Description

Quickly setup 'SpaDES' project directories and add modules using templates.

Author(s)

Maintainer: Eliot J B McIntire [email protected] (ORCID)

Other contributors:

• Alex M Chubaty [email protected] (ORCID) [contributor]

• Ian Eddy [email protected] (ORCID) [contributor]

• Ceres Barros [email protected] [contributor]

See Also

Useful links:

• https://spades-project.predictiveecology.org/

• https://github.com/PredictiveEcology/SpaDES.project

• Report bugs at https://github.com/PredictiveEcology/SpaDES.project/issues

Description

For a given name, this will return the default library for packages.

Usage

.libPathDefault(name)

Arguments

Value

A path where the packages will be installed.

Description

Render a scenario (or list of them) as the canonical output path.

Usage

as_path(x, pre = "outputs", withFieldLabel = .scenario_env$withFieldLabel)

Arguments

Description

Method-specific arguments (mapping, name_col, fields, withFieldLabel) are forwarded via ... to the dispatched method; see the method definitions in R/scenario.R.

Usage

as_scenario(x, ...)

Arguments

Value

A scenario (single input) or list of scenarios.

Description

Render a scenario as an upload tar filename.

Usage

as_tarname(
  x,
  ext = ".tar.gz",
  pre = "outputs",
  withFieldLabel = .scenario_env$withFieldLabel
)

Arguments

Description

This is particularly useful to build plots using the tidyverse, e.g., ggplot2. Ported here from the now-unmaintained SpaDES.experiment package.

Usage

## S3 method for class 'simLists'
as.data.table(
  x,
  keep.rownames = FALSE,
  ...,
  vals,
  objectsFromSim = NULL,
  objectsFromOutputs = NULL
)

Arguments

Details

See examples.

Value

This returns a data.table class object.

Description

Assess simulation status from PNG outputs

Usage

assessDoneInFigure(
  runName,
  timeout_min = 20,
  statusCalculate = getOption("spades.statusCalculate")
)

Arguments

Description

Blocks the calling R session until every worker has completed. Optionally prints a summary of final queue statuses.

Usage

awaitExperimentFuture(ef, verbose = TRUE)

Arguments

Value

The ef object, invisibly.

Description

Polls squeue -j <ids> every interval_s seconds until every job ID has left the queue. Optionally prints a final queue-status summary.

Usage

awaitExperimentSBATCH(es, interval_s = 30, verbose = TRUE)

Arguments

Value

The es object, invisibly.

Run an experiment using SpaDES.core::spades()

Description

A wrapper around experiment2() that builds a fully-factorial set of simLists from a single base simList plus alternative params / modules / inputs / objects, then runs them via experiment2()'s future backend. The factorial design is built with factorialDesign().

Usage

experiment(
  sim,
  replicates = 1,
  params,
  modules,
  objects = list(),
  inputs,
  dirPrefix = "simNum",
  substrLength = 3,
  saveExperiment = TRUE,
  experimentFile = "experiment.RData",
  clearSimEnv = FALSE,
  notOlderThan,
  cl,
  ...
)

Arguments

Details

This function (and the simLists class it produces) was moved here from the now-unmaintained SpaDES.experiment package. Two behavioural notes versus the historical version: parallelism is now controlled by future::plan() rather than a cl cluster object (the cl argument is accepted but ignored, with a message), and the return value is a simLists object (as from experiment2()) rather than a plain list. The experimental design table is still saved to experimentFile and is attached to the result (see Value).

Value

Invisibly, a simLists object. The experimental design list (expDesign + expVals) is attached as an attribute named "experiment" on the object's data environment, i.e. attr([email protected], "experiment"), and is also written to experimentFile.

Author(s)

Eliot McIntire

See Also

experiment2(), factorialDesign(), as.data.table.simLists(), experiment_family

Description

A SpaDES "experiment" is a way of running a simulation many times with varying inputs, parameters, paths, scenarios, or replicates. This lets you run, for example, replication of stochastic models, hypothesis testing with different data inputs, scenario analysis of different human decisions, building large datasets of alternative mechanisms to enable ensemble modeling, and other possibilities.

Details

There are five functions to choose from; these can be classified into two groups. The first group (experiment() / experiment2()) is conceptually simpler: it works on in-memory simList objects directly. The second group (experimentTmux() / experimentFuture() / experimentSBATCH()) is built around a project global.R script (typically where setupProject() is run) and a shared job queue. This second group becomes more useful as the number of runs (e.g., scenarios, replicates) become numerous, long, spread across machines, or being run from a High Performance Compute cluster.

In-memory simLists

These take simList object(s) directly and are analogous to running SpaDES.core::spades(). They return results as a simLists ("plural") object you can post-process e.g., with as.data.table.simLists() or any other custom methods. These functions are best when the run set is modest, fits in RAM, and you want the result objects back in your session. They are not built for resume-after-crash, cross-machine pulls, or HPC. (Moved here from the now-unmaintained SpaDES.experiment package.)

experiment2()

The core in-memory runner: give it one or more simLists (and optionally replicates) and it runs them all and returns a simLists. You build the variation yourself, e.g. with several SpaDES.core::simInit() calls.

experiment()

A light wrapper around experiment2() that builds the variation for you: give it one base simList plus alternative params / modules / inputs / objects and it constructs the fully-factorial set of simLists (via factorialDesign()) and runs them. factorialDesign() is exported separately, so the same design can also seed the df of the second group below.

With a source file (e.g., global.R)

Here, the user describes the experiment using a data.frame (or data.table) in which each column name and row value defines the set of object-value pairs that will be assigned to variables in the .GlobalEnv. When the user runs one of these functions, the data.frame is translated into a queue data.frame that has all the same columns and rows, plus a few more (status, claimed_by, etc.) to coordinate the run. After creating the queue, the function spawns a number of independent R "worker" sessions (according to n_workers or cores). Each worker selects a single row, assigns the values in each user-specified column to an object in the .GlobalEnv whose name is the column name, then source()s global.R. For example, if the data.frame has 2 rows and a column named runName with values "trial1" and "trial2", the first worker runs ⁠runName <- "trial1"; source(global_path)⁠ and the second runs ⁠runName <- "trial2"; source(global_path)⁠. The status column starts as "PENDING" for all rows; workers take the next "PENDING" (or "INTERRUPTED") row, skipping "DONE" rows, and mark a row "DONE" when it finishes without error before moving to the next.

These three share the queue and the run-naming convention and differ only in how parallel workers are spawned:

experimentTmux()

Allows the most interactivity and so is helpful when there is still debugging to perform. This will only work on a computer that has tmux installed. The function spawns one tmux pane per worker, optionally across ssh-reachable machines. Best for interactive use where you want to watch workers live (⁠tmux attach⁠). Workers can be stopped with tmuxKillPanes().

experimentFuture()

When there is little to no debugging necessary, this function will use background R processes using either callr::r_bg() if all workers are local, or future::cluster if some of the workers are on different machines. Best for stable scripts. Workers can be stopped with killExperimentFuture().

experimentSBATCH()

One Slurm batch job per worker. Best for HPC clusters with sbatch / squeue / scancel. Block with awaitExperimentSBATCH() (polls squeue) or stop with killExperimentSBATCH() (graceful via stop files; force = TRUE issues scancel). Inspect generated job scripts with dry_run = TRUE.

All three of these accept the same core arguments:

df

The parameter grid; one row = one job. Column names become R variables in the worker's .GlobalEnv before global.R is sourced.

global_path

Path to the R script each worker sources per job. Must be on a filesystem visible to all workers (matters for experimentSBATCH() and remote-host modes of the other two).

queue_path

Path to the local RDS queue file. Workers coordinate through file-based locks on this file; remove it (or point a fresh path) to start over, leave it to resume.

runNameLabel

Quoted expression evaluated against each row to derive a human-readable identifier (used in log messages, sentinel filenames, and tmuxListPanes() output). Default is the first two non-meta columns of the queue.

statusCalculate

Optional quoted expression that inspects the job's outputs and returns up-to-date status / heartbeat metadata. statusCalculate_LandR and statusCalculate_FireSenseFit are pre-built blocks for the most common SpaDES module outputs.

ss_id

Optional Google Sheets / Drive folder ID. When provided, workers mirror queue state to a sheet so a remote stakeholder can watch progress in a browser. With ss_id = NULL (default) the queue is purely local – no Google APIs are touched.

A typical usage pattern:

df <- expand.grid(.scenario = c("A", "B"), .rep = 1:2,
                  stringsAsFactors = FALSE)
ef <- experimentFuture(df = df, global_path = "global.R",
                       n_workers = 2L, log_dir = "logs")

Swap experimentFuture() for experimentTmux() or experimentSBATCH() (adjusting cores / n_workers / sbatch_opts) and the rest of the driver script is unchanged.

Why not just run ⁠Rscript -e ...⁠ per row?

At its core, that is exactly what each worker does. A worker assigns the row's columns into .GlobalEnv and calls source("global.R"), which is equivalent to:

Rscript -e '.ELFind <- "6.3.1"; .rep <- 1; source("global.R")'
Rscript -e '.ELFind <- "6.3.1"; .rep <- 2; source("global.R")'

When the number of sets to run is small, this works. As you add scenarios, machines, authentication, race conditions, etc. the bookkeeping grows past what's comfortable to maintain by hand. The experimentXXX functions are just that bookkeeping.

experimentXXX functions deal with several issues that arise when running "parallel" scripts, including:

Concurrency control

Two shells launched at the same second can both pick the same row. The experimentXXX functions take an exclusive filelock lock on the queue between read and write, so each row is claimed at most once across all workers and machines.

Resume after crash / ctrl-C

If a worker dies mid-job, the row is stuck "in progress" with no record. The experimentXXX functions mark the row RUNNING when claimed and DONE / INTERRUPTED when finished, so the next launch skips DONE rows and (optionally, via tmuxRefreshQueueStatus() or experimentFutureList() (kill = TRUE)) demotes orphaned RUNNING rows back to PENDING for re-claim.

Worker-pool sizing

⁠Rscript &; Rscript &; Rscript &⁠ scales as "one process per row", which thrashes the box once you exceed the core count. The experimentXXX functions take n_workers and let each worker pull rows in sequence, so you cap parallelism explicitly.

Cross-machine claims

Spawning N rows on each of M machines means either replicating the parameter grid by hand (and risking duplicate work) or sharding it (and losing dynamic load-balancing). With the experimentXXX functions, every worker on every machine pulls from the same queue, so a slow machine just claims fewer rows.

Live observability

Rscript -e writes nothing structured – you scrape PIDs and tail logs. The experimentXXX functions maintain a queue with status / claimed_by / started_at / process_id / machine_name so queueRead() gives a full snapshot, and experimentFutureList() can enumerate live workers (and kill them) cluster-wide.

Remote-stakeholder visibility

When ss_id is supplied, the queue is mirrored to a Google Sheet a collaborator can open in a browser; without that, "how is the run going?" requires SSH access to the runner machine.

Outputs accounting

queueUploadMissing() / outScenarios() anti-join the queue against the Drive upload folder so you can see which DONE rows still need to be packaged and uploaded.

Run-name + status hooks

runNameLabel and statusCalculate give one place to derive directory names and inspect output artifacts – both per-runner and in tmuxRefreshQueueStatus() for post-hoc rescans – without each global.R re-implementing them.

If you only ever run two rows on one machine and never restart, the two-line shell version is fine. The experimentXXX functions exist for the cases past that.

Cross-machine propagation (cluster modes)

When you launch on more than one machine – experimentTmux(cores = c("mega", "birds")) or experimentFuture(cores = c("localhost", "camas")) – .setup_remote_machine() runs once per unique remote host before any worker starts. It tries to make the remote R session look enough like the local one that global.R runs the same way. What it propagates / sets up:

Package versions

SpaDES.project itself is rsynced from the local .libPaths()[1] to the remote (or, if loaded via devtools::load_all(), the source tree is rsynced and R CMD INSTALL-ed). Require is version- and RemoteSha-checked and rsynced if it's older or comes from a different source than locally. Then Require::Install() installs every package in SpaDES.project's Imports / Depends / LinkingTo, plus any Suggests installed locally (so optional runtime dependencies like googlesheets4, cli, etc. follow along but the dev toolchain doesn't).

Compiled-from-source packages

terra, sf, rgdal, rgeos, lwgeom are forced to compile from source on the remote (so they link against the remote's libgdal.so etc., which may be a different soversion than localhost's).

System libraries

A best-effort sudo -n apt-get install -y --no-install-recommends of the dev headers needed for the source-compiled packages (libgdal-dev, libssl-dev, libcurl4-openssl-dev, libxml2-dev, fonts/graphics, ...). Runs non-interactively; if passwordless sudo isn't configured the failure is logged and setup continues, expecting the libraries to be there already.

R startup environment

The remote ~/.Rprofile gets refreshed with .libPaths(c(<local_lib>, .libPaths())), options(repos = c("https://predictiveecology.r-universe.dev", <local repos>)), options(defaultPackages = ...) (so the remote uses the same minimal default-attached set as a fresh Rscript), and Sys.setenv(CURL_CA_BUNDLE, SSL_CERT_FILE) pointing at the system CA bundle (so libcurl can do HTTPS even when /etc/profile.d/ isn't sourced under non-login SSH). The remote $BASH_ENV, if set, is wrapped in a subshell guard so a misbehaving sleep $UNSET can't kill the SSH command shell before R starts.

GitHub credentials

The local GITHUB_PAT (read from gitcreds::gitcreds_get() or a caller-supplied local_pat_file) is written to the remote ~/.Renviron (chmod 0600) and to a per-lib file <local_lib>/.spades_github_pat that's read at the top of ~/.Rprofile. git credential approve is also called so command-line git on the remote authenticates the same way. Required for pak to install private modules / dev packages from GitHub.

Google credentials

The experimentXXX functions pass email + cache_path into each worker; the worker calls googlesheets4::gs4_auth(email = email, cache = cache_path) non-interactively against the same cached OAuth token directory the local session uses. The token directory itself isn't pushed (it's expected to already exist via NFS or a prior login on the remote); only the gargle_oauth_email / gargle_oauth_cache options are forwarded so the same identity is selected. If the cache isn't there, the worker prints a gs4_auth warning and continues without GS access.

User code: R/ folder + modules

The directory next to global.R called R/ (where project-specific helper functions live) is rsync -a --delete-ed to the remote, so anything global.R source()s from R/ works there too. With copyModules = TRUE the SpaDES module path (getOption("spades.modulePath")) is also rsynced, so module code stays in step.

The job artifacts themselves

global.R and the queue .rds are scp'd into the same path on the remote (or, if the path is already on NFS such as /mnt/shared_cache/..., they're effectively no-ops – same absolute path on both ends).

Net effect: global.R on camas sees the same packages at the same versions, the same GITHUB_PAT, the same R/ helpers, the same SSL trust store, and the same Google identity as global.R on mega. Hand-rolling all of that for each remote machine before each run is the bulk of what makes "Rscript -e ... on N hosts" miserable in practice; the experimentXXX functions do it once per unique host per call.

Managing remote workers from the calling machine

Once experimentFuture(cores = c("localhost", "camas", "dougfir")) is launched, the workers on camas and dougfir are no longer reachable via local ps / tools::pskill() – they are R processes on other machines. experimentFutureList() is the cluster-wide handle for them. Pass it the ef object and it will:

1. Read the queue file (which is the authoritative record: every claim writes machine_name + process_id under a filelock, so workers on every machine appear there even when they didn't redirect their stdout to a discoverable ⁠worker_NN.log⁠).

2. Probe each entry in ef$cores once with ssh <core> hostname -s to build a map from OS hostname (which is what Sys.info()[["nodename"]] writes to the queue) to the SSH alias the master used to reach it (e.g. A159604 -> dougfir). This is needed because ssh A159604 typically fails – only ssh dougfir resolves via ~/.ssh/config / /etc/hosts.

3. For every status == "RUNNING" row, verify the worker is actually alive: file.exists("/proc/<pid>") for the local machine, batched ssh <alias> "[ -d /proc/<pid> ]" for each remote machine (one SSH connection per machine).

4. Return a data.frame with pid, machine, started_at, queue_path, runName for every live worker – local and remote, in one table.

kill = TRUE uses the same map to send the chosen signal (TERM default, INT or KILL on request): tools::pskill() for local PIDs and a single batched ssh <alias> "kill -<sig> p1 p2 ..." per remote machine. After signalling, it polls (locally via /proc, remotely via SSH) for up to 10 s until the workers actually exit, then runs tmuxRefreshQueueStatus() on each unique queue file to demote the now-orphaned ⁠RUNNING⁠ rows back to ⁠PENDING⁠. When ss_id was supplied to the original experimentFuture() call, an <queue_path>.ss_id sidecar is left behind; kill = TRUE reads it and pushes the same demotion to the Google Sheet via .gs_demote_after_kill(), so the GS view converges with the local queue without a separate cleanup step.

Three usage shapes:

experimentFutureList(ef)                   # list everything live
experimentFutureList(ef, kill = TRUE)      # graceful TERM + queue refresh
experimentFutureList(ef, kill = TRUE, signal = "KILL")  # immediate

Across R sessions, when ef is gone, drive discovery off the queue path directly:

experimentFutureList(queue_paths = "/mnt/shared_cache/.../future_queue.rds")

Without ef, the hostname-to-alias probe is skipped, so the SSH check uses machine_name verbatim – which only works if the OS hostname is itself reachable via SSH on the calling node (i.e. it appears in ~/.ssh/config or /etc/hosts as a Host entry). If not, you'll need to either keep ef in scope or add the OS hostnames to your SSH config.

Concretely, the things you can do post-launch from the calling machine without ever opening a terminal on camas / dougfir:

• See which row each remote worker is currently on.

• Confirm that a remote worker actually died after a crash / network blip (otherwise the queue would stay stuck at ⁠RUNNING⁠ and no one would re-claim).

• Send SIGTERM cluster-wide to abort an experiment mid-run, then immediately re-launch a fixed global.R against the same queue (any DONE rows are skipped, demoted RUNNING rows are re-claimed).

• Mirror that demotion to the Google Sheet so a stakeholder watching in a browser sees the change without needing to be told.

Resource monitoring (CPU + RAM)

experimentMonitor() is the read-only entry point. Discovery depends on what you pass:

• experimentMonitor() (no args) – enumerates every tmux pane on the calling machine across all tmux servers, same as the historical tmuxListPanes().

• experimentMonitor(ef) – queue-driven discovery across all machines in ef$cores (with the hostname-to-SSH-alias probe described above).

• experimentMonitor(queue_paths = "...") – same as ef mode, but for cross-session use when the ef handle is gone.

stats = TRUE batches ps -o pid=,%cpu=,rss=,state= (locally and via one SSH connection per remote node) to append:

• state – R (running on CPU), S (sleeping / waiting), D (uninterruptible sleep, often disk I/O – persistent D = hang), T (stopped), Z (zombie), Closed (R session exited but tmux pane still open).

• cpuAvg – percent CPU averaged over the process's lifetime (note: not the instantaneous rate htop shows).

• ⁠RAM (GB)⁠ – resident memory (RSS), 1 decimal place.

• availableCores – total CPUs on the node, from nproc.

• ⁠total RAM (GB)⁠ – total RAM on the node, from /proc/meminfo.

availableCores and ⁠total RAM (GB)⁠ are constant across all rows on the same node, so each pane's resource use is visible relative to its node capacity. Unreachable nodes get NA for all their rows; titles missing a parseable ⁠<node>-<pid>⁠ get NA too – one bad pane / unreachable host doesn't poison the rest of the table.

Single function, three sources, same stats columns either way – so a stakeholder running experimentMonitor(ef, stats = TRUE) on a laptop sees the same per-worker CPU / RAM picture that experimentMonitor(stats = TRUE) (legacy tmux mode) gives on the master node. tmuxListPanes() is preserved as a thin alias that calls experimentMonitor() with no ef, so older code keeps working unchanged.

Related families:

• scenario_family – canonical record for one row of df, reversibly convertible between field values, an output directory path, and an upload tarball filename.

• queueRead() / queueUploadMissing() / outList() / outScenarios() – helpers for queues persisted to a Google Sheet plus a Drive upload folder, including the queue-vs-uploads anti-join.

• experimentMonitor() – read-only worker / pane lister. With no args, scans tmux panes; with ef or queue_paths, scans the queue file's RUNNING rows and verifies each PID is alive (local /proc, batched SSH for remotes). stats = TRUE adds per-worker CPU / RSS / state and per-node nproc / total RAM via batched ps. tmuxListPanes() is a thin alias for the no-args form.

• tmuxRefreshQueueStatus() / tmuxFindDuplicates() / tmuxKillPanes() – operational tools that work regardless of which runner produced the queue.

• experimentFutureList() – experimentFuture-side equivalent of tmuxListPanes(): discovers live workers across the cluster (driven off the queue file's RUNNING rows plus an ssh <core> hostname -s alias probe), and with kill = TRUE sends TERM / INT / KILL to all of them in one call (local via tools::pskill(), remote batched per machine via SSH), then refreshes the queue and demotes the matching Google-Sheet rows when an <queue_path>.ss_id sidecar is present.

Controlling which events run

All of these honour spades()'s events argument, which restricts the events executed for each module (see SpaDES.core::spades()):

• experiment() / experiment2(): pass events as a named argument; it is forwarded to every spades() call, e.g. experiment2(sim1, sim2, events = list(fireSpread = "init")). The same events apply to all simulations / replicates.

• experimentTmux() / experimentFuture() / experimentSBATCH(): there is no events argument because these functions do not call spades() – your global.R does. To control events, add an events column to df (each cell is the events spec for that row) and, inside global.R, call spades(sim, events = events). Because each row carries its own value, this gives per-scenario control of which events run for any particular module – something the single shared events of the in-memory family cannot do.

Run experiment, algorithm 2, using SpaDES.core::spades()

Description

Given one or more simList objects, run a series of spades calls in a structured, organized way. Methods are available to deal with outputs, such as as.data.table.simLists() which can pull out simple to complex values from every resulting simList or object saved by outputs in every simList run. This uses future internally, allowing for various backends and parallelism.

Usage

experiment2(
  ...,
  replicates = 1,
  clearSimEnv = FALSE,
  createUniquePaths = c("outputPath"),
  useCache = FALSE,
  debug = getOption("spades.debug"),
  drive_auth_account = NULL,
  meanStaggerIntervalInSecs = 1
)

Arguments

Details

This function was moved here from the now-unmaintained SpaDES.experiment package. See also the file-queue based experiment_family (e.g. experimentFuture()) for a different, script-oriented approach.

This function, because of its class formalism, allows for methods to be used. For example, as.data.table.simLists() allows user to pull out specific objects (in the simList objects or on disk saved in outputPath(sim)).

The outputPath is changed so that every simulation puts outputs in a sub-directory of the original outputPath of each simList (unless createUniquePaths is character(0)/NULL).

Value

Invisibly returns a simLists object. This class extends the environment class and contains simList objects.

Controlling events

Any named argument in ... that is not consumed by experiment2 is passed straight to SpaDES.core::spades(). In particular, spades()'s events argument is honoured, so experiment2(sim1, sim2, events = list(...)) restricts the events that run for every simulation. Note this applies the same events specification to all simLists / replicates. For per-scenario control of events, use the file-queue experiment_family with an events column in df (see experiment_family).

Note

A simLists object can be made manually, if, say, many manual spades calls have already been run. See example, via new("simLists")

Author(s)

Eliot McIntire

See Also

as.data.table.simLists(), SpaDES.core::spades(), experiment(), experiment_family

Examples

## Not run: 
  if (require("ggplot2", quietly = TRUE) &&
      require("NLMR", quietly = TRUE) &&
      require("RColorBrewer", quietly = TRUE)) {
    library(SpaDES.core)
    library(SpaDES.project)

    tmpdir <- file.path(tempdir(), "examples")
    # Make 3 simLists -- set up scenarios
    endTime <- 2

    # Example of changing parameter values
    # Make 3 simLists with some differences between them
    mySim <- lapply(c(10, 20, 30), function(nFires) {
      simInit(
        times = list(start = 0.0, end = endTime, timeunit = "year"),
        params = list(
          .globals = list(stackName = "landscape", burnStats = "nPixelsBurned"),
          # Turn off interactive plotting
          fireSpread = list(.plotInitialTime = NA, spreadprob = c(0.2), nFires = c(10)),
          caribouMovement = list(.plotInitialTime = NA),
          randomLandscapes = list(.plotInitialTime = NA, .useCache = "init")
        ),
        modules = list("randomLandscapes", "fireSpread", "caribouMovement"),
        paths = list(modulePath = system.file("sampleModules", package = "SpaDES.core"),
                     outputPath = tmpdir),
        # Save final state of landscape and caribou
        outputs = data.frame(
          objectName = c(rep("landscape", endTime), "caribou", "caribou"),
          saveTimes = c(seq_len(endTime), unique(c(ceiling(endTime / 2), endTime))),
          stringsAsFactors = FALSE
        )
      )
    })

    planTypes <- c("sequential") # try others! ?future::plan
    sims <- experiment2(sim1 = mySim[[1]], sim2 = mySim[[2]], sim3 = mySim[[3]],
                        replicates = 3)

    # Try pulling out values from simulation experiments
    # 2 variables
    df1 <- as.data.table(sims, vals = c("nPixelsBurned", NCaribou = quote(length(caribou$x1))))

    # Now use objects that were saved to disk at different times during spades call
    df1 <- as.data.table(sims,
                         vals = c("nPixelsBurned", NCaribou = quote(length(caribou$x1))),
                         objectsFromOutputs = list(nPixelsBurned = NA, NCaribou = "caribou"))


    # now calculate 4 different values, some from data saved at different times
    # Define new function -- this calculates perimeter to area ratio
    fn <- quote({
      landscape$Fires[landscape$Fires[] == 0] <- NA;
      a <- boundaries(landscape$Fires, type = "inner");
      a[landscape$Fires[] > 0 & a[] == 1] <- landscape$Fires[landscape$Fires[] > 0 & a[] == 1];
      peri <- table(a[]);
      area <- table(landscape$Fires[]);
      keep <- match(names(area),names(peri));
      mean(peri[keep]/area)
    })

    df1 <- as.data.table(sims,
                         vals = c("nPixelsBurned",
                                  perimToArea = fn,
                                  meanFireSize = quote(mean(table(landscape$Fires[])[-1])),
                                  caribouPerHaFire = quote({
                                    NROW(caribou) /
                                      mean(table(landscape$Fires[])[-1])
                                  })),
                         objectsFromOutputs = list(NA, c("landscape"), c("landscape"),
                                                   c("landscape", "caribou")),
                         objectsFromSim = "nPixelsBurned")

    if (interactive()) {
      # with an unevaluated string
      library(ggplot2)
      p <- lapply(unique(df1$vals), function(var) {
        ggplot(df1[vals == var,],
               aes(x = saveTime, y = value, group = simList, color = simList)) +
          stat_summary(geom = "point", fun.y = mean) +
          stat_summary(geom = "line", fun.y = mean) +
          stat_summary(geom = "errorbar", fun.data = mean_se, width = 0.2) +
          ylab(var)
      })

      # Arrange all 4 -- could use gridExtra::grid.arrange -- easier
      pushViewport(viewport(layout = grid.layout(2, 2)))
      vplayout <- function(x, y) viewport(layout.pos.row = x, layout.pos.col = y)
      print(p[[1]], vp = vplayout(1, 1))
      print(p[[2]], vp = vplayout(1, 2))
      print(p[[3]], vp = vplayout(2, 1))
      print(p[[4]], vp = vplayout(2, 2))
    }
  }

## End(Not run)

Description

A tmux-free alternative to experimentTmux that dispatches parallel workers as background R processes via callr. Workers claim jobs from a GoogleSheets or local-RDS queue, source global_path for each job, and write all console output to per-worker log files on localhost.

The function returns immediately (non-blocking) with a handle object of class "experimentFuture". Use awaitExperimentFuture to block until all workers finish, or print(ef) to check live status. Because workers write to files via callr::r_bg()'s stdout / stderr arguments, logs appear in real time and can be followed with tail -f.

Usage

experimentFuture(
  df,
  global_path = "global.R",
  cores = NULL,
  n_workers = if (is.null(cores)) 4L else length(cores),
  queue_path = NULL,
  on_interrupt = c("requeue", "fail"),
  ss_id = NULL,
  forceLocalQueueToGS = FALSE,
  email = getOption("gargle_oauth_email"),
  cache_path = getOption("gargle_oauth_cache"),
  runNameLabel = quote(colnames(q)[1:2]),
  log_dir = "logs",
  activeRunningPath = getOption("spades.activeRunningPath"),
  sp_dev_path = NULL,
  local_pat_file = NULL,
  copyModules = FALSE,
  ...
)

Arguments

Value

An object of class "experimentFuture" (a list) containing:

procs

List of callr::r_bg process objects, one per local worker (or future objects for remote cluster workers).

log_files

Character vector of log file paths.

log_dir

Absolute path to the log directory.

queue_path

Absolute path to the queue RDS file.

cores

The cores argument as supplied.

See Also

experimentTmux, awaitExperimentFuture, tmuxRunWorkerLoop

Examples

## Not run: 
## -- Minimal: build a tiny global.R, then run a 2 x 2 experiment ---------
tdir <- file.path(tempdir(), "experimentFuture-demo")
dir.create(tdir, showWarnings = FALSE, recursive = TRUE)
writeLines(
  'message("scenario=", .scenario, " rep=", .rep); Sys.sleep(2)',
  file.path(tdir, "global.R")
)
expt <- expand.grid(.scenario = c("A", "B"), .rep = 1:2,
                    stringsAsFactors = FALSE)

ef <- experimentFuture(
  df          = expt,
  global_path = file.path(tdir, "global.R"),
  n_workers   = 2L,
  queue_path  = file.path(tdir, "future_queue.rds"),
  log_dir     = file.path(tdir, "logs")
)

## -- Live inspection while workers run -----------------------------------
print(ef)                                         # alive/done per worker
experimentMonitor(ef)                             # pid + machine + runName
experimentMonitor(ef, stats = TRUE)               # adds CPU / RAM / state
queueRead(ef$queue_path)                          # full queue snapshot
experimentFutureList(ef)                          # cluster-wide pid list
cat(readLines(ef$log_files[[1L]]), sep = "\n")    # tail one log

awaitExperimentFuture(ef)    # blocks until both workers exit

## -- Killing workers ----------------------------------------------------

# Graceful stop: workers finish their CURRENT job, then exit.
# Any remaining PENDING jobs stay in the queue and can be resumed later
# by calling experimentFuture() again with the same queue_path.
killExperimentFuture(ef)

# Immediate stop (force): workers are killed immediately.
# Jobs that were mid-execution may remain as RUNNING in the queue; reset them with:
#   tmuxRefreshQueueStatus(ef$queue_path)   # file-based backend
# The GS backend reclaims stale RUNNING entries automatically before each new claim.
killExperimentFuture(ef, force = TRUE)
tmuxRefreshQueueStatus(ef$queue_path)   # clean up stale RUNNING entries

# Cluster-wide kill (works for `cores = c(...)` clusters too):
# sends SIGTERM to every worker on every machine, waits for exit, runs
# tmuxRefreshQueueStatus(), and pushes the demotion to the Google Sheet
# if `ss_id` was used (via the <queue_path>.ss_id sidecar).
experimentFutureList(ef, kill = TRUE)

## -- Resuming after a kill ----------------------------------------------

# Jobs left as PENDING (or INTERRUPTED with on_interrupt = "requeue") are
# automatically picked up when you call experimentFuture() again with the
# same queue_path  -- no need to re-specify df.
ef2 <- experimentFuture(
  df          = expt,         # ignored if queue_path already exists
  global_path = file.path(tdir, "global.R"),
  n_workers   = 2L,
  queue_path  = file.path(tdir, "future_queue.rds"),
  log_dir     = file.path(tdir, "logs")
)
awaitExperimentFuture(ef2)   # wait for remaining jobs to finish

queueRead(ef2$queue_path)    # full snapshot (data.table)
table(queueRead(ef2$queue_path)$status)   # all DONE

cat(readLines(ef2$log_files[[1]]), sep = "\n")   # inspect worker 1 log

## -- Remote workers (pre-setup required) -------------------------------
ef <- experimentFuture(
  df             = expt,
  global_path    = file.path(tdir, "global.R"),
  cores          = c("node01", "node02"),
  n_workers      = 2L,
  ss_id          = "YOUR_GOOGLE_SHEET_ID",
  email          = "[email protected]",
  cache_path     = "~/.cache/gargle",
  local_pat_file = "~/.github_pat"
)
killExperimentFuture(ef)     # graceful stop on remote workers too

## End(Not run)

Description

Cross-session worker discovery for experimentFuture. Scans /proc for R processes whose redirected stdout points to a ⁠worker_<NN>.log⁠ file (the convention written by callr::r_bg(stdout = log_files[[i]]) in experimentFuture), regardless of which R session originally spawned them. This is the right tool when:

• you re-ran the experimentFuture example in a new R session and a fresh tail -f is silent because the previous run's workers are still claiming queue rows;

• you want to clean up orphans without remembering each ef handle;

• you want a one-glance view of which row each worker is currently running (joined against the queue's status == "RUNNING" process_id).

Linux-only (uses /proc/<pid>/fd/1 to find the log file each worker is writing). For other Unixes use lsof -p <pid> or ps -ef | grep tmuxRunWorkerLoop as a manual substitute.

Usage

experimentFutureList(
  ef = NULL,
  kill = FALSE,
  signal = c("TERM", "INT", "KILL"),
  queue_paths = NULL
)

Arguments

Value

A data.frame (one row per live worker) with columns:

pid

Worker process ID.

started_at

Approximate process start time (ctime of /proc/<pid>).

log_file

Path the worker is writing stdout/stderr to.

queue_path

The first ⁠*_queue.rds⁠ found in the log directory's parent (where experimentFuture puts it by default), or NA if not located.

runName

Hyphen-joined data column values of the row this worker is currently running, derived from the queue's status == "RUNNING" entry whose process_id matches. NA if the worker is between jobs.

When kill = TRUE, the same data.frame is returned (invisibly) describing the workers that were signalled.

See Also

experimentFuture, killExperimentFuture, tmuxRefreshQueueStatus

Examples

## Not run: 
# Just list everything that's running (auto-discovery via /proc only)
experimentFutureList()

# Pass the ef handle to also pick up PSOCK cluster workers and remote
# workers (anything in the queue, on any machine in `cores`).
ef <- experimentFuture(df = df, global_path = "global.R",
                       cores = c("localhost", "camas"), ...)
experimentFutureList(ef)
experimentFutureList(ef, kill = TRUE)

# Across R sessions, when ef is gone, drive discovery off the queue path:
experimentFutureList(queue_paths = "/mnt/shared_cache/.../future_queue.rds")

# Hard kill (SIGKILL, no chance to update queue meta on the worker side --
# but the post-kill tmuxRefreshQueueStatus() still demotes the rows).
experimentFutureList(ef, kill = TRUE, signal = "KILL")

## End(Not run)

Description

Single read-only entry point for inspecting workers regardless of which runner spawned them. Discovery is driven by what you pass:

Usage

experimentMonitor(ef = NULL, queue_paths = NULL, stats = FALSE)

Arguments

Details

• Default (ef = NULL, queue_paths = NULL) – enumerates tmux panes via ⁠tmux -S <socket> list-panes -a⁠ across every tmux server under ⁠$TMUX_TMPDIR/tmux-<uid>/⁠. Same behaviour the historical tmuxListPanes() had. Per-socket failures are swallowed so one broken socket cannot poison the rest; works outside a tmux pane and across multiple tmux servers (e.g. sessions started under different -L names). Cluster_Monitor panes are filtered out.

• ef supplied (or queue_paths) – reads each queue file's status == "RUNNING" rows, probes ⁠ssh <core> hostname -s⁠ once per non-local entry in ef$cores to map OS hostnames (which is what Sys.info()[["nodename"]] writes to the queue) back to SSH aliases (⁠~/.ssh/config⁠ / ⁠/etc/hosts⁠ entries), and verifies each PID is alive (⁠/proc/<pid>⁠ locally, batched ⁠ssh <alias> "[ -d /proc/<pid> ]"⁠ remotely). This is the experimentFuture() / experimentSBATCH() equivalent of the tmux pane scan – workers there don't necessarily live in a tmux pane, so the queue file is the authoritative record.

Either way, stats = TRUE runs the same ⁠ps -o pid=,%cpu=,rss=,state=⁠ batch (locally and via one SSH connection per remote node) to append CPU / RSS / state plus per-node nproc / total RAM.

Value

Data.frame whose columns depend on the discovery mode:

• tmux mode – session, window, pane, pane_id, pane_ref (the "session:window.pane" string), title, node (first dash-separated token in title that matches a cluster alias from ⁠/etc/hosts⁠; falls back to localHostLabel() when the title contains only the raw local hostname; NA if no match).

• queue mode – pid, machine, started_at, log_file (NA when the worker isn't a callr::r_bg writer), queue_path, runName.

With stats = TRUE, five additional columns appear in either mode: state, cpuAvg, RAM (GB), availableCores, ⁠total RAM (GB)⁠. Returns an empty data.frame (0 rows, same columns) if no workers are found.

State codes

The state column is the best single signal for hang-detection because it is a snapshot (no time window needed). Values:

See Also

experimentFutureList() for the same queue-mode discovery plus cluster-wide kill / queue refresh / GS demotion. tmuxListPanes() is preserved as a thin alias that calls this function with no ef.

Description

A Slurm-native sibling of experimentTmux and experimentFuture. Submits n_workers long-lived SBATCH jobs that each call tmuxRunWorkerLoop against the shared queue, claiming and running rows until the queue is empty (or the worker's stop file appears). Same queue / global.R / runNameLabel / statusCalculate semantics as the other two runners.

Returns a non-blocking handle of class "experimentSBATCH" carrying the Slurm job IDs. Use awaitExperimentSBATCH to poll squeue until all jobs leave the queue, or killExperimentSBATCH to stop them (gracefully via stop files, or immediately via scancel).

Usage

experimentSBATCH(
  df,
  global_path = "global.R",
  n_workers = 4L,
  queue_path = NULL,
  on_interrupt = c("requeue", "fail"),
  ss_id = NULL,
  forceLocalQueueToGS = FALSE,
  email = getOption("gargle_oauth_email"),
  cache_path = getOption("gargle_oauth_cache"),
  runNameLabel = quote(colnames(q)[1:2]),
  log_dir = "logs",
  activeRunningPath = getOption("spades.activeRunningPath"),
  sbatch_opts = list(),
  sbatch_cmd = "sbatch",
  r_cmd = file.path(R.home("bin"), "Rscript"),
  r_libs = .libPaths(),
  dry_run = FALSE,
  ...
)

Arguments

Value

An object of class "experimentSBATCH" (a list) containing:

job_ids

Integer vector of Slurm job IDs (or NA under dry_run = TRUE).

job_scripts

Character vector of generated SBATCH script paths.

log_files

Character vector of log file paths.

stop_files

Character vector of stop-file paths.

log_dir

Absolute path to the log directory.

queue_path

Absolute path to the queue RDS file.

See Also

experimentTmux, experimentFuture, awaitExperimentSBATCH, killExperimentSBATCH

Examples

## Not run: 
## -- Minimal: build a tiny global.R, then run a 2 x 2 experiment ---------
# Use a directory on your shared HPC filesystem (NFS / Lustre / BeeGFS).
tdir <- file.path(tempdir(), "experimentSBATCH-demo")
dir.create(tdir, showWarnings = FALSE, recursive = TRUE)
writeLines(
  'message("scenario=", .scenario, " rep=", .rep); Sys.sleep(2)',
  file.path(tdir, "global.R")
)
expt <- expand.grid(.scenario = c("A", "B"), .rep = 1:2,
                    stringsAsFactors = FALSE)

es <- experimentSBATCH(
  df          = expt,
  global_path = file.path(tdir, "global.R"),
  n_workers   = 2L,
  queue_path  = file.path(tdir, "sbatch_queue.rds"),
  log_dir     = file.path(tdir, "logs"),
  sbatch_opts = list(partition = "compute", time = "00:30:00", mem = "1G")
)

## -- Live inspection while jobs run --------------------------------------
print(es)                                          # job IDs + squeue status
queueRead(es$queue_path)                           # full queue snapshot
experimentMonitor(queue_paths = es$queue_path)     # cluster-wide pid + machine
experimentMonitor(queue_paths = es$queue_path, stats = TRUE)  # + CPU/RAM

awaitExperimentSBATCH(es)                          # block until squeue empty

## -- Larger experiment with full sbatch_opts ------------------------------
es <- experimentSBATCH(
  df          = expt,
  global_path = file.path(tdir, "global.R"),
  n_workers   = 4L,
  queue_path  = file.path(tdir, "sbatch_queue.rds"),
  log_dir     = file.path(tdir, "logs"),
  sbatch_opts = list(
    partition     = "compute",
    time          = "24:00:00",
    mem           = "16G",
    cpus_per_task = 4,
    account       = "my_alloc"
  )
)

print(es)                       # job IDs + squeue status per worker
awaitExperimentSBATCH(es)        # blocks until every job ID leaves squeue

# Graceful stop (workers exit between jobs, queue rows stay PENDING):
killExperimentSBATCH(es)

# Immediate stop (scancel; stale RUNNING entries can be cleaned up via:
#   tmuxRefreshQueueStatus(es$queue_path)):
killExperimentSBATCH(es, force = TRUE)
tmuxRefreshQueueStatus(es$queue_path)

## -- Resume after stop ---------------------------------------------------
# Same `queue_path` -> DONE rows are skipped, demoted PENDING rows are
# re-claimed by the new sbatch jobs.
es2 <- experimentSBATCH(
  df          = expt,                # ignored if queue exists
  global_path = file.path(tdir, "global.R"),
  n_workers   = 2L,
  queue_path  = file.path(tdir, "sbatch_queue.rds"),
  log_dir     = file.path(tdir, "logs"),
  sbatch_opts = list(partition = "compute", time = "00:30:00", mem = "1G")
)
awaitExperimentSBATCH(es2)
table(queueRead(es2$queue_path)$status)            # all DONE

## End(Not run)

Description

Creates n_workers tmux panes in the current window, tiles them, and starts a worker loop in each one that claims and runs jobs from a file-backed queue (queue_path). Control returns immediately to the master pane; all work happens asynchronously inside the worker panes.

Worker loop modes (pane_mode)

"killAndNewPane" (default)

Each worker runs one job per R session, then exits. A fresh R session starts automatically for the next job, freeing all memory between runs.

• localhost panes: After each job, tmuxRunWorkerLoop() calls ⁠tmux respawn-pane -k⁠, which replaces the current pane's process in-place with a new Rscript invocation. No retiling needed.

• Remote panes (cores = "hostname"): The local pane runs a bash while-loop that repeatedly calls ⁠ssh -t host bash -c 'exec env R_PROFILE_USER=<script> R --interactive'⁠. ssh -t allocates a PTY so R runs interactively (readline, OSC 2 title updates, Ctrl+C propagation). A startup script injected via R_PROFILE_USER runs one job then exits; q(status = 1L) (job done or queue empty) lets the while-loop start a fresh R session, q() (status 0) stops the loop. R_PROFILE_USER is unset inside R immediately after startup so workers spawned by makeClusterPSOCK() do not inherit it and inadvertently re-run the startup script.

"reuse"

Each worker loops inside a single R session (repeat { tmuxRunNextWorker() }). Memory accumulates across jobs – useful for lightweight simulations.

Remote machine setup (cores)

Supplying a hostname in cores triggers .setup_remote_machine() once per unique host before any workers start. Steps run in this order:

1. Guard BASH_ENV – wraps the remote ⁠$BASH_ENV⁠ file's existing content in a subshell (⁠( ... ) 2>/dev/null || true⁠) so that any exit or failing command inside it cannot abort the non-interactive SSH shell that carries setup commands.

2. Create remote directory; copy files – mkdir -p the remote working directory (same relative path from ~ as on localhost), then scp global_path, queue_path, and dots_path (if supplied) into it.

3. Rsync project ⁠R/⁠ folder – syncs the ⁠R/⁠ subdirectory next to global_path to the remote with rsync --delete so user-defined helper functions sourced by global.R are up to date.

4. Write ⁠~/.Rprofile⁠ on remote – injects three lines (replacing any previous versions): .libPaths(c(local_lib, ...)) so the project library takes precedence over system libraries; options(repos = ...) including the PredictiveEcology r-universe; and an SSL block that sets CURL_CA_BUNDLE/SSL_CERT_FILE so HTTPS downloads work in non-login SSH sessions where ⁠/etc/profile.d/⁠ is not sourced.

5. Verify/install Require – compares the remote Require version and git commit SHA to the local installation. If they differ, rsyncs the installed directory (GitHub source) or runs install.packages("Require") (CRAN source).

6. Install usethis on the remote via Require::Install().

7. Propagate GitHub credentials – reads the local token via gitcreds::gitcreds_get() and pipes it into ⁠git credential approve⁠ on the remote so private GitHub packages can be installed without interactive setup. Falls back to checking whether the remote already has credentials; errors if neither is true.

8. Install system libraries via ⁠sudo -n apt-get install -y --no-install-recommends⁠ (non-interactive; fails gracefully if passwordless sudo is not configured). Libraries installed: spatial (libgdal-dev, libgeos-dev, libproj-dev, libsqlite3-dev, libudunits2-dev), HTTP/TLS (libssl-dev, libcurl4-openssl-dev), XML (libxml2-dev), archive (libarchive-dev), git (libgit2-dev), fonts/graphics (libfontconfig1-dev, libharfbuzz-dev, libfribidi-dev, libpng-dev, libjpeg-dev, libtiff-dev, libfreetype6-dev), protobuf (libabsl-dev), and R compilation headers (r-base-dev).

9. Ensure remote lib path exists – mkdir -p the project library path on the remote (must match localhost exactly so installed file paths are identical).

10. Rsync SpaDES.project – copies the locally installed SpaDES.project directory to the same path on the remote. Both machines must share the same platform and R version so compiled lazy-load databases are compatible.

11. Install SpaDES.project dependencies via Require::Install(). Spatial packages (terra, sf, rgdal, rgeos, lwgeom) are compiled from source so they link against the remote's actual GDAL/GEOS/PROJ versions. All other hard dependencies (Imports/Depends/LinkingTo) plus any Suggests packages installed locally are installed as binaries via Require::setLinuxBinaryRepo(). Common packages with strict version requirements (⁠purrr >= 1.2.1⁠, ⁠rlang >= 1.1.7⁠, ⁠cli >= 3.6.0⁠, ⁠vctrs >= 0.6.0⁠) are pre-installed to the project library to avoid stale system-library versions being picked up during compilation.

12. Rsync Require package cache (Require::cachePkgDir()) to the remote to accelerate future package installations.

13. Rsync gargle OAuth cache (cache_path or getOption("gargle_oauth_cache")) to the remote so the worker can authenticate with Google APIs (Sheets, Drive) without a browser prompt.

Staggered starts

Pane 1 starts immediately. Pane i > 1 waits delay_before_source + (i - 2) * stagger_by seconds inside R before claiming its first job, avoiding simultaneous queue contention at startup. For remote workers in killAndNewPane mode the stagger only applies to the first R session; subsequent while-loop iterations start immediately.

Restarting a broken pane

If a worker pane is manually interrupted (e.g. Ctrl+C) and drops to a shell prompt, restart it by pressing (up-arrow) (up-arrow) in that pane and hitting Enter. The full command is always in the pane's bash history:

• localhost: ⁠Rscript -e "..."⁠ (re-enters tmuxRunWorkerLoop; in killAndNewPane mode respawn-pane takes over from the first job onward).

• remote: ⁠if setup && scp; then first_run; _st=$?; while [ $_st -ne 0 ]; do sleep 2; loop_run; _st=$?; done; fi⁠ command (restarts the sh loop from scratch; plain POSIX – works in bash, dash, and sh).

ANSI colour support

At startup, experimentTmux sets the tmux session option default-terminal = "tmux-256color". This ensures that all subsequently created panes advertise a full-colour ANSI terminal, which is required for R packages such as cli and crayon to render coloured/dynamic output correctly. Without this, connections that arrive via Windows PowerShell -> SSH -> tmux often inherit TERM=screen or no TERM at all, causing R to fall back to plain-text output. The setting is applied globally to the session (-g) and persists for the session's lifetime; it does not modify ⁠~/.tmux.conf⁠.

Usage

experimentTmux(
  df,
  global_path = "global.R",
  cores = NULL,
  n_workers = if (is.null(cores)) 4L else length(cores),
  delay_after_split = 0.4,
  delay_after_layout = 0.4,
  delay_between_R_start = 0,
  delay_before_source = 60,
  stagger_by = delay_before_source,
  set_mouse = TRUE,
  statusCalculate = getOption("spades.statusCalculate"),
  folderWithIterInFilename = getOption("spades.folderWithIterInFilename"),
  activeRunningPath = getOption("spades.activeRunningPath"),
  continue = TRUE,
  queue_path = NULL,
  on_interrupt = c("requeue", "fail"),
  pane_mode = c("killAndNewPane", "reuse"),
  ss_id = NULL,
  forceLocalQueueToGS = FALSE,
  enableGSSync = FALSE,
  email = getOption("gargle_oauth_email"),
  cache_path = getOption("gargle_oauth_cache"),
  workersToMonitor = unique(if (is.null(cores)) "localhost" else cores),
  runNameLabel = quote(colnames(q)[1:2]),
  copyModules = FALSE,
  ...
)

Arguments

Value

Invisibly returns a character vector of tmux pane IDs for the spawned workers. Pass these to tmuxKillPanes() to tear down all workers at once.

Related tmux helpers

Examples

## Not run: 
# --- Minimal: build a tiny global.R, then run a 2 x 2 experiment ---
tdir <- file.path(tempdir(), "experimentTmux-demo")
dir.create(tdir, showWarnings = FALSE, recursive = TRUE)
writeLines(
  'message("scenario=", .scenario, " rep=", .rep); Sys.sleep(2)',
  file.path(tdir, "global.R")
)
expt <- expand.grid(.scenario = c("A", "B"), .rep = 1:2,
                    stringsAsFactors = FALSE)

workers <- experimentTmux(
  df          = expt,
  global_path = file.path(tdir, "global.R"),
  cores       = rep("localhost", 2L),
  queue_path  = file.path(tdir, "queue.rds")
)

# --- Live inspection while panes run ---
experimentMonitor()                       # tmux pane scan (no args)
experimentMonitor(stats = TRUE)           # adds CPU / RAM / state per pane
tmuxListPanes()                           # alias of experimentMonitor()
queueRead(file.path(tdir, "queue.rds"))   # full queue snapshot
tmuxFindDuplicates(workers)               # any double-claimed jobs?
tmuxRefreshQueueStatus(file.path(tdir, "queue.rds"))   # reset stuck rows

# --- Basic local usage with explicit pane sizing ---
workers <- experimentTmux(
  global_path         = "/abs/path/to/global.R",
  queue_path          = "/abs/path/to/queue.rds",
  n_workers           = 4,
  pane_mode           = "killAndNewPane",
  delay_before_source = 60,
  stagger_by          = 60,
  set_mouse           = TRUE
)

# --- Mixed local + remote ---
# Runs 2 workers on localhost and 2 on remote host "sbw".
# .setup_remote_machine("sbw", ...) is called automatically before workers start.
workers <- experimentTmux(
  global_path = "/abs/path/to/global.R",
  queue_path  = "/abs/path/to/queue.rds",
  cores       = c("localhost", "localhost", "sbw", "sbw"),
  pane_mode   = "killAndNewPane",
  email       = "[email protected]",
  cache_path  = "/abs/path/to/.secret",
  ss_id       = "your-google-sheet-id"
)

# --- Tear down all workers ---
tmuxKillPanes(workers)

# --- Restart a single broken pane ---
# In the broken pane, press Up then Enter to re-run the last command.

## End(Not run)

Description

Extracts the "all meaningful combinations" factorial-design logic that used to live inside experiment(). Given a simList plus lists of alternative params / modules / inputs / objects, it returns one row per run. Values are stored as indices into the supplied alternatives (because an alternative may itself be a vector and so cannot live in a single data.frame cell); column names are module.parameter, plus a modules index, an expLevel, and (when relevant) input, object and replicate columns.

Usage

factorialDesign(sim, params, modules, objects = list(), inputs, replicates = 1)

Arguments

Details

This is the engine behind experiment(). It is exported so the same design can also seed the file-queue experiment_family (experimentFuture() etc.): map each row's indices back to values to build their df.

Value

A data.frame, one row per run.

See Also

experiment(), experiment_family

Description

Hard-coded to 1024 – the value compiled into glibc and used by R's socket layer on Linux, macOS, and most BSDs. Not user-configurable without rebuilding R.

Usage

fdSelectLimit()

Value

integer scalar.

Description

Searches from current working directory for and Rstudio project file or git repository, falling back on using the current working directory.

Usage

findProjectPath()

findProjectName()

Value

findProjectPath returns an absolute path; findProjectName returns the basename of the path.

Description

Scans an output directory for files matching the pattern ⁠<file_prefix>_year<XXXX>.<ext>⁠ (e.g. cohortData_year2920.rds) and returns the furthest simulation year reached, the wall-clock elapsed time since the first checkpoint, and a percentage-complete estimate.

Usage

get_sim_year_heartbeat(
  output_path,
  start_year = NULL,
  end_year = NULL,
  file_prefix = "cohortData"
)

Arguments

Value

A named list with elements:

ts

Character. Modification timestamp of the latest checkpoint file.

iter

Integer. Simulation year of the latest checkpoint.

started

Character. Modification timestamp of the first checkpoint file.

elapsed

difftime. Wall-clock time between first and latest checkpoint.

pct_complete

Numeric 0-100. Percentage of the simulation completed, or NA if start_year == end_year.

All elements are NA / NA_character_ when no matching files are found.

Examples

## Not run: 
hb <- get_sim_year_heartbeat(
  output_path = "outputs/6.5/1991-2020/NRV_ssp370/rep1",
  end_year    = 3020L
)
message("Year: ", hb$iter, " (", hb$pct_complete, "%)  -- last checkpoint: ", hb$ts)

## End(Not run)

Description

This can be used within e.g., the options or params arguments for setupProject to get a ready-made file for a project.

Usage

getGithubFile(
  gitRepoFile,
  overwrite = FALSE,
  destDir = ".",
  verbose = getOption("Require.verbose")
)

Arguments

See Also

getModule

Examples

filename <- getGithubFile("PredictiveEcology/LandWeb@development/01b-options.R",
                          destDir = Require::tempdir2())

Description

Simple function to download a SpaDES module as GitHub repository

Usage

getModule(
  modules,
  modulePath,
  overwrite = FALSE,
  verbose = getOption("Require.verbose", 1L)
)

Arguments

See Also

getGithubFile

Description

Given the name or the definition of a class, plus optionally data to be included in the object, new returns an object from that class.

Usage

## S4 method for signature 'simLists'
initialize(.Object, ...)

Arguments

Description

Two modes are available:

Graceful (force = FALSE, default): creates a per-worker sentinel file. Each worker checks for this file between jobs and exits cleanly once its current job finishes. Remaining PENDING jobs stay in the queue and are picked up automatically when experimentFuture is called again with the same queue_path.

Immediate (force = TRUE): sends SIGTERM to each live worker, causing the process to exit as soon as possible. Because callr workers run non-interactively, the process typically exits before R's interrupt handler has a chance to update the queue. Any jobs that were RUNNING at the time of the kill will remain as RUNNING in the queue until the next reclaim pass. Call tmuxRefreshQueueStatus(ef$queue_path) afterwards to reset stale RUNNING entries to INTERRUPTED, or use the GS backend which reclaims dead workers automatically before each new claim.

Usage

killExperimentFuture(ef, force = FALSE)

Arguments

Value

ef, invisibly.

See Also

experimentFuture, awaitExperimentFuture

Description

Graceful (force = FALSE, default): creates per-worker stop files; each worker exits cleanly between jobs once it observes its file. Slurm jobs end normally; remaining PENDING rows stay in the queue and can be resumed with another experimentSBATCH call against the same queue_path.

Usage

killExperimentSBATCH(es, force = FALSE, scancel_cmd = "scancel")

Arguments

Details

Immediate (force = TRUE): runs scancel <ids> to kill the Slurm jobs straight away. Any rows that were RUNNING at the time of cancellation will remain RUNNING in the queue until the next reclaim pass; clean them up with tmuxRefreshQueueStatus(es$queue_path).

Value

es, invisibly.

See Also

experimentSBATCH, awaitExperimentSBATCH

Description

Worker panes started by experimentTmux() / tmuxRunNextWorker() evaluate the user's global.R inside a fresh scenario environment (not .GlobalEnv). When the source call errors, the package captures sys.calls() and stashes it on that scenario env so a post-mortem traceback is still possible without polluting the user's global state. Use this accessor to retrieve it.

Usage

lastTraceback()

Value

A list of calls (as from sys.calls()) suitable for passing to base::traceback(); NULL if no error has been captured in the current session.

Examples

## Not run: 
  # After a worker pane errors:
  traceback(SpaDES.project::lastTraceback())

## End(Not run)

Description

When exploring existing modules, these tools help identify and navigate modules and their interdependencies.

Usage

listModules(
  keywords,
  accounts,
  includeForks = FALSE,
  includeArchived = FALSE,
  excludeStale = TRUE,
  omit = c("fireSense_dataPrepFitRas"),
  purge = FALSE,
  returnList = FALSE,
  verbose = getOption("Require.verbose", 1L)
)

moduleDependencies(
  modules,
  modulePath = getOption("reproducible.modulePath", ".")
)

moduleDependenciesToGraph(md)

PlotModuleGraph(graph)

Arguments

Value

listModules returns a character vector of paste0(account, "/", Repository) for all SpaDES modules in the given repositories with the accounts and keywords provided.

See Also

metadataInModules() helps to see different metadata elements in a folder of modules.

Examples

listModules(accounts = "PredictiveEcology", "none")

Description

Resolves the cluster-facing short name for this host, trying in order:

1. ⁠/etc/hosts⁠ lookup by a local IP (shortest alias wins);

2. ⁠~/.ssh/config⁠ Host entry whose Hostname is a local IP (CRLF-safe);

3. hostname -s.

Usage

localHostLabel()

Details

Useful for deriving the pane-title host prefix when the cluster knows this machine by a name different from hostname -s (e.g. mega, whose raw hostname is the node id but whose cluster alias is mega via ⁠/etc/hosts⁠).

Value

Character(1) short name, or NULL if none could be determined.

Description

Make DESCRIPTION file(s) from SpaDES module metadata

Usage

makeDESCRIPTIONproject(
  modules,
  modulePath,
  projectPath = ".",
  singleDESCRIPTION = TRUE,
  package = "Project",
  title = "Project",
  description = "Project",
  version = "1.0.0",
  authors = Sys.info()["user"],
  write = TRUE,
  verbose = getOption("Require.verbose")
)

makeDESCRIPTION(
  modules,
  modulePath,
  projectPath = ".",
  singleDESCRIPTION = FALSE,
  package,
  title,
  date,
  description,
  version,
  authors,
  write = TRUE,
  verbose,
  metadataList,
  ...
)

Arguments

Description

On Linux, reads ⁠/proc/self/fd⁠ and resolves each symlink to identify what the fd points to. Returns a data.frame with one row per open fd; useful for diagnosing the cryptic "file descriptor is too large for select()" failure from parallelly::makeClusterPSOCK().

Usage

openFds()

Details

Returns an empty data.frame on non-Linux systems or if ⁠/proc/self/fd⁠ is unreadable.

Value

data.frame with columns fd (integer), target (character, with any Linux (deleted) suffix preserved), and bucket (character holder category: "socket", "pipe", "terra scratch", "terra scratch (deleted)", "tif (other)", "vrt", "sqlite", "qs/qs2", "anon_inode", "other file", "unknown").

See Also

openFdsReport() for a printable summary; fdSelectLimit().

Description

Wraps openFds() and returns a multi-line string suitable for warning or error messages. By default summarizes only fds at or above fdSelectLimit() (the select() failure threshold); pass threshold = 0L to summarize everything.

Usage

openFdsReport(threshold = fdSelectLimit())

Arguments

Value

character scalar; "" if ⁠/proc/self/fd⁠ is not available.

Examples

cat(openFdsReport())
cat(openFdsReport(threshold = 0L))

Description

List uploaded scenario output archives.

Usage

outList(folder, pattern = "\\.tar\\.gz$")

Arguments

Value

A dribble of the matching files.

See Also

outScenarios(), queueUploadMissing()

Description

Saves a simList to an RDS file via SpaDES.core::saveSimList(). Heavy ancillary data (inputs, outputs, cache, files) are excluded so the file contains only the simulation state; pair with outTar() to bundle the output files separately.

Usage

outSave(sim, runName, simFilename = NULL, lazy = TRUE)

Arguments

Value

Invisibly returns simFilename.

See Also

outTar(), outUpload(), outSaveTarUpload()

Description

Convenience wrapper that calls outSave(), outTar(), and outUpload() in sequence. The sim is saved to an RDS file, bundled with its output files into a .tar.gz archive, and the archive is uploaded to a Google Drive folder.

Usage

outSaveTarUpload(
  runName,
  sim,
  gFolder = NULL,
  simFilename = NULL,
  tarDir = NULL,
  tarball = NULL,
  overwrite = TRUE,
  cleanup = FALSE,
  verbose = TRUE,
  lazy = TRUE
)

Arguments

Value

Invisibly returns the dribble from googledrive::drive_upload().

See Also

outSave(), outTar(), outUpload()

Description

Uploaded outputs as scenario records.

Usage

outScenarios(folder, pattern = "\\.tar\\.gz$")

Arguments

Value

A list of scenario objects.

See Also

outList(), as_scenario()

Description

Creates a .tar.gz archive containing simFilename and any additional outputFiles. Files that do not exist are silently skipped so a partially completed simulation can still be archived.

Usage

outTar(
  simFilename,
  outputFiles = character(0),
  runName,
  tarDir = dirname(simFilename),
  verbose = TRUE
)

Arguments

Value

Invisibly returns the path to the created tarball.

See Also

outSave(), outUpload(), outSaveTarUpload()

Description

Uploads a local file (typically a tarball produced by outTar()) to a Google Drive folder via googledrive::drive_upload().

Usage

outUpload(tarball, gFolder, overwrite = TRUE, cleanup = FALSE)

Arguments

Value

Invisibly returns the dribble returned by googledrive::drive_upload().

See Also

outSave(), outTar(), outSaveTarUpload()

Description

Parses module code, looking for the metadataItem (default = "reqdPkgs") element in the defineModule function.

Usage

packagesInModules(modules, modulePath = getOption("spades.modulePath"))

metadataInModules(
  modules,
  metadataItem = "reqdPkgs",
  modulePath = getOption("spades.modulePath"),
  needUnlist,
  verbose = getOption("Require.verbose", 1L)
)

Arguments

Value

A character vector of sorted, unique packages that are identified in all named modules, or if modules is omitted, then all modules in modulePath.

Description

Generic format: each non-empty field's value becomes one path segment. Field order is taken from the order of the input (or, for positional calls, from scenarioFields()). Integer-and-contiguous vectors are encoded as start-end. Empty / NA fields are dropped entirely (yielding one fewer segment); for round-tripping see pathParse().

Usage

pathBuild(..., pre = "outputs", withFieldLabel = .scenario_env$withFieldLabel)

Arguments

Details

Fields whose name appears in withFieldLabel get their segment prefixed by the field name itself (e.g. .rep with value 5 renders as .rep5 instead of bare 5). Useful when path readers must distinguish two integer fields, or when round-tripping with mid-list NAs (the label disambiguates which segments are present).

Accepts three calling styles, all equivalent:

• pathBuild(scenarioObj) — a single named-list / scenario;

• pathBuild(.fieldA = vA, .fieldB = vB, ...) — explicit named args;

• pathBuild(vA, vB, ...) — positional, in cached-field order.

Value

Character scalar.

Description

Inverse of the default pathBuild(): splits the path on / (or, for tarname inputs, on ⁠_⁠), strips archive extensions and the pre prefix, then matches segments positionally to scenarioFields(). Integer ranges of the form start-end decode to integer vectors.

Usage

pathParse(
  path,
  fields = scenarioFields(),
  pre = "outputs",
  withFieldLabel = .scenario_env$withFieldLabel
)

Arguments

Details

Without per-segment labels there is no way to recover which field a missing segment corresponds to, so when the path has fewer segments than there are fields, the trailing fields are treated as NA. Round-trip is therefore only lossless when NA-bearing fields are last in the field order unless you label the ambiguous fields through withFieldLabel: any field named there has its label prefix stripped from the segment, and segments not starting with a labeled field's name are assigned positionally to the next unlabeled field. With every potentially-NA field labeled, mid-list NAs round-trip cleanly.

Value

Named list of field values (in fields order).

Description

pkgload::load_all does not automatically deal with dependency chains: the user must manually load the dependency chain in order with separate calls to pkgload::load_all. Also, it does not use caching. This function allows nested caching for a sequence of packages that depend on one another. For example, if a user has 3 packages that have dependency chain: A is a dependency of B which is a dependency of C. If a change happens in C, then pkgload::load_all will only be called on C. If a change happens in A, then pkgload::load_all will be called on A, then B, then C.

Usage

pkgload2(
  depsPaths = file.path("~/GitHub", c("reproducible", "SpaDES.core", "LandR")),
  envir = parent.frame()
)

Arguments

Value

This is called for its side effects, which are 2: pkgload::load_all on the packages that need it, and an object, .prevDigs that is assigned to envir.

Description

Plot all studyArea** and rasterToMatch** objects within a list-like object.

Usage

plotSAs(
  ll,
  ...,
  include = TRUE,
  exclude,
  saCols = c("purple", "blue", "green", "red"),
  title,
  rasterToMatchLabel = "Stand Age",
  rasterToMatchPalette = c("Set1", "Set2", "Set3"),
  country = "CAN",
  latlong = FALSE,
  minArea = 7e+11
)

plotSAsLeaflet(
  ll,
  ...,
  include = TRUE,
  exclude,
  saCols = c("purple", "blue", "green", "red"),
  title = "Study Areas",
  rasterToMatchLabel = "Stand Age",
  rasterToMatchPalette = c("Set1", "Set2", "Set3")
)

Arguments

Value

Run primarily for side effects. plotSAs plots (and returns) a ggplot2 object. plotSAsLeaflet creates a leaflet page in a viewer (if using Rstudio).

Description

preRunSetupProject parses an R script (default: "global.R") and evaluates its contents up to the setupProject() call, either fully or partially based on the upTo argument. This is useful for initializing only certain parts of a project without executing the entire setup.

Usage

preRunSetupProject(file = "global.R", upTo = TRUE, envir = parent.frame())

Arguments

Details

The function:

1. Parses the specified file using parse().

2. Identifies the line where setupProject() is called.

3. Evaluates all code before the setupProject() call.

4. Depending on upTo, evaluates either the full call or a subset of its arguments.

This allows selective initialization of project components for debugging or partial setup in large projects.

Value

The evaluated result of the executed portion of setupProject(). i.e., a list returned by setupProject().

See Also

setupProject

Examples

## Not run: 
# Run file up to and including the setupProject, but only to the 'paths' argument
result <- preRunSetupProject(file = "global.R", upTo = "paths")

# Run file up to and including full setupProject()
result <- preRunSetupProject(file = "global.R", upTo = TRUE)

## End(Not run)

Description

Two call shapes:

Usage

queueRead(folder, name, sheet = NULL, col_types = "c")

Arguments

Details

Local: queueRead("path/to/queue.rds")

When the first argument is an existing local .rds file and name is not supplied, the queue is loaded via readRDS(). Useful for the file-backed queues written by experimentTmux() / experimentFuture() / experimentSBATCH() when no ss_id was supplied.

Google Sheet: queueRead(folder, name)

Convenience wrapper around googledrive::drive_ls() + googlesheets4::read_sheet(). folder is the Drive folder URL/id, name is the spreadsheet name within it.

Either way the result is passed through revertDotNames() so callers see canonical .ELFind/.GCM/... column names rather than the dotELFind/dotGCM/... names Google Sheets forces. As a side effect, the non-meta column names are cached as the active scenario field set (see scenarioFields()).

Value

A data.table. Pipe through as_scenario() for scenario records.

See Also

queueUploadMissing(), outList(), outScenarios(), experimentFuture(), experimentTmux(), experimentSBATCH()

Description

Anti-join of the driver queue against the upload folder's .tar.gz listing, keyed on rendered tarname (see as_tarname()). Independent of the queue's status column.

Usage

queueUploadMissing(folder, name, uploadFolder, ...)

Arguments

Value

Subset of the queue data.table for rows whose expected tarball is not present in uploadFolder.

See Also

queueRead(), outList()

Description

Inverse of outUpload(). Downloads one or more tar.gz archives from a Google Drive folder to a local directory, using reproducible::preProcess() (so re-runs hit the local copy when present). Vectorised: typically called with the multi-row dribble returned by outList() / outScenarios().

Usage

reGet(gFiles, destDir, overwrite = FALSE, verbose = TRUE)

Arguments

Value

A data.table with columns name and local_path, one row per downloaded file.

See Also

reUntar(), reLoad(), reGetUntarLoad(), outUpload()

Description

Convenience wrapper around reGet(), reUntar(), and reLoad() – the inverse of outSaveTarUpload(). Operates on a batch: typically called with the multi-row dribble returned by outList() / outScenarios().

Usage

reGetUntarLoad(
  gFiles,
  destDir,
  pathRemap = NULL,
  projectPath = getwd(),
  method = c("loadSimList", "readRDS"),
  overwrite = FALSE,
  verbose = TRUE
)

Arguments

Value

A named list of simList objects, one per row of gFiles, named by the archive's name (sans .tar.gz).

See Also

reGet(), reUntar(), reLoad(), outSaveTarUpload()

Description

Pass a function (or, for withFieldLabel, a character vector) to register it; pass NULL explicitly to clear that slot; omit the argument to leave it untouched. Call with no arguments to inspect.

Usage

register_scenario_format(build, parse, withFieldLabel)

Arguments

Details

Lookup precedence (highest first): registered slot -> pathBuild/pathParse defined in the global environment -> the package defaults.

Override signature contract: build(..., pre = "outputs") — receives the scenario as named ... args (one per field) plus pre; returns a path string. parse(path, pre = "outputs") — returns a named list of fields.

Value

Invisibly, the current ⁠(build, parse, withFieldLabel)⁠ triple.

Description

Inverse of outSave(). Loads one or more simLists from .rds files produced by outSave(). Defaults to SpaDES.core::loadSimList(); set method = "readRDS" to bypass .unwrap entirely.

Note that SpaDES.core::saveSimList() uses .wrapResiliently to NULL out file-backed objects with inaccessible backing files at save time. Load-time failures (e.g. backing files missing on this machine even though they were present at save time) are independent of that, and are handled by loadSimList's pre-.unwrap resilient pass.

Usage

reLoad(
  simFilenames,
  projectPath = getwd(),
  method = c("loadSimList", "readRDS"),
  ...
)

Arguments

Value

A list of simList objects, named by basename(simFilenames).

See Also

reGet(), reUntar(), reGetUntarLoad(), outSave()

Description

Inverse of outTar(). Extracts one or more .tar.gz archives produced by outTar() / outSaveTarUpload(), which contain absolute paths. If pathRemap is supplied, the leading path prefix is rewritten on extraction (handy when the archive was created on another user's machine, e.g. paths starting with ⁠/home/emcintir/...⁠).

Path rewriting uses GNU tar's --transform. On systems without GNU tar, supply pathRemap = NULL and the archive's absolute paths are restored as-is.

Usage

reUntar(tarballs, pathRemap = NULL, verbose = FALSE)

Arguments

Value

A character vector (same length as tarballs) of absolute paths to the .rds simList file inside each archive (after any remap), suitable for reLoad().

See Also

reGet(), reLoad(), reGetUntarLoad(), outTar()

Description

A thin wrapper around tmuxRunWorkerLoop that optionally redirects console output to a log file before entering the job loop. Used internally by experimentFuture for remote (cluster) workers. Local workers use callr::r_bg() and do not need this wrapper.

Usage

runWorkerLoopFuture(
  queue_path,
  global_path,
  on_interrupt = c("requeue", "fail"),
  ss_id = NULL,
  email = NULL,
  cache_path = NULL,
  runNameLabel = quote(colnames(q)[1:2]),
  activeRunningPath = NULL,
  dots_path = NULL,
  stop_file = NULL,
  log_file = NULL
)

Arguments

Value

Invisibly returns the worker identifier string.

Description

Accepts any named arguments. Each name becomes a field label; each value the field's value. No specific field set is required by the package (fields are project-defined via the queue).

Usage

scenario(...)

Arguments

Details

Light coercion: a single character of the form "a:b" is evaled as an R expression (so queue cells like "1991:2020" become integer vectors).

Value

An S3 object of class "scenario".

Description

A "scenario" identifies a single simulation run. Field names and values are discovered from the driver queue (Google Sheet); they are not hardcoded in this package. The same run can be referred to in three interchangeable ways:

Details

1. Field values (one column per field in the queue), e.g. ⁠(.ELFind = "6.3.1", .samplingRange = 2071:2100, ...)⁠.

2. An output directory path under ⁠outputs/⁠.

3. An upload tar filename (path with / -> ⁠_⁠ and .tar.gz suffix).

This file defines:

• a canonical record (S3 class "scenario");

• the generic as_scenario() for coercing any representation into it;

• formatters as_path() / as_tarname() for going back;

• default builders pathBuild() / pathParse(): each non-empty field's value (no label) is one path segment, joined by /, in the order given by scenarioFields(). Integer-and-contiguous vectors render as start-end. Empty / NA fields are skipped entirely (one fewer segment); see pathParse() for the trailing-NA round-trip caveat.

Per-project format overrides: define your own pathBuild (and matching pathParse) in the global environment, or register them explicitly with register_scenario_format(). Lookup order, highest first: register_scenario_format slot -> a pathBuild/pathParse in the global environment -> the package default.

Field discovery: queueRead() caches the queue's non-meta column names as the active field set. Subsequent pathParse() calls use those labels for positional decoding. If you parse paths without first reading a queue, pass fields = c(...) explicitly (or call scenarioFieldsSet()).

Examples

## Not run: 
## --- Default (generic) format -----------------------------------------
queue <- queueRead(folder = ss_id, name = "longRuns")
#  -> data.table with columns .ELFind, .samplingRange, .GCM, .SSP, .rep
#     plus meta columns (status, started_at, ...). Non-meta columns are
#     auto-cached as scenarioFields().

scens <- as_scenario(queue)                # list of `scenario` objects
as_path(scens[[1]])
#> "outputs/6.3.1/2071-2100/CNRM-ESM2-1/370/5"
as_tarname(scens[[1]])
#> "6.3.1_2071-2100_CNRM-ESM2-1_370_5.tar.gz"

# Round-trip
s2 <- as_scenario("outputs/6.3.1/2071-2100/CNRM-ESM2-1/370/5")
identical(unclass(scens[[1]]), unclass(s2))   # TRUE

# Cross-reference queue against uploaded tarballs
uploads <- outScenarios(.uploadGSdir)         # list of scenarios
missing <- queueUploadMissing(folder = ss_id, name = "longRuns",
                              uploadFolder = .uploadGSdir)  # queue rows only

## --- Per-field labels in the path -------------------------------------
# `withFieldLabel` accepts two forms.

# 1) Unnamed character vector: prefix with the field name itself.
as_path(scens[[1]], withFieldLabel = c(".rep", ".SSP"))
#> "outputs/6.3.1/2071-2100/CNRM-ESM2-1/.SSP370/.rep5"

# 2) Named character vector: prefix with the *mapped* label
#    (e.g., emit `.rep` as `rep`, `.SSP` as `_ssp`).
as_path(scens[[1]], withFieldLabel = c(.rep = "rep", .SSP = "_ssp"))
#> "outputs/6.3.1/2071-2100/CNRM-ESM2-1/_ssp370/rep5"

# Set once for every subsequent as_path() / as_tarname():
register_scenario_format(withFieldLabel = c(.rep = "rep", .SSP = "_ssp"))
as_path(scens[[1]])
#> "outputs/6.3.1/2071-2100/CNRM-ESM2-1/_ssp370/rep5"
as_tarname(scens[[1]])
#> "6.3.1_2071-2100_CNRM-ESM2-1__ssp370_rep5.tar.gz"
# Round-trip parses back to canonical fields:
as_scenario("outputs/6.3.1/2071-2100/CNRM-ESM2-1/_ssp370/rep5")

## --- Project-specific format (FireSenseTesting layout) ----------------
# Layout: outputs/<.ELFind>/<range>/<GCM>_ssp<SSP>/rep<.rep>
# E.g.    outputs/6.3.1/2071-2100/CNRM-ESM2-1_ssp370/rep5

myBuild <- function(.ELFind, .samplingRange, .GCM, .SSP, .rep,
                    pre = "outputs") {
  sr <- if (is.numeric(.samplingRange)) .samplingRange
        else                            eval(parse(text = .samplingRange))
  file.path(pre, .ELFind,
            paste(range(sr), collapse = "-"),
            paste0(.GCM, ifelse(is.na(.SSP), "", paste0("_ssp", .SSP))),
            paste0("rep", .rep))
}

myParse <- function(path, fields = scenarioFields(), pre = "outputs") {
  clean <- sub("\\.tar\\.gz$", "", path)
  clean <- sub(paste0("^", pre, "[/_]"), "", clean)
  parts <- if (grepl("/", clean)) strsplit(clean, "/")[[1L]]
           else                    strsplit(clean, "_")[[1L]]
  repIdx   <- which(grepl("^rep[0-9]+$",   parts))
  rangeIdx <- which(grepl("^[0-9]+-[0-9]+$", parts))
  gcmSsp   <- paste(parts[(rangeIdx + 1L):(repIdx - 1L)], collapse = "_")
  gs       <- if (grepl("_ssp", gcmSsp)) strsplit(gcmSsp, "_ssp")[[1L]]
              else                       c(gcmSsp, NA_character_)
  rng      <- as.integer(strsplit(parts[rangeIdx], "-")[[1L]])
  list(.ELFind        = paste(parts[seq_len(rangeIdx - 1L)], collapse = "_"),
       .samplingRange = rng[1L]:rng[2L],
       .GCM           = gs[1L],
       .SSP           = gs[2L],
       .rep           = as.integer(sub("^rep", "", parts[repIdx])))
}

register_scenario_format(build = myBuild, parse = myParse)
as_path(scens[[1]])
#> "outputs/6.3.1/2071-2100/CNRM-ESM2-1_ssp370/rep5"
as_tarname(scens[[1]])
#> "6.3.1_2071-2100_CNRM-ESM2-1_ssp370_rep5.tar.gz"

# Equivalent: define pathBuild / pathParse in your global environment
# (e.g. in a project global.R) -- they will be auto-detected.
pathBuild <- myBuild
pathParse <- myParse

## End(Not run)

Description

Returns the field labels currently used to (a) parse paths/tarnames back into scenario records and (b) determine which queue columns constitute the scenario (vs. queue meta-columns). Set automatically by queueRead(); can be set manually with scenarioFieldsSet().

Usage

scenarioFields()

scenarioFieldsSet(fields)

Arguments

Value

Character vector of field names, or NULL if not yet known.

Description

This function will create a sub-folder of the lib.loc directory that is based on the R version and the platform, as per the standard R package directory naming convention

Usage

setProjPkgDir(lib.loc = "packages", verbose = getOption("Require.verbose", 1L))

Arguments

Description

Convenience helper, intended primarily for interactive use, that parses each file (local path or github.com URL with ⁠@branch⁠ notation) into a named list.

Usage

setupFiles(
  files,
  paths,
  envir = parent.frame(),
  verbose = getOption("Require.verbose", 1L)
)

Arguments

Details

setupFiles is a convenience function intended for interactive use to verify the files being parsed. This is similar to parse, but each element must be a named list or a named object, such as a function. It uses the same specification for https://github.com files as setupProject, i.e., using @ for branch.

setupFiles("PredictiveEcology/PredictiveEcology.org@main/tutos/castorExample/params.R")

Value

setupFiles a named list with each element that was parsed.

See Also

setupProject() for the high-level wrapper, setup_family for an overview.

Description

Source the functions supplied to setupProject() so they are available to subsequent ⁠setup*⁠ steps and to the user's session.

Usage

setupFunctions(
  functions,
  name,
  sideEffects,
  paths,
  overwrite = FALSE,
  envir = parent.frame(),
  callingEnv = sys.frame(-2),
  verbose = getOption("Require.verbose", 1L),
  dots,
  defaultDots,
  ...
)

Arguments

Details

setupFunctions will source the functions supplied, with a parent environment being the internal temporary environment of the setupProject, i.e., they will have access to all the objects in the call.

Value

setupFunctions returns NULL. All functions will be placed in envir.

See Also

setupProject() for the high-level wrapper, setup_family for an overview.

Examples

## simplest case; just creates folders
out <- setupProject(
  paths = list(projectPath = ".") #
)
# specifying functions argument, with a local file and a definition here
tf <- tempfile(fileext = ".R")
fnDefs <- c("fn <- function(x) x\n",
            "fn2 <- function(x) x\n",
            "fn3 <- function(x) terra::rast(x)")
cat(text = fnDefs, file = tf)
funHere <- function(y) y
out <- setupProject(functions = list(a = function(x) return(x),
                                     tf,
                                     funHere = funHere), # have to name it
                    # now use the functions when creating objects
                    drr = 1,
                    b = a(drr),
                    q = funHere(22),
                    ddd = fn3(terra::ext(0,b,0,b)))

Description

Helper that keeps the .gitignore of a project under git control in sync with the project's resolved paths.

Usage

setupGitIgnore(
  paths,
  gitignore = getOption("SpaDES.project.gitignore", TRUE),
  verbose
)

Arguments

Details

setupGitIgnore will add the relevant paths to .gitignore.

Value

setupGitIgnore is run for its side effects, i.e., adding either paths$packagePath and/or paths$modulePath to the .gitignore file. It will check whether packagePath is located inside the paths$projectPath and will add this folder to the .gitignore if TRUE. If the project is a git repository with git submodules, then it will add nothing else. If the project is a git repository without git submodules, then the paths$modulePath will be added to the .gitignore file. It is assumed that these modules are used in a ⁠read only⁠ manner.

See Also

setupProject() for the high-level wrapper, setup_family for an overview.

Description

Materialise the modules requested in setupProject() beneath paths[["modulePath"]], optionally as git submodules.

Usage

setupModules(
  name,
  paths,
  modules,
  inProject,
  useGit = getOption("SpaDES.project.useGit", FALSE),
  overwrite = FALSE,
  envir = parent.frame(),
  callingEnv = sys.frame(-2),
  gitUserName,
  verbose = getOption("Require.verbose", 1L),
  dots,
  defaultDots,
  updateRprofile = getOption("SpaDES.project.updateRprofile", TRUE),
  ...
)

Arguments