Package 'reproducible'

The reproducible package

Description

This package aims at making high-level, robust, machine and OS independent tools for making deeply reproducible and reusable content in R. The core user functions are Cache and prepInputs. Each of these is built around many core and edge cases required to have reproducible code of arbitrary complexity.

Main Tools

There are many elements within the reproducible package. However, there are currently two main ones that are critical for reproducible research. The key element for reproducible research is that the code must always return the same content every time it is run, but it must be vastly faster the 2nd, 3rd, 4th etc, time it is run. That way, the entire code sequence for a project of arbitrary size can be run from the start every time.

Cache():

A robust wrapper for any function, including those with environments, disk-backed storage (currently on Raster) class), operating-system independent, whose first time called will execute the function, second time will compare the inputs to a database of entries, and recover the first result if inputs are identical. If options("reproducible.useMemoise" = TRUE), the second time will be very fast as it will recover the answer from RAM.

prepInputs()for other specifics for other classes.:

Download, or load objects, and possibly post-process them. The main advantage to using this over more direct routes is that it will automatically build checksums tables, use Cache internally where helpful, and possibly run a variety of post-processing actions. This means this function can also itself be cached for even more speed. This allows all project data to be stored in custom cloud locations or in their original online data repositories, ⁠without altering code⁠ between the first, second, third, etc., times the code is run.

Package options

See reproducibleOptions() for a complete description of package options() to configure behaviour.

Author(s)

Maintainer: Eliot J B McIntire [email protected] (ORCID)

Authors:

• Eliot J B McIntire [email protected] (ORCID)

• Alex M Chubaty [email protected] (ORCID)

Other contributors:

• Tati Micheletti [email protected] (ORCID) [contributor]

• Ceres Barros [email protected] (ORCID) [contributor]

• Ian Eddy [email protected] (ORCID) [contributor]

• His Majesty the King in Right of Canada, as represented by the Minister of Natural Resources Canada [copyright holder]

See Also

Useful links:

• https://reproducible.predictiveecology.org

• https://github.com/PredictiveEcology/reproducible

• Report bugs at https://github.com/PredictiveEcology/reproducible/issues

---

Add a Tag to a Cached Object in the Repository

Description

This hidden function appends a single tag (key-value pair) to the metadata of a cached object identified by its cacheId. Tags can be stored either in a database (via DBI) or in a file-based cache system.

Updates the value of an existing tag for a cached object identified by its cacheId. If the tag does not exist and add = TRUE, the tag will be added. This function supports both database-backed and file-based cache systems.

Usage

.addTagsRepo(
  cacheId,
  cachePath = getOption("reproducible.cachePath"),
  tagKey = character(),
  tagValue = character(),
  cacheSaveFormat = getOption("reproducible.cacheSaveFormat"),
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  verbose = getOption("reproducible.verbose")
)

.updateTagsRepo(
  cacheId,
  cachePath = getOption("reproducible.cachePath"),
  tagKey = character(),
  tagValue = character(),
  add = TRUE,
  cacheSaveFormat = getOption("reproducible.cacheSaveFormat"),
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  verbose = getOption("reproducible.verbose")
)

Arguments

cacheId	character(1) Unique identifier of the cached object. Must be of length 1.
cachePath	character(1) Path to the cache directory. Defaults to getOption("reproducible.cachePath").
tagKey	character(1) The key for the tag. Must be supplied.
tagValue	character(1) The new value for the tag. Must be supplied.
cacheSaveFormat	character(1) Format used for saving cache files. Defaults to getOption("reproducible.cacheSaveFormat").
drv	A DBI driver object. Defaults to getDrv(getOption("reproducible.drv", NULL)).
conn	A DBI connection object. If NULL, a new connection is created internally.
verbose	integer or logical. If ⁠> 0⁠ or TRUE, print informational messages. Defaults to getOption("reproducible.verbose").
add	logical(1) If TRUE, adds the tag if it does not exist. Defaults to TRUE.

Details

This function is primarily used internally by the reproducible package to maintain metadata about cached objects. It supports both database-backed and file-based caching systems.

• If useDBI() returns TRUE, the tag update is performed in the database table.

• If no rows are affected and add = TRUE, the tag is inserted using .addTagsRepo().

• For file-based caches, the function modifies the tag in the corresponding metadata file.

Value

NULL (invisibly). The function is called for its side effects.

NULL (invisibly). Called for its side effects.

See Also

.addTagsRepo() for adding tags without updating.

Examples

a <- Cache(rnorm(1))
.addTagsRepo(cacheId = gsub("cacheId:", "", attr(a, "tags")),
             tagKey = "status", tagValue = "processed")
showCache() # last entry is the above line

a <- Cache(rnorm(1))
# Update an existing tag
.updateTagsRepo(cacheId = gsub("cacheId:", "", attr(a, "tags")),
             tagKey = "status", tagValue = "second")

# Add a tag if it doesn't exist
.updateTagsRepo(cacheId = gsub("cacheId:", "", attr(a, "tags")),
             tagKey = "status", tagValue = "new", add = TRUE)

---

Attach debug info to return for Cache

Description

Internal use only. Attaches an attribute to the output, usable for debugging the Cache.

Usage

.debugCache(obj, preDigest, ..., fullCall)

Arguments

obj	An arbitrary R object.
preDigest	A list of hashes.
...	Dots passed from Cache
fullCall	The original call to Cache

Value

The same object as obj, but with 2 attributes set.

Author(s)

Eliot McIntire

---

Move a file to a new location – Defunct – use hardLinkOrCopy

Description

This will first try to file.rename, and if that fails, then it will file.copy then file.remove.

Usage

.file.move(from, to, overwrite = FALSE)

Arguments

from, to	character vectors, containing file names or paths.
overwrite	logical indicating whether to overwrite destination file if it exists.

Value

Logical indicating whether operation succeeded.

---

Deal with class for saving to and loading from Cache or Disk

Description

This generic and some methods will do whatever is required to prepare an object for saving to disk (or RAM) via e.g., saveRDS. Some objects (e.g., terra's ⁠Spat*⁠) cannot be saved without first wrapping them. Also, file-backed objects are similar.

Usage

.filebackedPath(
  filebackedPath,
  filebackedPathMissing,
  cachePath,
  cachePathMissing
)

.wrap(
  obj,
  filebackedPath = getOption("reproducible.cachePath"),
  preDigest,
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  verbose = getOption("reproducible.verbose"),
  outputObjects = NULL,
  cacheId = NULL,
  ...,
  cachePath
)

## S3 method for class 'list'
.wrap(
  obj,
  filebackedPath = getOption("reproducible.cachePath"),
  preDigest,
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  verbose = getOption("reproducible.verbose"),
  outputObjects = NULL,
  cacheId = NULL,
  ...,
  cachePath
)

## S3 method for class 'environment'
.wrap(
  obj,
  filebackedPath = getOption("reproducible.cachePath"),
  preDigest,
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  verbose = getOption("reproducible.verbose"),
  outputObjects = NULL,
  cacheId = NULL,
  ...,
  cachePath
)

## Default S3 method:
.wrap(
  obj,
  filebackedPath = getOption("reproducible.cachePath"),
  preDigest,
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  verbose = getOption("reproducible.verbose"),
  outputObjects = NULL,
  cacheId = NULL,
  ...,
  cachePath
)

## Default S3 method:
.unwrap(
  obj,
  filebackedPath = getOption("reproducible.cachePath"),
  cacheId = NULL,
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  ...,
  cachePath
)

.unwrap(
  obj,
  filebackedPath = getOption("reproducible.cachePath"),
  cacheId = NULL,
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  ...,
  cachePath
)

## S3 method for class 'environment'
.unwrap(
  obj,
  filebackedPath = getOption("reproducible.cachePath"),
  cacheId = NULL,
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  ...,
  cachePath
)

## S3 method for class 'list'
.unwrap(
  obj,
  filebackedPath = getOption("reproducible.cachePath"),
  cacheId = NULL,
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  ...,
  cachePath
)

## S3 method for class 'PackedSpatExtent2'
.unwrap(
  obj,
  filebackedPath = getOption("reproducible.cachePath"),
  cacheId = NULL,
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  ...,
  cachePath
)

## S3 method for class 'PackedSpatVector2'
.unwrap(
  obj,
  filebackedPath = getOption("reproducible.cachePath"),
  cacheId = NULL,
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  ...,
  cachePath
)

## S3 method for class 'data.table'
.unwrap(
  obj,
  filebackedPath = getOption("reproducible.cachePath"),
  cacheId = NULL,
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  ...,
  cachePath
)

## S3 method for class 'PackedSpatVector'
.unwrap(
  obj,
  filebackedPath = getOption("reproducible.cachePath"),
  cacheId = NULL,
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  ...,
  cachePath
)

Arguments

filebackedPath	Directory under which file-backed objects (e.g., a file-backed SpatRaster) are anchored and, with copyFiles = TRUE, stored (in ⁠cacheOutputs/⁠). Cache() uses its cachePath; outside of Cache() it is any directory, e.g., the one holding an .rds.
filebackedPathMissing, cachePathMissing	missing() of the two arguments in the calling method.
cachePath	Deprecated; the former name of filebackedPath. It still works, with a message.
obj	Any arbitrary R object.
preDigest	The list of preDigest that comes from CacheDigest of an object
drv	If using a database backend, drv must be an object that inherits from DBIDriver (e.g., RSQLite::SQLite).
conn	an optional DBIConnection object, as returned by dbConnect().
verbose	Numeric, -1 silent (where possible), 0 being very quiet, 1 showing more messaging, 2 being more messaging, etc. Default is 1. Above 3 will output much more information about the internals of Caching, which may help diagnose Caching challenges. Can set globally with an option, e.g., ⁠options('reproducible.verbose' = 0) to reduce to minimal⁠
outputObjects	Optional character vector indicating which objects to return. This is only relevant for list, environment (or similar) objects
cacheId	Used strictly for messaging. This should be the cacheId of the object being recovered. Default is NULL.
...	Arguments passed to methods. For a file-backed SpatRaster, copyFiles = TRUE copies the backing file(s) into ⁠cacheOutputs/⁠ under filebackedPath (as Cache() does), so the wrapped object can be saved with saveRDS and restored by .unwrap(obj, filebackedPath) even if the original file is gone. The default FALSE only records the file's location.

Value

Returns an object that can be saved to disk e.g., via saveRDS.

Examples

# For SpatExtent
if (requireNamespace("terra")) {
  ex <- terra::ext(c(0, 2, 0, 3))
  exWrapped <- .wrap(ex)
  ex1 <- .unwrap(exWrapped)
}

---

Some spatial helper functions

Description

Some spatial helper functions

Usage

.isGridded(x)

.isVector(x)

.isSF(x)

.isSpat(x)

.isSpatialAny(x)

.isCRSany(x)

Arguments

x	A spatial object.

Details

.isGridded returns TRUE if the object is a SpatRaster or Raster

.isVector returns TRUE if the object is SpatVector, spatial or sf

.isSF returns TRUE if the object is sf or sfc

.isSpat returns TRUE if the object is SpatVector or SpatRaster

.isSpatialAny returns TRUE if the object returns TRUE for .isGridded or .isVector

Value

Logical.

---

Evaluate whether a cacheId is memoised

Description

Intended for internal use. Exported so other packages can use this function.

Usage

.isMemoised(cacheId, cachePath = getOption("reproducible.cachePath"))

Arguments

cacheId

Character string. If passed, this will override the calculated hash of the inputs, and return the result from this cacheId in the cachePath. Setting this is equivalent to manually saving the output of this function, i.e., the object will be on disk, and will be recovered in subsequent This may help in some particularly finicky situations where Cache is not correctly detecting unchanged inputs. This will guarantee the object will be identical each time; this may be useful in operational code.

cachePath

A repository used for storing cached objects. This is optional if Cache is used inside a SpaDES module.

Value

A logical, length 1 indicating whether the cacheId is memoised.

---

lobstr::obj_size with a try to address issue #72

Description

It is not clear why, but it appears that running lobstr::obj_size again, after a bad binding error, it will work.

Usage

.objSizeWithTry(x, useTry = TRUE)

Arguments

x	An object
useTry	Logical. If TRUE, the default, then it will use try. Can optionally avoid this if set to FALSE. The try takes sufficient extra compute time that it is worth avoiding it if possible.

Value

The size of an object, using lobstr::obj_size or object.size if the first fails

---

Add a prefix or suffix to the basename part of a file path

Description

Prepend (or postpend) a filename with a prefix (or suffix). If the directory name of the file cannot be ascertained from its path, it is assumed to be in the current working directory.

Usage

.prefix(f, prefix = "")

.suffix(f, suffix = "")

Arguments

f	A character string giving the name/path of a file.
prefix	A character string to prepend to the filename.
suffix	A character string to postpend to the filename.

Value

A character string or vector with the prefix pre-pended or suffix post-pended on the basename of the f, before the file extension.

Author(s)

Jean Marchal and Alex Chubaty

Examples

# file's full path is specified (i.e., dirname is known)
myFile <- file.path("~/data", "file.tif")
.prefix(myFile, "small_") ## "/home/username/data/small_file.tif"
.suffix(myFile, "_cropped") ## "/home/username/data/myFile_cropped.shp"

# file's full path is not specified
.prefix("myFile.shp", "small") ## "./small_myFile.shp"
.suffix("myFile.shp", "_cropped") ## "./myFile_cropped.shp"

---

Copy the file-backing of a file-backed Raster* object

Description

Rasters are sometimes file-based, so the normal save and copy and assign mechanisms in R don't work for saving, copying and assigning. This function creates an explicit file copy of the file that is backing the raster, and changes the pointer (i.e., filename(object)) so that it is pointing to the new file.

Usage

.prepareFileBackedRaster(
  obj,
  repoDir = NULL,
  overwrite = FALSE,
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  verbose = getOption("reproducible.verbose"),
  ...
)

Arguments

obj	The raster object to save to the repository.
repoDir	Character denoting an existing directory in which an artifact will be saved.
overwrite	Logical. Should the raster be saved to disk, overwriting existing file.
drv	If using a database backend, drv must be an object that inherits from DBIDriver (e.g., RSQLite::SQLite).
conn	an optional DBIConnection object, as returned by dbConnect().
verbose	Numeric, -1 silent (where possible), 0 being very quiet, 1 showing more messaging, 2 being more messaging, etc. Default is 1. Above 3 will output much more information about the internals of Caching, which may help diagnose Caching challenges. Can set globally with an option, e.g., ⁠options('reproducible.verbose' = 0) to reduce to minimal⁠
...	Not used

Value

A raster object and its newly located file backing. Note that if this is a legitimate Cache repository, the new location will be a subdirectory called ‘rasters/’ of ‘repoDir/’. If this is not a repository, the new location will be within repoDir.

Author(s)

Eliot McIntire

---

Remove attributes that are highly varying

Description

Remove attributes that are highly varying

Usage

.removeCacheAtts(x)

Arguments

x	Any arbitrary R object that could have attributes

---

Provide standard messaging for missing package dependencies

Description

This provides a standard message format for missing packages, e.g., detected via requireNamespace.

Usage

.requireNamespace(
  pkg = "methods",
  minVersion = NULL,
  stopOnFALSE = FALSE,
  messageStart = NULL
)

Arguments

pkg	Character string indicating name of package required
minVersion	Character string indicating minimum version of package that is needed
stopOnFALSE	Logical. If TRUE, this function will create an error (i.e., stop) if the function returns FALSE; otherwise it simply returns FALSE
messageStart	A character string with a prefix of message to provide

Value

A logical or stop if the namespace is not available to be loaded.

---

Set subattributes within a list by reference

Description

Sets only a single element within a list attribute.

Usage

.setSubAttrInList(object, attr, subAttr, value)

Arguments

object	An arbitrary object
attr	The attribute name (that is a list object) to change
subAttr	The list element name to change
value	The new value

Value

This sets or updates the subAttr element of a list that is located at attr(object, attr), with the value. This, therefore, updates a sub-element of a list attribute and returns that same object with the updated attribute.

---

Search for objects in the call stack

Description

Normally, this is only used in special, advanced uses. The standard approach to getting an object from an environment in the call stack is to explicitly pass it into the function.

Usage

.whereInStack(obj, startingEnv = parent.frame())

Arguments

obj	Character string. The object name to search.
startingEnv	An environment to start searching in.

Value

The environment in which the object exists. It will return the first environment it finds, searching outwards from where the function is used.

---

Assess the appropriate raster layer data type

Description

When writing raster-type objects to disk, a datatype can be specified. These functions help identify what smallest datatype can be used.

Usage

assessDataType(ras, type = "writeRaster")

## Default S3 method:
assessDataType(ras, type = "writeRaster")

Arguments

ras	The RasterLayer or RasterStack for which data type will be assessed.
type	Character. "writeRaster" (default) or "GDAL" (defunct) to return the recommended data type for writing from the raster packages, respectively, or "projectRaster" to return recommended resampling type.

Value

A character string indicating the data type of the spatial layer (e.g., "INT2U"). See terra::datatype()

Examples

if (requireNamespace("terra", quietly = TRUE)) {
  ## LOG1S
  rasOrig <- terra::rast(ncols = 10, nrows = 10)
  ras <- rasOrig
  ras[] <- rep(c(0,1),50)
  assessDataType(ras)

  ras <- rasOrig
  ras[] <- rep(c(0,1),50)
  assessDataType(ras)

  ras[] <- rep(c(TRUE,FALSE),50)
  assessDataType(ras)

  ras[] <- c(NA, NA, rep(c(0,1),49))
  assessDataType(ras)

  ras <- rasOrig
  ras[] <- c(0, NaN, rep(c(0,1),49))
  assessDataType(ras)


  ## INT1S
  ras[] <- -1:98
  assessDataType(ras)

  ras[] <- c(NA, -1:97)
  assessDataType(ras)

  ## INT1U
  ras <- rasOrig
  ras[] <- 1:100
  assessDataType(ras)

  ras[] <- c(NA, 2:100)
  assessDataType(ras)

  ## INT2U
  ras <- rasOrig
  ras[] <- round(runif(100, min = 64000, max = 65000))
  assessDataType(ras)

  ## INT2S
  ras <- rasOrig
  ras[] <- round(runif(100, min = -32767, max = 32767))
  assessDataType(ras)

  ras[54] <- NA
  assessDataType(ras)

  ## INT4U
  ras <- rasOrig
  ras[] <- round(runif(100, min = 0, max = 500000000))
  assessDataType(ras)

  ras[14] <- NA
  assessDataType(ras)

  ## INT4S
  ras <- rasOrig
  ras[] <- round(runif(100, min = -200000000, max = 200000000))
  assessDataType(ras)

  ras[14] <- NA
  assessDataType(ras)

  ## FLT4S
  ras <- rasOrig
  ras[] <- runif(100, min = -10, max = 87)
  assessDataType(ras)

  ras <- rasOrig
  ras[] <- round(runif(100, min = -3.4e+26, max = 3.4e+28))
  assessDataType(ras)

  ras <- rasOrig
  ras[] <- round(runif(100, min = 3.4e+26, max = 3.4e+28))
  assessDataType(ras)

  ras <- rasOrig
  ras[] <- round(runif(100, min = -3.4e+26, max = -1))
  assessDataType(ras)

  ## FLT8S
  ras <- rasOrig
  ras[] <- c(-Inf, 1, rep(c(0,1),49))
  assessDataType(ras)

  ras <- rasOrig
  ras[] <- c(Inf, 1, rep(c(0,1),49))
  assessDataType(ras)

  ras <- rasOrig
  ras[] <- round(runif(100, min = -1.7e+30, max = 1.7e+308))
  assessDataType(ras)

  ras <- rasOrig
  ras[] <- round(runif(100, min = 1.7e+30, max = 1.7e+308))
  assessDataType(ras)

  ras <- rasOrig
  ras[] <- round(runif(100, min = -1.7e+308, max = -1))
  assessDataType(ras)

  # 2 layer with different types LOG1S and FLT8S
  ras <- rasOrig
  ras[] <- rep(c(0,1),50)
  ras1 <- rasOrig
  ras1[] <- round(runif(100, min = -1.7e+308, max = -1))
  sta <- c(ras, ras1)
  assessDataType(sta)

}

---

A version of base::basename that is NULL resistant

Description

A version of base::basename that is NULL resistant

Usage

basename2(x)

Arguments

x	A character vector of paths

Value

NULL if x is NULL, otherwise, as basename.

Same as base::basename()

Examples

basename2(NULL)
basename2("/home/user/file.txt")

---

Saves a wide variety function call outputs to disk and optionally RAM, for recovery later

Description

A function that can be used to wrap around other functions to cache function calls for later use. This is normally most effective when the function to cache is slow to run, yet the inputs and outputs are small. The benefit of caching, therefore, will decline when the computational time of the "first" function call is fast and/or the argument values and return objects are large. The default setting (and first call to Cache) will always save to disk. The 2nd call to the same function will return from disk, unless options("reproducible.useMemoise" = TRUE), then the 2nd time will recover the object from RAM and is normally much faster (at the expense of RAM use).

Usage

Cache(
  FUN,
  ...,
  dryRun = getOption("reproducible.dryRun", FALSE),
  notOlderThan = NULL,
  .objects = NULL,
  .cacheExtra = NULL,
  .functionName = NULL,
  .cacheChaining = getOption("reproducible.cacheChaining", NULL),
  outputObjects = NULL,
  algo = "xxhash64",
  cachePath = NULL,
  length = getOption("reproducible.length", Inf),
  userTags = c(),
  omitArgs = NULL,
  classOptions = list(),
  debugCache = character(),
  quick = getOption("reproducible.quick", FALSE),
  verbose = getOption("reproducible.verbose", 1),
  cacheId = NULL,
  cacheSaveFormat = getOption("reproducible.cacheSaveFormat"),
  useCache = getOption("reproducible.useCache", TRUE),
  useCloud = getOption("reproducible.useCloud", FALSE),
  cloudFolderID = getOption("reproducible.cloudFolderID", NULL),
  showSimilar = getOption("reproducible.showSimilar", FALSE),
  drv = getOption("reproducible.drv", NULL),
  conn = getOption("reproducible.conn", NULL)
)

cache2(
  FUN,
  ...,
  dryRun = getOption("reproducible.dryRun", FALSE),
  notOlderThan = NULL,
  .objects = NULL,
  .cacheExtra = NULL,
  .functionName = NULL,
  .cacheChaining = getOption("reproducible.cacheChaining", NULL),
  outputObjects = NULL,
  algo = "xxhash64",
  cachePath = NULL,
  length = getOption("reproducible.length", Inf),
  userTags = c(),
  omitArgs = NULL,
  classOptions = list(),
  debugCache = character(),
  quick = getOption("reproducible.quick", FALSE),
  verbose = getOption("reproducible.verbose", 1),
  cacheId = NULL,
  cacheSaveFormat = getOption("reproducible.cacheSaveFormat"),
  useCache = getOption("reproducible.useCache", TRUE),
  useCloud = getOption("reproducible.useCloud", FALSE),
  cloudFolderID = getOption("reproducible.cloudFolderID", NULL),
  showSimilar = getOption("reproducible.showSimilar", FALSE),
  drv = getOption("reproducible.drv", NULL),
  conn = getOption("reproducible.conn", NULL)
)

CacheV2(
  FUN,
  ...,
  notOlderThan = NULL,
  .objects = NULL,
  .cacheExtra = NULL,
  .functionName = NULL,
  outputObjects = NULL,
  algo = "xxhash64",
  cacheRepo = NULL,
  cachePath = NULL,
  length = getOption("reproducible.length", Inf),
  compareRasterFileLength,
  userTags = c(),
  omitArgs = NULL,
  classOptions = list(),
  debugCache = character(),
  makeCopy = FALSE,
  quick = getOption("reproducible.quick", FALSE),
  verbose = getOption("reproducible.verbose", 1),
  cacheId = NULL,
  useCache = getOption("reproducible.useCache", TRUE),
  useCloud = FALSE,
  cloudFolderID = NULL,
  showSimilar = getOption("reproducible.showSimilar", FALSE),
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL)
)

Arguments

FUN	Either a function (e.g., rnorm), a function call (e.g., rnorm(1)), or an unevaluated function call (e.g., using quote()).
...	Arguments passed to FUN, if FUN is not an expression.
dryRun	Logical. If TRUE, digest the arguments and report on the closest previous call, but neither evaluate FUN nor write anything. Returns, invisibly, the element-by-element digest of the call that would have run; pass it to whyNoCacheHit() to be told which element differs from the closest previous call, before paying for the miss. See also reproducibleOptions.
notOlderThan	A time. Load an object from the Cache if it was created after this.
.objects	Character vector of objects to be digested. This is only applicable if there is a list, environment (or similar) with named objects within it. Only this/these objects will be considered for caching, i.e., only use a subset of the list, environment or similar objects. In the case of nested list-type objects, this will only be applied outermost first.
.cacheExtra	A an arbitrary R object that will be included in the CacheDigest, but otherwise not passed into the FUN. If the user supplies a named list, then Cache will report which individual elements of .cacheExtra have changed when options("reproducible.showSimilar" = TRUE). This can allow a user more control and understanding for debugging.
.functionName	A an arbitrary character string that provides a name that is different than the actual function name (e.g., "rnorm") which will be used for messaging. This can be useful when the actual function is not helpful for a user, such as do.call.
.cacheChaining	A logical or a the name of a function. If TRUE, then the current Cache call will evaluate the function "outside" the Cache call (via sys.function(-1)) and attach the digest of that outer function to the entry for this Cache call. This will then be used by any subsequent Cache call within the same function. If the outer function is unchanged, and there is one or more objects that had been returned by a previous Cache call, then those objects will not be digested; rather their cacheId tag will be used in place of a new digest. This should cause no change in Caching outcomes, and it should be faster in cases where there are several Cache calls within the same function. If FALSE (current default), then this feature is not used. If set to NULL (i.e., unset, the current default), then it will not use cache chaining, but it will attach more information to the Cache entries for each cacheId, as well as new entries for "surroundingFunction" digest, so that if a user switches to .cacheChaining = TRUE, then it will be able to begin using cache chaining without needing to rerun the calls again. Can be set by an option.
outputObjects	Optional character vector indicating which objects to return. This is only relevant for list, environment (or similar) objects
algo	The digest algorithm to use. Default xxhash64 (see digest::digest() for others).
cachePath	A repository used for storing cached objects. This is optional if Cache is used inside a SpaDES module.
length	Numeric. If the element passed to Cache is a Path class object (from e.g., asPath(filename)) or it is a Raster with file-backing, then this will be passed to digest::digest, essentially limiting the number of bytes to digest (for speed). This will only be used if quick = FALSE. Default is getOption("reproducible.length"), which is set to Inf.
userTags	A character vector with descriptions of the Cache function call. These will be added to the Cache so that this entry in the Cache can be found using userTags e.g., via showCache().
omitArgs	Optional. A character vector of argument names in FUN to omit from the cache digest, or TRUE to omit every captured argument (the digest is then based on FUN itself – including its body, so a meaningful edit to the function source still busts the cache – and on .cacheExtra). Useful when the developer wants the cache to be insensitive to the function's inputs and pin freshness via .cacheExtra instead.
classOptions	Optional list. This will pass into .robustDigest for specific classes. Should be options that the .robustDigest knows what to do with.
debugCache	Character or Logical. Either "complete" or "quick" (uses partial matching, so "c" or "q" work). TRUE is equivalent to "complete". If "complete", then the returned object from the Cache function will have two attributes, debugCache1 and debugCache2, which are the entire list(...) and that same object, but after all .robustDigest calls, at the moment that it is digested using digest, respectively. This attr(mySimOut, "debugCache2") can then be compared to a subsequent call and individual items within the object attr(mySimOut, "debugCache1") can be compared. If "quick", then it will return the same two objects directly, without evalutating the FUN(...).
quick	Logical or character. If TRUE, no disk-based information will be assessed, i.e., only memory content. See Details section about quick in Cache().
verbose	Numeric, -1 silent (where possible), 0 being very quiet, 1 showing more messaging, 2 being more messaging, etc. Default is 1. Above 3 will output much more information about the internals of Caching, which may help diagnose Caching challenges. Can set globally with an option, e.g., ⁠options('reproducible.verbose' = 0) to reduce to minimal⁠
cacheId	Character string. If passed, this will override the calculated hash of the inputs, and return the result from this cacheId in the cachePath. Setting this is equivalent to manually saving the output of this function, i.e., the object will be on disk, and will be recovered in subsequent This may help in some particularly finicky situations where Cache is not correctly detecting unchanged inputs. This will guarantee the object will be identical each time; this may be useful in operational code.
cacheSaveFormat	Character string: currently either qs or rds. Defaults to getOption("reproducible.cacheSaveFormat"). qs may be faster but appears to have narrower range of conditions that work; rds is safer, and may be slower.
useCache	Logical, numeric or "overwrite", "devMode" or "always". See details.
useCloud	Logical (TRUE / FALSE / NULL) or one of "pull" / "push". See Details.
cloudFolderID	A googledrive dribble of a folder, e.g., using drive_mkdir(). If left as NULL, the function will create a cloud folder with name from last two folder levels of the cachePath path, : paste0(basename(dirname(cachePath)), "_", basename(cachePath)). This cloudFolderID will be added to options("reproducible.cloudFolderID"), but this will not persist across sessions. If this is a character string, it will treat this as a folder name to create or use on GoogleDrive.
showSimilar	A logical or numeric. Useful for debugging. If TRUE or 1, then if the Cache does not find an identical archive in the cachePath, it will report (via message) the next most recent similar archive, and indicate which argument(s) is/are different. If a number larger than 1, then it will report the N most recent similar archived objects.
drv	If using a database backend, drv must be an object that inherits from DBIDriver (e.g., RSQLite::SQLite).
conn	an optional DBIConnection object, as returned by dbConnect().
cacheRepo	Same as cachePath, but kept for backwards compatibility.
compareRasterFileLength	Being deprecated; use length.
makeCopy	Now deprecated. Ignored if used.

Details

There are other similar functions in the R universe. This version of Cache has been used as part of a robust continuous workflow approach. As a result, we have tested it with many "non-standard" R objects (e.g., RasterLayer, ⁠Spat*⁠ objects) and environments (which are always unique, so do not cache readily).

This version of the Cache function accommodates those four special, though quite common, cases by:

1. converting any environments into list equivalents;

2. identifying the dispatched S4 method (including those made through inheritance) before hashing so the correct method is being cached;

3. by hashing the linked file, rather than the raster object. Currently, only file-backed ⁠Raster*⁠ or ⁠Spat*⁠ objects are digested (e.g., not ff objects, or any other R object where the data are on disk instead of in RAM);

4. Uses digest::digest() This is used for file-backed objects as well.

5. Cache will save arguments passed by user in a hidden environment. Any nested Cache functions will use arguments in this order: 1) actual arguments passed at each Cache call; 2) any inherited arguments from an outer Cache call; 3) the default values of the Cache function. See section on Nested Caching.

The algorithm used to compute the cacheId (the hash of the inputs) is selectable via the reproducible.digestVersion option; see the digestVersion entry in reproducibleOptions() for the available versions and what each changes (notably version 4, which makes sf/SpatVector digests identical across operating systems).

Cache will add a tag to the entry in the cache database called accessed, which will assign the time that it was accessed, either read or write. That way, cached items can be shown (using showCache) or removed (using clearCache) selectively, based on their access dates, rather than only by their creation dates. See example in clearCache().

Value

Returns the value of the function call or the cached version (i.e., the result from a previous call to this same cached function with identical arguments).

Nested Caching

Commonly, Caching is nested, i.e., an outer function is wrapped in a Cache function call, and one or more inner functions are also wrapped in a Cache function call. A user can always specify arguments in every Cache function call, but this can get tedious and can be prone to errors. The normal way that R handles arguments is it takes the user passed arguments if any, and default arguments for all those that have no user passed arguments. We have inserted a middle step. The order or precedence for any given Cache function call is

1. user arguments, 2. inherited arguments, 3. default arguments. At this time, the top level Cache arguments will propagate to all inner functions unless each individual Cache call has other arguments specified, i.e., "middle" nested Cache function calls don't propagate their arguments to further "inner" Cache function calls. See example.

userTags is unique of all arguments: its values will be appended to the inherited userTags.

quick

The quick argument is attempting to sort out an ambiguity with character strings: are they file paths or are they simply character strings. When quick = TRUE, Cache will treat these as character strings; when quick = FALSE, they will be attempted to be treated as file paths first; if there is no file, then it will revert to treating them as character strings. If user passes a character vector to this, then this will behave like omitArgs: quick = "file" will treat the argument "file" as character string.

The most often encountered situation where this ambiguity matters is in arguments about filenames: is the filename an input pointing to an object whose content we want to assess (e.g., a file-backed raster), or an output (as in saveRDS) and it should not be assessed. If only run once, the output file won't exist, so it will be treated as a character string. However, once the function has been run once, the output file will exist, and Cache(...) will assess it, which is incorrect. In these cases, the user is advised to use quick = "TheOutputFilenameArgument" to specify the argument whose content on disk should not be assessed, but whose character string should be assessed (distinguishing it from omitArgs = "TheOutputFilenameArgument", which will not assess the file content nor the character string).

This is relevant for objects of class character, Path and Raster currently. For class character, it is ambiguous whether this represents a character string or a vector of file paths. If it is known that character strings should not be treated as paths, then quick = TRUE is appropriate, with no loss of information. If it is file or directory, then it will digest the file content, or basename(object). For class Path objects, the file's metadata (i.e., filename and file size) will be hashed instead of the file contents if quick = TRUE. If set to FALSE (default), the contents of the file(s) are hashed. If quick = TRUE, length is ignored. Raster objects are treated as paths, if they are file-backed.

Caching Speed

Caching speed may become a critical aspect of a final product. For example, if the final product is a shiny app, rerunning the entire project may need to take less then a few seconds at most. There are 3 arguments that affect Cache speed: quick, length, and algo. quick is passed to .robustDigest, which currently only affects Path and ⁠Raster*⁠ class objects. In both cases, quick means that little or no disk-based information will be assessed.

Filepaths

If a function has a path argument, there is some ambiguity about what should be done. Possibilities include:

1. hash the string as is (this will be very system specific, meaning a Cache call will not work if copied between systems or directories);

2. hash the basename(path);

3. hash the contents of the file.

If paths are passed in as is (i.e,. character string), the result will not be predictable. Instead, one should use the wrapper function asPath(path), which sets the class of the string to a Path, and one should decide whether one wants to digest the content of the file (using quick = FALSE), or just the filename ((quick = TRUE)). See examples.

Stochasticity or randomness

In general, it is expected that caching will only be used when randomness is not desired, e.g., Cache(rnorm(1)) is unlikely to be useful in many cases. However, Cache captures the call that is passed to it, leaving all functions unevaluated. As a result Cache(glm, x ~ y, rnorm(1)) will not work as a means of forcing a new evaluation each time, as the rnorm(1) is not evaluated before the call is assessed against the cache database. To force a new call each time, evaluate the randomness prior to the Cache call, e.g., ran = rnorm(1) then pass this to .cacheExtra, e.g., Cache(glm, x ~ y, .cacheExtra = ran)

drv and conn

By default, drv uses an SQLite database. This can be sufficient for most cases. However, if a user has dozens or more cores making requests to the Cache database, it may be insufficient. A user can set up a different database backend, e.g., PostgreSQL that can handle multiple simultaneous read-write situations. See https://github.com/PredictiveEcology/SpaDES/wiki/Using-alternate-database-backends-for-Cache.

useCache

Logical or numeric. If FALSE or 0, then the entire Caching mechanism is bypassed and the function is evaluated as if it was not being Cached. Default is getOption("reproducible.useCache")), which is TRUE by default, meaning use the Cache mechanism. This may be useful to turn all Caching on or off in very complex scripts and nested functions. Increasing levels of numeric values will cause deeper levels of Caching to occur (though this may not work as expected in all cases). The following is no longer supported: Currently, only implemented in postProcess: to do both caching of inner cropInputs, projectInputs and maskInputs, and caching of outer postProcess, use useCache = 2; to skip the inner sequence of 3 functions, use useCache = 1. For large objects, this may prevent many duplicated save to disk events.

If useCache = "overwrite" (which can be set with options("reproducible.useCache" = "overwrite")), then the function invoke the caching mechanism but will purge any entry that is matched, and it will be replaced with the results of the current call.

If useCache = "devMode": The point of this mode is to facilitate using the Cache when functions and datasets are continually in flux, and old Cache entries are likely stale very often. In devMode, the cache mechanism will work as normal if the Cache call is the first time for a function OR if it successfully finds a copy in the cache based on the normal Cache mechanism. It differs from the normal Cache if the Cache call does not find a copy in the cachePath, but it does find an entry that matches based on userTags. In this case, it will delete the old entry in the cachePath (identified based on matching userTags), then continue with normal Cache. For this to work correctly, userTags must be unique for each function call. This should be used with caution as it is still experimental. Currently, if userTags are not unique to a single entry in the cachePath, it will default to the behaviour of useCache = TRUE with a message. This means that "devMode" is most useful if used from the start of a project.

If useCache = "always", the call is cached even when options(reproducible.useCache = FALSE) (as SpaDES.core sets it under options(spades.useCache = "eventsOnly")) and even when it is nested inside a Cache() call whose useCache is FALSE. Otherwise it behaves exactly as TRUE: the same key, so it hits entries made with TRUE and vice versa. It is meant for a package's own calls whose repetition is known to be expensive. There is no option to turn it off; to recompute, delete the entry (e.g. clearCache()). It applies to that call only: a Cache() nested inside it keeps its own useCache, and an inner TRUE inherits TRUE, not "always". Spelled out in full; it is not matched by abbreviation.

useCloud

This is experimental and there are many conditions under which this is known to not work correctly. This is a way to store all or some of the local Cache in the cloud. Currently, the only cloud option is Google Drive, via googledrive. For this to work, the user must be or be able to be authenticated with googledrive::drive_auth. The principle behind this useCloud is that it will be a full or partial mirror of a local Cache. It is not intended to be used independently from a local Cache. To share objects that are in the Cloud with another person, it requires 2 steps. 1) share the cloudFolderID$id, which can be retrieved by getOption("reproducible.cloudFolderID")$id after at least one Cache call has been made. 2) The other user must then set their cacheFolderID in a ⁠Cache\(..., reproducible.cloudFolderID = \"the ID here\"\)⁠ call or set their option manually ⁠options\(\"reproducible.cloudFolderID\" = \"the ID here\"\)⁠.

If TRUE, then this Cache call will download (if local copy doesn't exist, but cloud copy does exist), upload (local copy does or doesn't exist and cloud copy doesn't exist), or will not download nor upload if object exists in both. If TRUE will be at least 1 second slower than setting this to FALSE, and likely even slower as the cloud folder gets large. If a user wishes to keep "high-level" control, set this to getOption("reproducible.useCloud", FALSE) or getOption("reproducible.useCloud", TRUE) (if the default behaviour should be FALSE or TRUE, respectively) so it can be turned on and off with this option. NOTE: This argument will not be passed into inner/nested Cache calls.)

Two character values are also accepted, intended for separating developer and user roles when sharing a cloud-cache folder:

• "push" is equivalent to TRUE (developer role) – bidirectional; downloads on a cloud hit, uploads on a miss.

• "pull" is read-only (user role) – downloads on a cloud hit, but never uploads. If the local cache already has the object, the cloud is not consulted at all (the Google Drive listing is deferred until after the local lookup fails). When neither local nor cloud has the object, the call falls back to a normal local-only Cache run.

Reading does not need a Google sign-in when the folder is shared "Anyone with the link" and cloudFolderID is its id or link. Google Drive is reached as in prepInputs(): a token already loaded, else a configured identity (gargle_oauth_email, a service-account file in GOOGLEDRIVE_AUTH), else anonymously. With no identity, cloud hits are downloaded as usual, and a miss is computed and saved locally only: nothing is uploaded and no folder is created, and one message says so (useCloud then acts as "pull"). An interactive session, and a folder given by name, keep googledrive's own sign-in.

Object attributes

Users should be cautioned that object attributes may not be preserved, especially in the case of objects that are file-backed, such as Raster or SpatRaster objects. If a user needs to keep attributes, they may need to manually re-attach them to the object after recovery. With the example of SpatRaster objects, saving to disk requires terra::wrap if it is a memory-backed object. When running terra::unwrap on this object, any attributes that a user had added are lost.

sideEffect

This feature is now deprecated. Do not use as it is ignored.

Note

As indicated above, several objects require pre-treatment before caching will work as expected. The function .robustDigest accommodates this. It is an S4 generic, meaning that developers can produce their own methods for different classes of objects. Currently, there are methods for several types of classes. See .robustDigest().

Author(s)

Eliot McIntire

See Also

showCache(), clearCache(), keepCache(), CacheDigest() to determine the digest of a given function or expression, as used internally within Cache, movedCache(), .robustDigest(), reproducibleOptions() (e.g. the digestVersion option that selects the cacheId algorithm), and for more advanced uses there are several helper functions, e.g., rmFromCache(), CacheStorageDir()

Examples

data.table::setDTthreads(2)
tmpDir <- tempdir()
opts <- options(reproducible.cachePath = tmpDir)

# Usage -- All below are equivalent; even where args are missing or provided,
#   Cache evaluates using default values, if these are specified in formals(FUN)
a <- list()
b <- list(fun = rnorm)
bbb <- 1
ee <- new.env(parent = emptyenv())
ee$qq <- bbb

a[[1]] <- Cache(rnorm(1)) # no evaluation prior to Cache
a[[2]] <- Cache(rnorm, 1) # no evaluation prior to Cache
a[[3]] <- Cache(do.call, rnorm, list(1))
a[[4]] <- Cache(do.call(rnorm, list(1)))
a[[5]] <- Cache(do.call(b$fun, list(1)))
a[[6]] <- Cache(do.call, b$fun, list(1))
a[[7]] <- Cache(b$fun, 1)
a[[8]] <- Cache(b$fun(1))
a[[10]] <- Cache(quote(rnorm(1)))
a[[11]] <- Cache(stats::rnorm(1))
a[[12]] <- Cache(stats::rnorm, 1)
a[[13]] <- Cache(rnorm(1, 0, get("bbb", inherits = FALSE)))
a[[14]] <- Cache(rnorm(1, 0, get("qq", inherits = FALSE, envir = ee)))
a[[15]] <- Cache(rnorm(1, bbb - bbb, get("bbb", inherits = FALSE)))
a[[16]] <- Cache(rnorm(sd = 1, 0, n = get("bbb", inherits = FALSE))) # change order
a[[17]] <- Cache(rnorm(1, sd = get("ee", inherits = FALSE)$qq), mean = 0)

# with base pipe -- this is put in quotes ('') because R version 4.0 can't understand this
#  if you are using R >= 4.1 or R >= 4.2 if using the _ placeholder,
#  then you can just use pipe normally
usingPipe1 <- "b$fun(1) |> Cache()"  # base pipe

# For long pipe, need to wrap sequence in { }, or else only last step is cached
usingPipe2 <-
  '{"bbb" |>
      parse(text = _) |>
      eval() |>
      rnorm()} |>
    Cache()'
a[[9]] <- eval(parse(text = usingPipe1)) # recovers cached copy
a[[18]] <- eval(parse(text = usingPipe2)) # recovers cached copy

length(unique(a)) == 1 #  all same

### Pipe -- have to use { } or else only final function is Cached
b1a <- 'sample(1e5, 1) |> rnorm() |> Cache()'
b1b <- 'sample(1e5, 1) |> rnorm() |> Cache()'
b2a <- '{sample(1e5, 1) |> rnorm()} |> Cache()'
b2b <- '{sample(1e5, 1) |> rnorm()} |> Cache()'
b1a <- eval(parse(text = b1a))
b1b <- eval(parse(text = b1b))
b2a <- eval(parse(text = b2a))
b2b <- eval(parse(text = b2b))
all.equal(b1a, b1b) # Not TRUE because the sample is run first
all.equal(b2a, b2b) # TRUE because of {  }, sample is not run

#########################
# Advanced examples
#########################

# .cacheExtra -- add something to digest
Cache(rnorm(1), .cacheExtra = "sfessee11") # adds something other than fn args
Cache(rnorm(1), .cacheExtra = "nothing") # even though fn is same, the extra is different

# omitArgs -- remove something from digest (kind of the opposite of .cacheExtra)
Cache(rnorm(2, sd = 1), omitArgs = "sd") # removes one or more args from cache digest
Cache(rnorm(2, sd = 2), omitArgs = "sd") # b/c sd is not used, this is same as previous

# cacheId -- force the use of a digest -- can give undesired consequences
Cache(rnorm(3), cacheId = "k323431232") # sets the cacheId for this call
Cache(runif(14), cacheId = "k323431232") # recovers same as above, i.e, rnorm(3)

# Turn off Caching session-wide
opts <- options(reproducible.useCache = FALSE)
Cache(rnorm(3)) # doesn't cache
options(opts)

# showSimilar can help with debugging why a Cache call isn't picking up a cached copy
Cache(rnorm(4), showSimilar = TRUE) # shows that the argument `n` is different

###############################################
# devMode -- enables cache database to stay
#            small even when developing code
###############################################
opt <- options("reproducible.useCache" = "devMode")
clearCache(tmpDir, ask = FALSE)
centralTendency <- function(x) {
  mean(x)
}
funnyData <- c(1, 1, 1, 1, 10)
uniqueUserTags <- c("thisIsUnique", "reallyUnique")
ranNumsB <- Cache(centralTendency, funnyData, cachePath = tmpDir,
                  userTags = uniqueUserTags) # sets new value to Cache
showCache(tmpDir) # 1 unique cacheId -- cacheId is 71cd24ec3b0d0cac

# During development, we often redefine function internals
centralTendency <- function(x) {
  median(x)
}
# When we rerun, we don't want to keep the "old" cache because the function will
#   never again be defined that way. Here, because of userTags being the same,
#   it will replace the entry in the Cache, effetively overwriting it, even though
#   it has a different cacheId
ranNumsD <- Cache(centralTendency, funnyData, cachePath = tmpDir, userTags = uniqueUserTags)
showCache(tmpDir) # 1 unique artifact -- cacheId is 632cd06f30e111be

# If it finds it by cacheID, doesn't matter what the userTags are
ranNumsD <- Cache(centralTendency, funnyData, cachePath = tmpDir, userTags = "thisIsUnique")
options(opt)

#########################################
# For more in depth uses, see vignette
if (interactive())
  browseVignettes(package = "reproducible")

---

The exact digest function that Cache uses

Description

This can be used by a user to pre-test their arguments before running Cache, for example to determine whether there is a cached copy.

Usage

CacheDigest(
  objsToDigest,
  ...,
  algo = "xxhash64",
  calledFrom = "CacheDigest",
  .functionName = NULL,
  quick = FALSE
)

Arguments

objsToDigest	A list of all the objects (e.g., arguments) to be digested
...	passed to .robustDigest.
algo	The digest algorithm to use. Default xxhash64 (see digest::digest() for others).
calledFrom	a Character string, length 1, with the function to compare with. Default is "Cache". All other values may not produce robust CacheDigest results.
.functionName	A an arbitrary character string that provides a name that is different than the actual function name (e.g., "rnorm") which will be used for messaging. This can be useful when the actual function is not helpful for a user, such as do.call.
quick	Logical or character. If TRUE, no disk-based information will be assessed, i.e., only memory content. See Details section about quick in Cache().

Value

A list of length 2 with the outputHash, which is the digest that Cache uses for cacheId and also preDigest, which is the digest of each sub-element in objsToDigest.

Examples

data.table::setDTthreads(2)
a <- Cache(rnorm, 1)

# like with Cache, user can pass function and args in a few ways
CacheDigest(rnorm(1)) # shows same cacheId as previous line
CacheDigest(rnorm, 1) # shows same cacheId as previous line

---

A ledger of spatial rows, shared through a folder or Google Drive

Description

A ledger is one .rds file of spatial rows: a data.frame with a geometry column (and any other columns, including list-columns such as fitted parameters or models). The functions here describe a ledger, read the rows that overlap an area, add rows, and run a computation only for an area the ledger does not yet cover.

• CacheGeoLedger() describes a ledger. It does no input or output.

• CacheGeoRead() returns the rows that match an area, as an sf object in the ledger's CRS (0 rows if none match). It never writes, never creates a folder and never uploads. If the ledger has a remote, the file is fetched only when its md5 differs from the local copy's. Reading a file on Google Drive that is shared "Anyone with the link" needs no Google login.

• CacheGeoWrite() adds rows. mode = "upsert" replaces the rows that have the same key (or, with no key, the same geometry); mode = "append" always adds. It reads the remote again just before writing, writes, pushes only if the content changed, and checks the push. If another writer changed the remote in between, it merges again, up to 3 times. The whole write is held under a file lock, so writers on one machine (or, on a shared-disk folder, on every machine that mounts it) take turns. Writing to Google Drive needs a Google login (see googledrive::drive_auth()).

• CacheGeo() reads; if the area is not covered by the ledger it calls compute(area), writes the result and returns the rows. An error in compute is an error in CacheGeo().

Usage

CacheGeoLedger(
  file,
  remote = NULL,
  destinationPath = getOption("reproducible.destinationPath", "."),
  key = NULL,
  remoteType = .cacheGeoRemoteTypes
)

CacheGeoRead(
  ledger,
  area,
  match = .cacheGeoMatches,
  tolerance = 0,
  verbose = getOption("reproducible.verbose", 1)
)

CacheGeoWrite(
  ledger,
  rows,
  mode = .cacheGeoModes,
  verbose = getOption("reproducible.verbose", 1)
)

CacheGeo(
  ...,
  ledger,
  area,
  compute,
  match = .cacheGeoMatches,
  tolerance = 0,
  verbose = getOption("reproducible.verbose", 1),
  targetFile = NULL,
  domain,
  FUN,
  destinationPath = getOption("reproducible.destinationPath", "."),
  useCloud = getOption("reproducible.useCloud", FALSE),
  cloudFolderID = NULL,
  purge = FALSE,
  useCache = getOption("reproducible.useCache"),
  overwrite = getOption("reproducible.overwrite"),
  action = c("nothing", "update", "replace", "append"),
  bufferOK = FALSE
)

Arguments

file	The ledger's file name, e.g. "fireSenseParams.rds".
remote	NULL (a local ledger), a folder on a shared disk, a Google Drive folder (URL, id, or a dribble), or a read-only URL.
destinationPath	The local folder that holds the ledger file (or its local copy).
key	The name of the column that identifies a row, e.g. "polygonID". NULL for a ledger without one.
remoteType	How to treat remote. "auto" decides from remote: a Google Drive URL or id is "drive", another http(s) URL is "url", anything else is "disk". A folder name on Google Drive has to be given as "drive".
ledger	A CacheGeoLedger object.
area	The area of interest: an sf, sfc or SpatVector object, in any CRS (it is transformed to the ledger's). NULL asks for the whole ledger and has to be given explicitly.
match	Which rows to return for the area. "intersects" (the default): every row whose polygon overlaps the area. "covers": the overlapping rows, but only when together they cover the area (otherwise none). "within": the rows whose polygon lies inside the area.
tolerance	A distance, in the ledger's map units (degrees for a longitude/latitude CRS). A row whose overlap with the area is gone after shrinking it inward by tolerance is a sliver and is not returned; each one is named in the message. For "covers", gaps that narrow are ignored. For "within", the area is widened by tolerance.
verbose	0 is silent, 1 (default) gives one short message per call, 2 also prints the rows returned.
rows	The rows to add: an sf or SpatVector object, or a data.frame with a geometry column. They are transformed to the ledger's CRS.
mode	"upsert" (default) or "append"; see above.
...	In CacheGeo(), the named objects that the old FUN call uses. Unnamed arguments are ledger, area, compute, match, tolerance and verbose, in that order. ... comes first so that an object named e.g. le is not partially matched to ledger; the arguments after it must be named in full.
compute	A function of one argument, the area, that returns the rows for it (an sf object or a data.frame with a geometry column).
targetFile, domain, FUN, cloudFolderID, useCloud, action, bufferOK, purge, useCache, overwrite	The old arguments of CacheGeo(); see above.

Value

CacheGeoLedger() a CacheGeoLedger object. CacheGeoRead() and CacheGeo() an sf object in the ledger's CRS, possibly with 0 rows. CacheGeoWrite() the whole ledger as an sf object, invisibly.

Remotes

remote can be a folder on a shared disk (e.g. "/mnt/fast/ledgers"), a Google Drive folder (a drive.google.com URL or a folder id; the folder is created only by CacheGeoWrite()), or a read-only URL (the ledger is remote/file; CacheGeoWrite() stops). With no remote the ledger is the local file only.

To share a ledger through Google Drive, create a folder, share it "Anyone with the link" as a viewer, and give the folder's URL as remote to everyone. Only the people who write need a Google login.

Old arguments of CacheGeo()

targetFile, domain, FUN (an unevaluated call that may use domain and the named objects in ...), cloudFolderID, useCloud, action, bufferOK, purge, useCache and overwrite are still accepted, with one message per session. action = "nothing" writes nothing (it computes FUN only when there is no ledger file yet, and returns its rows), "update" always calls FUN and replaces the rows with the same polygonID (or the same geometry), "append" adds. Unless match is given, the rows are chosen as the old CacheGeo() chose them: every row that intersects domain (one that only touches it included), and none unless together they cover it; with bufferOK = TRUE a gap is closed by buffering the rows by 10 km. As before, the call returns NULL when it finds no rows and writes nothing, and the whole ledger after a write. purge, useCache and overwrite are ignored. A targetFile that is not .rds is written as .rds.

Examples

square <- function(id, x0, x1) {
  box <- sf::st_polygon(list(rbind(c(x0, 0), c(x1, 0), c(x1, 10), c(x0, 10), c(x0, 0))))
  out <- sf::st_sf(polygonID = id, geometry = sf::st_sfc(box, crs = 32618))
  out$params <- I(list(list(fit = id)))
  out
}
led <- CacheGeoLedger("params.rds", destinationPath = tempdir2("cacheGeoEx"), key = "polygonID")

# write: one row per key; writing the same key again replaces the row
CacheGeoWrite(led, rbind(square("A", 0, 10), square("B", 10, 20)))
CacheGeoWrite(led, square("A", 0, 10))

# read: the rows that overlap an area; always an sf object, 0 rows if none
area <- sf::st_as_sf(sf::st_sfc(sf::st_point(c(12, 5)), crs = 32618) |> sf::st_buffer(2))
CacheGeoRead(led, area)$polygonID
CacheGeoRead(led, area, match = "covers")$polygonID

# read-or-compute: compute() runs only for an area the ledger does not cover
area2 <- sf::st_as_sf(sf::st_sfc(sf::st_point(c(25, 5)), crs = 32618) |> sf::st_buffer(2))
rows <- CacheGeo(led, area2, compute = function(area) square("C", 20, 30))
rows$polygonID

---

Extract the cache id of an object

Description

Any object that was returned from the Cache or was calculated as part of a Cache call will have an attribute, tags and an entry with ⁠cacheId:⁠ prefix. This is a lightweight helper to extract that cacheId.

Usage

cacheId(obj)

Arguments

obj	Any R object

Value

The cacheId if this was part of a Cache call. Otherwise NULL

Examples

# An object that was never Cached has no cacheId
cacheId(1:10)

---

Check directory path

Description

Checks the specified path to a directory for formatting consistencies, such as trailing slashes, etc.

Usage

checkPath(path, create)

## S4 method for signature 'character,logical'
checkPath(path, create)

## S4 method for signature 'character,missing'
checkPath(path)

## S4 method for signature 'NULL,ANY'
checkPath(path)

## S4 method for signature 'missing,ANY'
checkPath()

Arguments

path	A character string corresponding to a directory path.
create	A logical indicating whether the path should be created if it does not exist. Default is FALSE.

Value

Character string denoting the cleaned up filepath.

Note

This will not work for paths to files. To check for existence of files, use file.exists(). To normalize a path to a file, use normPath() or normalizePath().

See Also

file.exists(), dir.create(), normPath()

Examples

## normalize file paths
paths <- list("./aaa/zzz",
              "./aaa/zzz/",
              ".//aaa//zzz",
              ".//aaa//zzz/",
              ".\\\\aaa\\\\zzz",
              ".\\\\aaa\\\\zzz\\\\",
              file.path(".", "aaa", "zzz"))

checked <- normPath(paths)
length(unique(checked)) ## 1; all of the above are equivalent

## check to see if a path exists
tmpdir <- file.path(tempdir(), "example_checkPath")

dir.exists(tmpdir) ## FALSE
tryCatch(checkPath(tmpdir, create = FALSE), error = function(e) FALSE) ## FALSE

checkPath(tmpdir, create = TRUE)
dir.exists(tmpdir) ## TRUE

unlink(tmpdir, recursive = TRUE)

---

An alternative to basename and dirname when there are sub-folders

Description

This confirms that the files which may be absolute actually exist when compared makeRelative(knownRelativeFiles, absolutePrefix). This is different than just using basename because it will include any sub-folder structure within the knownRelativePaths

Usage

checkRelative(
  files,
  absolutePrefix,
  knownRelativeFiles,
  verbose = getOption("reproducible.verbose")
)

Arguments

files	A character vector of files to check to see if they are the same as knownRelativeFiles, once the absolutePrefix is removed
absolutePrefix	A directory to "remove" from files to compare to knownRelativeFiles
knownRelativeFiles	A character vector of relative filenames, that could have sub-folder structure.
verbose	Numeric, -1 silent (where possible), 0 being very quiet, 1 showing more messaging, 2 being more messaging, etc. Default is 1. Above 3 will output much more information about the internals of Caching, which may help diagnose Caching challenges. Can set globally with an option, e.g., ⁠options('reproducible.verbose' = 0) to reduce to minimal⁠

---

Calculate checksum

Description

Verify (and optionally write) checksums. Checksums are computed using .digest(), which is simply a wrapper around digest::digest.

Usage

Checksums(
  path,
  write,
  quickCheck = getOption("reproducible.quickCheck", FALSE),
  checksumFile = identifyCHECKSUMStxtFile(path),
  files = NULL,
  verbose = getOption("reproducible.verbose", 1),
  ...
)

## S4 method for signature 'character,logical'
Checksums(
  path,
  write,
  quickCheck = getOption("reproducible.quickCheck", FALSE),
  checksumFile = identifyCHECKSUMStxtFile(path),
  files = NULL,
  verbose = getOption("reproducible.verbose", 1),
  ...
)

## S4 method for signature 'character,missing'
Checksums(
  path,
  write,
  quickCheck = getOption("reproducible.quickCheck", FALSE),
  checksumFile = identifyCHECKSUMStxtFile(path),
  files = NULL,
  verbose = getOption("reproducible.verbose", 1),
  ...
)

Arguments

path	Character string giving the directory path containing CHECKSUMS.txt file, or where it will be written if checksumFile = TRUE.
write	Logical indicating whether to overwrite CHECKSUMS.txt. Default is FALSE, as users should not change this file. Module developers should write this file prior to distributing their module code, and update accordingly when the data change.
quickCheck	Logical. If TRUE, then this will only use file sizes, rather than a digest::digest hash. This is generally faster, but will be much less robust.
checksumFile	The filename of the checksums file to read or write to. The default is ‘CHECKSUMS.txt’ located at file.path(path, module, "data", checksumFile). It is likely not a good idea to change this, and should only be used in cases such as Cache, which can evaluate if the checksumFile has changed.
files	An optional character string or vector of specific files to checksum. This may be very important if there are many files listed in a CHECKSUMS.txt file, but only a few are to be checksummed.
verbose	Numeric, -1 silent (where possible), 0 being very quiet, 1 showing more messaging, 2 being more messaging, etc. Default is 1. Above 3 will output much more information about the internals of Caching, which may help diagnose Caching challenges. Can set globally with an option, e.g., ⁠options('reproducible.verbose' = 0) to reduce to minimal⁠
...	Passed to digest::digest() and utils::write.table(). For digest, the notable argument is algo. For write.table, the notable argument is append.

Value

A data.table with columns: result, expectedFile, actualFile, checksum.x, checksum.y, algorithm.x, algorithm.y, filesize.x, filesize.y indicating the result of comparison between local file (x) and expectation based on the CHECKSUMS.txt file.

Note

In version 1.2.0 and earlier, two checksums per file were required because of differences in the checksum hash values on Windows and Unix-like platforms. Recent versions use a different (faster) algorithm and only require one checksum value per file. To update your ‘CHECKSUMS.txt’ files using the new algorithm, see https://github.com/PredictiveEcology/SpaDES/issues/295#issuecomment-246513405.

Author(s)

Alex Chubaty

Examples

modulePath <- file.path(tempdir(), "myModulePath")
dir.create(modulePath, recursive = TRUE, showWarnings = FALSE)
moduleName <- "myModule"
cat("hi", file = file.path(modulePath, moduleName)) # put something there for this example

## verify checksums of all data files
Checksums(modulePath, files = moduleName)

## write new CHECKSUMS.txt file
Checksums(files = moduleName, modulePath, write = TRUE)

---

Download from cloud, if necessary

Description

Meant for internal use, as there are internal objects as arguments.

Usage

cloudDownload(
  outputHash,
  newFileName,
  gdriveLs,
  cachePath,
  cloudFolderID,
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  verbose = getOption("reproducible.verbose")
)

Arguments

outputHash	The cacheId of the object to upload
newFileName	The character string of the local filename that the downloaded object will have
gdriveLs	The result of googledrive::drive_ls(googledrive::as_id(cloudFolderID), pattern = "outputHash")
cachePath	A repository used for storing cached objects. This is optional if Cache is used inside a SpaDES module.
cloudFolderID	A googledrive dribble of a folder, e.g., using drive_mkdir(). If left as NULL, the function will create a cloud folder with name from last two folder levels of the cachePath path, : paste0(basename(dirname(cachePath)), "_", basename(cachePath)). This cloudFolderID will be added to options("reproducible.cloudFolderID"), but this will not persist across sessions. If this is a character string, it will treat this as a folder name to create or use on GoogleDrive.
drv	If using a database backend, drv must be an object that inherits from DBIDriver (e.g., RSQLite::SQLite).
conn	an optional DBIConnection object, as returned by dbConnect().
verbose	Numeric, -1 silent (where possible), 0 being very quiet, 1 showing more messaging, 2 being more messaging, etc. Default is 1. Above 3 will output much more information about the internals of Caching, which may help diagnose Caching challenges. Can set globally with an option, e.g., ⁠options('reproducible.verbose' = 0) to reduce to minimal⁠

---

NA-aware comparison of two vectors

Description

Copied from http://www.cookbook-r.com/Manipulating_data/Comparing_vectors_or_factors_with_NA/. This function returns TRUE wherever elements are the same, including NA's, and FALSE everywhere else.

Usage

compareNA(v1, v2)

Arguments

v1	A vector
v2	A vector

Value

A logical vector, indicating positions where two vectors are same or differ.

Examples

a <- c(NA, 1, 2, NA)
b <- c(1, NA, 2, NA)
compareNA(a, b)

---

Convert all ways of calling a function into canonical form, including defaults

Description

e.g., stats::rnorm(1) –> rnorm(n = 1, mean = 0, sd = 1)

Usage

convertCallToCommonFormat(call, usesDots, isSquiggly, .callingEnv)

Arguments

call	The full captured call as it was passed by user.
usesDots	Logical. Whether the original Cache call used ...
isSquiggly	Logical. Whether there are curly braces e.g., as in a pipe sequence.
.callingEnv	Environment. The environment from which Cache was called.

---

Change the absolute path of a file

Description

convertPaths is simply a wrapper around gsub for changing the first part of a path. convertRasterPaths is useful for changing the path to a file-backed raster (e.g., after copying the file to a new location).

Usage

convertPaths(x, patterns, replacements)

convertRasterPaths(x, patterns, replacements)

Arguments

x	For convertPaths, a character vector of file paths. For convertRasterPaths, a disk-backed RasterLayer object, or a list of such rasters.
patterns	Character vector containing a pattern to match (see ?gsub).
replacements	Character vector of the same length of patterns containing replacement text (see ?gsub).

Value

A normalized path with the patterns replaced by replacements. Or a list of such objects if x was a list.

Author(s)

Eliot McIntire and Alex Chubaty

Examples

filenames <- c("/home/user1/Documents/file.txt", "/Users/user1/Documents/file.txt")
oldPaths <- dirname(filenames)
newPaths <- c("/home/user2/Desktop", "/Users/user2/Desktop")
convertPaths(filenames, oldPaths, newPaths)

---

Recursive copying of nested environments, and other "hard to copy" objects

Description

When copying environments and all the objects contained within them, there are no copies made: it is a pass-by-reference operation. Sometimes, a deep copy is needed, and sometimes, this must be recursive (i.e., environments inside environments).

Usage

Copy(object, ...)

## S4 method for signature 'ANY'
Copy(
  object,
  filebackedDir,
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  verbose = getOption("reproducible.verbose"),
  ...
)

## S4 method for signature 'data.table'
Copy(object, ...)

## S4 method for signature 'list'
Copy(object, ...)

## S4 method for signature 'refClass'
Copy(object, ...)

## S4 method for signature 'data.frame'
Copy(object, ...)

Arguments

object	An R object (likely containing environments) or an environment.
...	Only used for custom Methods
filebackedDir	A directory to copy any files that are backing R objects (Raster and SpatRaster classes). If missing, a fresh temporary directory is used. Can be NULL, which means that the file will not be copied and could therefore cause a collision as the pre-copied object and post-copied object would have the same file backing them. A copy is never written beside the original file.
drv	If using a database backend, drv must be an object that inherits from DBIDriver (e.g., RSQLite::SQLite).
conn	an optional DBIConnection object, as returned by dbConnect().
verbose	Numeric, -1 silent (where possible), 0 being very quiet, 1 showing more messaging, 2 being more messaging, etc. Default is 1. Above 3 will output much more information about the internals of Caching, which may help diagnose Caching challenges. Can set globally with an option, e.g., ⁠options('reproducible.verbose' = 0) to reduce to minimal⁠

Details

To create a new Copy method for a class that needs its own method, try something like shown in example and put it in your package (or other R structure).

Value

The same object as object, but with pass-by-reference class elements "deep" copied. reproducible has methods for several classes.

Author(s)

Eliot McIntire

See Also

.robustDigest(), Filenames()

Examples

e <- new.env()
e$abc <- letters
e$one <- 1L
e$lst <- list(W = 1:10, X = runif(10), Y = rnorm(10), Z = LETTERS[1:10])
ls(e)

# 'normal' copy
f <- e
ls(f)
f$one
f$one <- 2L
f$one
e$one ## uh oh, e has changed!

# deep copy
e$one <- 1L
g <- Copy(e)
ls(g)
g$one
g$one <- 3L
g$one
f$one
e$one
## To create a new deep copy method, use the following template
## setMethod("Copy", signature = "the class", # where = specify here if not in a package,
##           definition = function(object, filebackendDir, ...) {
##           # write deep copy code here
##           })

---

Copy a file using robocopy on Windows and rsync on Linux/macOS

Description

This is replacement for file.copy, but for one file at a time. The additional feature is that it will use robocopy (on Windows) or rsync on Linux or Mac, if they exist. It will default back to file.copy if none of these exists. If there is a possibility that the file already exists, then this function should be very fast as it will do "update only", i.e., nothing.

Usage

copySingleFile(
  from = NULL,
  to = NULL,
  useRobocopy = TRUE,
  overwrite = TRUE,
  delDestination = FALSE,
  create = TRUE,
  silent = FALSE
)

copyFile(
  from = NULL,
  to = NULL,
  useRobocopy = TRUE,
  overwrite = TRUE,
  delDestination = FALSE,
  create = TRUE,
  silent = FALSE
)

Arguments

from	The source file.
to	The new file.
useRobocopy	For Windows, this will use a system call to robocopy which appears to be much faster than the internal file.copy function. Uses ⁠/MIR⁠ flag. Default TRUE.
overwrite	Passed to file.copy
delDestination	Logical, whether the destination should have any files deleted, if they don't exist in the source. This is ⁠/purge⁠ for robocopy and –delete for rsync.
create	Passed to checkPath.
silent	Should a progress be printed.

Value

This function is called for its side effect, i.e., a file is copied from to to.

Author(s)

Eliot McIntire and Alex Chubaty

Examples

tmpDirFrom <- file.path(tempdir(), "example_fileCopy_from")
tmpDirTo <- file.path(tempdir(), "example_fileCopy_to")
tmpFile1 <- tempfile("file1", tmpDirFrom, ".csv")
tmpFile2 <- tempfile("file2", tmpDirFrom, ".csv")
dir.create(tmpDirFrom, recursive = TRUE, showWarnings = FALSE)
dir.create(tmpDirTo, recursive = TRUE, showWarnings = FALSE)
f1 <- normalizePath(tmpFile1, mustWork = FALSE)
f2 <- normalizePath(tmpFile2, mustWork = FALSE)
t1 <- normalizePath(file.path(tmpDirTo, basename(tmpFile1)), mustWork = FALSE)
t2 <- normalizePath(file.path(tmpDirTo, basename(tmpFile2)), mustWork = FALSE)

write.csv(data.frame(a = 1:10, b = runif(10), c = letters[1:10]), f1)
write.csv(data.frame(c = 11:20, d = runif(10), e = letters[11:20]), f2)
copyFile(c(f1, f2), c(t1, t2))
file.exists(t1) ## TRUE
file.exists(t2) ## TRUE
identical(read.csv(f1), read.csv(f2)) ## FALSE
identical(read.csv(f1), read.csv(t1)) ## TRUE
identical(read.csv(f2), read.csv(t2)) ## TRUE

---

Low-level functions to create and work with a cache

Description

These are intended for advanced use only.

Usage

createCache(
  cachePath = getOption("reproducible.cachePath"),
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  force = FALSE,
  verbose = getOption("reproducible.verbose")
)

loadFromCache(
  cachePath = getOption("reproducible.cachePath"),
  cacheId,
  preDigest,
  fullCacheTableForObj = NULL,
  cacheSaveFormat = getOption("reproducible.cacheSaveFormat"),
  .functionName = NULL,
  .dotsFromCache = NULL,
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  verbose = getOption("reproducible.verbose")
)

extractFromCache(sc, elem, ifNot = NULL)

rmFromCache(
  cachePath = getOption("reproducible.cachePath"),
  cacheId,
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  cacheSaveFormat = getOption("reproducible.cacheSaveFormat"),
  verbose = getOption("reproducible.verbose"),
  ...
)

CacheDBFile(
  cachePath = getOption("reproducible.cachePath"),
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL)
)

CacheStorageDir(cachePath = getOption("reproducible.cachePath"))

CacheStoredFile(
  cachePath = getOption("reproducible.cachePath"),
  cacheId,
  cacheSaveFormat = getOption("reproducible.cacheSaveFormat"),
  obj = NULL,
  readOnly = FALSE
)

CacheDBTableName(
  cachePath = getOption("reproducible.cachePath"),
  drv = getDrv(getOption("reproducible.drv", NULL))
)

CacheIsACache(
  cachePath = getOption("reproducible.cachePath"),
  create = FALSE,
  drv = getDrv(getOption("reproducible.drv", NULL)),
  conn = getOption("reproducible.conn", NULL),
  verbose = getOption("reproducible.verbose")
)

Arguments

cachePath	A path describing the directory in which to create the database file(s)
drv	A driver, passed to dbConnect
conn	an optional DBIConnection object, as returned by dbConnect().
force	Logical. Should it create a cache in the cachePath, even if it already exists, overwriting.
verbose	Numeric, -1 silent (where possible), 0 being very quiet, 1 showing more messaging, 2 being more messaging, etc. Default is 1. Above 3 will output much more information about the internals of Caching, which may help diagnose Caching challenges. Can set globally with an option, e.g., ⁠options('reproducible.verbose' = 0) to reduce to minimal⁠
cacheId	The cacheId or otherwise digested hash value, as character string.
preDigest	The list of preDigest that comes from CacheDigest of an object
fullCacheTableForObj	The result of showCache, but subsetted for only the cacheId being loaded or selected
cacheSaveFormat	The text string representing the file extension used normally by different save formats; currently only "rds" or "qs" (which now uses qs2 package. Defaults to getOption("reproducible.cacheSaveFormat", "rds")
.functionName	Optional. Used for messaging when this function is called from Cache
.dotsFromCache	Optional. Used internally.
sc	a cache tags data.table object
elem	character string specifying a tagKey value to match
ifNot	character (or NULL) specifying the return value to use if elem not matched
...	Arguments passed to FUN, if FUN is not an expression.
obj	The optional object that is of interest; it may have an attribute "saveRawFile" that would be important.
readOnly	Logical. Only relevant during transition from qs to qs2. Essentially, during transition, qs objects can be read, but not saved. If TRUE then the CacheStoredFile can return a .qs file; if FALSE, then this will not be able to return qs; instead it will return qs2 files.
create	Logical. Currently only affects non RSQLite default drivers. If TRUE and there is no Cache database, the function will create one.

Details

• createCache() will create a Cache folder structure and necessary files, based on the particular drv or conn provided;

• loadFromCache() retrieves a single object from the cache, given its cacheId;

• extractFromCache() retrieves a single tagValue from the cache based on the tagKey of elem;

• rmFromCache() removes one or more items from the cache, and updates the cache database files.

Value

• createCache() returns NULL (invisibly) and intended to be called for side effects;

• loadFromCache() returns the object from the cache that has the particular cacheId;

• extractFromCache() returns the tagValue from the cache corresponding to elem if found, otherwise the value of ifNot;

• rmFromCache() returns NULL (invisibly) and is intended to be called for side effects;

• CacheDBFile() returns the name of the database file for a given Cache, when useDBI() == FALSE, or NULL if TRUE;

• CacheDBFiles() (i.e,. plural) returns the name of all the database files for a given Cache when useDBI() == TRUE, or NULL if FALSE;

• CacheStoredFile() returns the file path to the file with the specified hash value, This can be loaded to memory with e.g., loadFile().;

• CacheStorageDir() returns the name of the directory where cached objects are stored;

• CacheStoredFile returns the file path to the file with the specified hash value;

• CacheDBTableName() returns the name of the table inside the SQL database, if that is being used;

• CacheIsACache() returns a logical indicating whether the cachePath is currently a reproducible cache database;

Examples

data.table::setDTthreads(2)
newCache <- tempdir2()
createCache(newCache)

out <- Cache(rnorm(1), cachePath = newCache)
cacheId <- gsub("cacheId:", "", attr(out, "tags"))
loadFromCache(newCache, cacheId = cacheId)

rmFromCache(newCache, cacheId = cacheId)

# clean up
unlink(newCache, recursive = TRUE)

data.table::setDTthreads(2)
newCache <- tempdir2()

# Given the drv and conn, creates the minimum infrastructure for a cache
createCache(newCache)

CacheDBFile(newCache) # identifies the database file
CacheStorageDir(newCache) # identifies the directory where cached objects are stored

out <- Cache(rnorm(1), cachePath = newCache)
cacheId <- gsub("cacheId:", "", attr(out, "tags"))
CacheStoredFile(newCache, cacheId = cacheId)

# The name of the table inside the SQL database
CacheDBTableName(newCache)

CacheIsACache(newCache) # returns TRUE

# clean up
unlink(newCache, recursive = TRUE)

---

Count Active Threads Based on CPU Usage

Description

This function counts the number of active system processes (threads) that match a given pattern and exceed a specified minimum CPU usage threshold. It works on Unix-like systems (e.g., Linux, macOS) and does not support Windows.

Usage

detectActiveCores(pattern = "", minCPU = 50)

Arguments

pattern	A character string used to filter process lines. Only processes whose command line matches this pattern will be considered. Default is "" (matches all).
minCPU	A numeric value specifying the minimum CPU usage (in percent) for a process to be considered active. Default is 50.

Value

An integer representing the number of active threads matching the pattern and exceeding the CPU usage threshold. Returns NULL with a message if run on Windows.

Note

This function uses the ps -ef system command and regular expressions to parse CPU usage. It may not be portable across all Unix variants.

Examples

detectActiveCores(pattern = "R", minCPU = 30)

---

Determine filename, either automatically or manually

Description

Determine the filename, given various combinations of inputs.

Usage

determineFilename(
  filename2 = NULL,
  filename1 = NULL,
  destinationPath = getOption("reproducible.destinationPath", "."),
  verbose = getOption("reproducible.verbose", 1),
  prefix = "Small",
  ...
)

Arguments

filename2	filename2 is optional, and is either NULL (no writing of outputs to disk), or several options for writing the object to disk. If TRUE (the default), it will give it a file name determined by .prefix(basename(filename1), prefix). If a character string, it will use this as its file name. See determineFilename().
filename1	Character strings giving the file paths of the input object (filename1) filename1 is only used for messaging (i.e., the object itself is passed in as x) and possibly naming of output (see details and filename2).
destinationPath	Optional. If filename2 is a relative file path, then this will be the directory of the resulting absolute file path.
verbose	Numeric, -1 silent (where possible), 0 being very quiet, 1 showing more messaging, 2 being more messaging, etc. Default is 1. Above 3 will output much more information about the internals of Caching, which may help diagnose Caching challenges. Can set globally with an option, e.g., ⁠options('reproducible.verbose' = 0) to reduce to minimal⁠
prefix	The character string to prepend to filename1, if filename2 not provided.
...	Passed to methods. None currently implemented.

Details

The post processing workflow, which includes this function, addresses several scenarios, and depending on which scenario, there are several file names at play. For example, Raster objects may have file-backed data, and so possess a file name, whereas Spatial objects do not. Also, if post processing is part of a prepInputs() workflow, there will always be a file downloaded. From the perspective of postProcess, these are the "inputs" or filename1. Similarly, there may or may not be a desire to write an object to disk after all post processing, filename2.

This subtlety means that there are two file names that may be at play: the "input" file name (filename1), and the "output" filename (filename2). When this is used within postProcess, it is straight forward.

However, when postProcess is used within a prepInputs call, the filename1 file is the file name of the downloaded file (usually automatically known following the downloading, and refered to as targetFile) and the filename2 is the file name of the of post-processed file.

If filename2 is TRUE, i.e., not an actual file name, then the cropped/masked raster will be written to disk with the original filenam1/targetFile name, with prefix prefixed to the basename(targetFile).

If filename2 is a character string, it will be the path of the saved/written object e.g., passed to writeOutput. It will be tested whether it is an absolute or relative path and used as is if absolute or prepended with destinationPath if relative.

If filename2 is logical, then the output filename will be prefix prefixed to the basename(filename1). If a character string, it will be the path returned. It will be tested whether it is an absolute or relative path and used as is if absolute or prepended with destinationPath if provided, and if filename2 is relative.

---

Download a remote file

Description

Download a remote file

Usage

downloadRemote(
  url,
  archive,
  targetFile,
  checkSums,
  dlFun = NULL,
  fileToDownload,
  messSkipDownload,
  destinationPath,
  overwrite,
  needChecksums,
  .tempPath,
  preDigest,
  alsoExtract = "similar",
  purge = FALSE,
  verbose = getOption("reproducible.verbose", 1),
  .callingEnv = parent.frame(),
  ...
)

Arguments

url	Optional character string indicating the URL to download from. If not specified, then no download will be attempted. If not entry exists in the CHECKSUMS.txt (in destinationPath), an entry will be created or appended to. This CHECKSUMS.txt entry will be used in subsequent calls to prepInputs or preProcess, comparing the file on hand with the ad hoc CHECKSUMS.txt. See table in preProcess().
archive	Optional character string giving the path of an archive containing targetFile, or a vector giving a set of nested archives (e.g., c("xxx.tar", "inner.zip", "inner.rar")). If there is/are (an) inner archive(s), but they are unknown, the function will try all until it finds the targetFile. See table in preProcess(). If it is NA, then it will not attempt to see it as an archive, even if it has archive-like file extension (e.g., .zip). This may be useful when an R function is expecting an archive directly.
targetFile	Character string giving the filename (without relative or absolute path) to the eventual file (raster, shapefile, csv, etc.) after downloading and extracting from a zip or tar archive. This is the file before it is passed to postProcess. The internal checksumming does not checksum the file after it is postProcessed (e.g., cropped/reprojected/masked). Using Cache around prepInputs will do a sufficient job in these cases. See table in preProcess().
checkSums	TODO
dlFun	Optional "download function" name, such as "raster::getData", which does custom downloading, in addition to loading into R. Still experimental. In a quoted dlFun, targetFile and destinationPath are the values prepInputs() is using (destinationPath is the shared folder when reproducible.destinationPathShared is set), the ... values are visible, and other names resolve from the calling environment.
fileToDownload	TODO
messSkipDownload	The character string text to pass to messaging if download skipped
destinationPath	Character string of a directory in which to download and save the file that comes from url and is also where the function will look for archive or targetFile. To prevent repeated downloads in different locations, the user can also set options("reproducible.destinationPathShared") to one or more local file paths to search for the file before attempting to download. Default for that option is NULL meaning do not search locally. The previous name options("reproducible.inputPaths") is still accepted as a backwards-compatible alias.
overwrite	Passed to googledrive::drive_download(), which writes into a temporary directory; it does not decide what is downloaded. A downloaded file always replaces an existing file of the same name, because it is only downloaded when that file is missing or fails its checksum.
needChecksums	Logical indicating whether to generate checksums. ## TODO: add overwrite arg to the function?
.tempPath	Optional temporary path for internal file intermediate steps. Will be cleared on.exit from this function.
preDigest	The list of preDigest that comes from CacheDigest of an object
alsoExtract	Which files to fetch or extract in addition to targetFile. Spatial data rarely arrives as one file – a shapefile is five or six, and a raster often has a .aux.xml, .ovr or .tfw alongside carrying its attribute table, overviews or georeferencing – so this controls how much of that company comes along. The default depends on where the files are coming from, because "everything" only means something when a container bounds it: alsoExtract from an archive from a url NULL (default) every file in the archive as for "similar" "similar" files sharing targetFile's name without its extension same, discovered by listing the remote directory NA or "none" targetFile only targetFile only a character vector those files only those files only An archive bounds what "all files" can mean, so NULL extracts all of it. A url has no such bound – the server may hold thousands of unrelated files – so NULL there means targetFile and its companions, never the whole remote directory. Each element of a character vector may also be a regular expression: if an element does not match any archive member literally (by relative path or basename), it is passed to grep() against the archive's file list and all matching members are extracted. For example, alsoExtract = "CMD_sm|CMD_sp" extracts every file whose name contains CMD_sm or CMD_sp. See table in preProcess().
purge	7 fetches every file of a Google Drive folder url again; see prepInputs().
verbose	Numeric, -1 silent (where possible), 0 being very quiet, 1 showing more messaging, 2 being more messaging, etc. Default is 1. Above 3 will output much more information about the internals of Caching, which may help diagnose Caching challenges. Can set globally with an option, e.g., ⁠options('reproducible.verbose' = 0) to reduce to minimal⁠
.callingEnv	The environment where the function was called from. Used to find objects, if necessary.
...	Additional arguments passed to postProcess() and Cache(). Since ... is passed to postProcess(), these will ... will also be passed into the inner functions, e.g., cropInputs(). Possibly useful other arguments include dlFun which is passed to preProcess. See details and examples.

---

Extract files from archive

Description

Extract zip or tar archive files, possibly nested in other zip or tar archives.

Usage

extractFromArchive(
  archive,
  destinationPath = getOption("reproducible.destinationPath", dirname(archive)),
  neededFiles = NULL,
  extractedArchives = NULL,
  checkSums = NULL,
  needChecksums = 0,
  filesExtracted = character(),
  checkSumFilePath = character(),
  quick = FALSE,
  verbose = getOption("reproducible.verbose", 1),
  .tempPath,
  ...
)

Arguments

archive	Character string giving the path of the archive containing the file to be extracted. This path must exist or be NULL
destinationPath	Character string giving the path where neededFiles will be extracted. Defaults to the archive directory.
neededFiles	Character string giving the name of the file(s) to be extracted.
extractedArchives	Used internally to track archives that have been extracted from.
checkSums	A checksums file, e.g., created by Checksums(..., write = TRUE)
needChecksums	A numeric, with 0 indicating do not write a new checksums, 1 write a new one, 2 append new information to existing one.
filesExtracted	Used internally to track files that have been extracted.
checkSumFilePath	The full path to the checksum.txt file
quick	Passed to Checksums
verbose	Numeric, -1 silent (where possible), 0 being very quiet, 1 showing more messaging, 2 being more messaging, etc. Default is 1. Above 3 will output much more information about the internals of Caching, which may help diagnose Caching challenges. Can set globally with an option, e.g., ⁠options('reproducible.verbose' = 0) to reduce to minimal⁠
.tempPath	Optional temporary path for internal file intermediate steps. Will be cleared on.exit from this function.
...	Passed to unzip or untar, e.g., overwrite

Value

A character vector listing the paths of the extracted archives.

Author(s)

Jean Marchal and Eliot McIntire

---

Return the filename(s) from a ⁠Raster*⁠ object

Description

This is mostly just a wrapper around filename from the raster package, except that instead of returning an empty string for a RasterStack object, it will return a vector of length >1 for RasterStack.

Usage

Filenames(obj, allowMultiple = TRUE, returnList = FALSE)

## S4 method for signature 'ANY'
Filenames(obj, allowMultiple = TRUE, returnList = FALSE)

## S4 method for signature 'environment'
Filenames(obj, allowMultiple = TRUE, returnList = FALSE)

## S4 method for signature 'list'
Filenames(obj, allowMultiple = TRUE, returnList = FALSE)

## S4 method for signature 'data.table'
Filenames(obj, allowMultiple = TRUE, returnList = FALSE)

## S4 method for signature 'Path'
Filenames(obj, allowMultiple = TRUE, returnList = FALSE)

Arguments

obj	A ⁠Raster*⁠ object (i.e., RasterLayer, RasterStack, RasterBrick)
allowMultiple	Logical. If TRUE, the default, then all relevant filenames will be returned, i.e., in cases such as .grd where multiple files are required. If FALSE, then only the first file will be returned, e.g., filename.grd, in the case of default Raster format in R.
returnList	Default FALSE. If FALSE, then return format will be a character vector. When TRUE, list or environment objects will return a list of character strings or vectors. When returned as a character vector, then the names of objects with >1 filename associated with them will be given a numeric suffix, which means the name in the returned vector does not match the object in the list or environment. When returned as a list, their names are preserved.

Details

New methods can be made for this generic.

Value

A character vector of filenames that are part of the objects passed to obj. This returns NULL is the object is not file-backed or does not have a method to recover the file-backed filename.

Author(s)

Eliot McIntire

---

Fix common errors in GIS layers, using terra

Description

Currently, this only tests for validity of a SpatVect file, then if there is a problem, it will run terra::makeValid

Usage

fixErrorsIn(
  x,
  error = NULL,
  verbose = getOption("reproducible.verbose"),
  fromFnName = ""
)

Arguments

x	The SpatStat or SpatVect object to try to fix.
error	The error message, e.g., coming from try(...)
verbose	Numeric, -1 silent (where possible), 0 being very quiet, 1 showing more messaging, 2 being more messaging, etc. Default is 1. Above 3 will output much more information about the internals of Caching, which may help diagnose Caching challenges. Can set globally with an option, e.g., ⁠options('reproducible.verbose' = 0) to reduce to minimal⁠
fromFnName	The function name that produced the error, e.g., maskTo

Value

An object of the same class as x, but with some errors fixed via terra::makeValid()

---

Forget what whyNoCacheHit() has read from a repository this session

Description

The per-function lookups are held for the session, so a run with many cache misses scans the repository once rather than once per miss. Call this if entries have been added by another process and the newest ones must be seen.

Usage

forgetCacheLookups(cachePath = NULL)

Arguments

cachePath

Repository to forget; NULL forgets all of them.

Value

Invisibly, the number of cached lookups dropped.

---

3-Step postProcess sequence for SpatRasters using gdalwarp

Description

DEFUNCT: Please use the postProcessTo functions.

gdalResample is a thin wrapper around sf::gdal_utils('gdalwarp', ...) with specific options set, notably, ⁠"-r", "near"⁠, -te, -te_srs, tr, -dstnodata = NA, -overwrite.

gdalMask is a thin wrapper around sf::gdal_utils('gdalwarp', ...) with specific options set, notably, -cutline, -dstnodata = NA, and -overwrite.

Usage

gdalProject(
  fromRas,
  toRas,
  filenameDest,
  verbose = getOption("reproducible.verbose"),
  ...
)

gdalResample(
  fromRas,
  toRas,
  filenameDest,
  verbose = getOption("reproducible.verbose"),
  ...
)

gdalMask(
  fromRas,
  maskToVect,
  writeTo = NULL,
  verbose = getOption("reproducible.verbose"),
  ...
)

Arguments

fromRas	see from argument from postProcessTo(), but can only be a SpatRaster.
toRas	see to argument from postProcessTo(), but can only be a SpatRaster.
filenameDest	A filename with an appropriate extension (e.g., .tif) for gdal to write the output to. Since this function is conceived to be part of a chain, and not the final step, this function does not use writeTo, which is reserved for the final step in the chain.
verbose	Numeric, -1 silent (where possible), 0 being very quiet, 1 showing more messaging, 2 being more messaging, etc. Default is 1. Above 3 will output much more information about the internals of Caching, which may help diagnose Caching challenges. Can set globally with an option, e.g., ⁠options('reproducible.verbose' = 0) to reduce to minimal⁠
...	For gdalProject, this can be method. For gdalMask can be destinationPath and touches. For all ⁠gdal*⁠, this can also be and datatype.
maskToVect	see maskTo argument from maskTo(), but can only be a SpatVector
writeTo	Optional character string of a filename to use writeRaster to save the final object. Default is NULL, which means there is no writeRaster

Details

gdalProject is a thin wrapper around sf::gdal_utils('gdalwarp', ...) with specific options set, notably, -r to method (in the ...), -t_srs to the crs of the toRas, -te to the extent of the toRas, -te_srs to the crs of the toRas, -dstnodata = NA, and -overwrite.

These three functions were an alternative sequence (gdalProject, gdalResample, gdalMask) for the from + projectTo SpatRaster / maskTo SpatVector case in postProcessTo. The sequence was determined to be faster and more accurate than any other ordering, including running all three steps in one gdalwarp call (one-step gdalwarp resulted in very coarse pixelation when converting from a coarse resolution to fine resolution). They are retained for reference only; the live postProcessTo path no longer dispatches to them.

See Also

gdalResample(), and gdalMask() and the overarching postProcessTo()

Examples

if (require("terra", quietly = TRUE)) {
  # prepare dummy data -- 3 SpatRasters, 2 SpatVectors
  # need 2 SpatRaster
  rf <- system.file("ex/elev.tif", package = "terra")
  elev1 <- terra::rast(rf)

  # a polygon vector
  f <- system.file("ex/lux.shp", package = "terra")
  vOrig <- terra::vect(f)
  v <- vOrig[1:2, ]

  # utm <- terra::crs("epsg:23028") # $wkt
  utm <- "+proj=utm +zone=28 +datum=WGS84 +units=m +no_defs"
  vInUTM <- terra::project(vOrig, utm)
  vAsRasInLongLat <- terra::rast(vOrig, resolution = 0.008333333)
  res100 <- 100
  rInUTM <- terra::rast(vInUTM, resolution = res100, vals = 1)
  # crop, reproject, mask, crop a raster with a vector in a different projection
  #  --> gives message about not enough information
  t1 <- postProcessTo(elev1, to = vInUTM)
  # crop, reproject, mask a raster to a different projection, then mask
  t2a <- postProcessTo(elev1, to = vAsRasInLongLat, maskTo = vInUTM)
  t3a <- postProcessTo(elev1, to = rInUTM, maskTo = vInUTM)

}

---

Relative paths

Description

Extracting relative file paths.

Usage

getRelative(path, relativeToPath)

makeRelative(files, absoluteBase)

Arguments

path	character vector or list specifying file paths
relativeToPath	directory against which path will be relativized.
files	character vector or list specifying file paths
absoluteBase	base directory (as absolute path) to prepend to files

Details

• getRelative() searches path "from the right" (instead of "from the left") and tries to reconstruct it relative to directory specified by relativeToPath. This is useful when dealing with symlinked paths.

• makeRelative() checks to see if files and normPath(absoluteBase) share a common path (i.e., "from the left"), otherwise it returns files.

Examples

## create a project directory (e.g., on a hard drive)
(tmp1 <- tempdir2("myProject", create = TRUE))

## create a cache directory elsewhere (e.g., on an SSD)
(tmp2 <- tempdir2("my_cache", create = TRUE))

## symlink the project cache directory to tmp2
## files created here are actually stored in tmp2
prjCache <- file.path(tmp1, "cache")
file.symlink(tmp2, prjCache)

## create a dummy cache object file in the project cache dir
(tmpf <- tempfile("cache_", prjCache))
cat(rnorm(100), file = tmpf)
file.exists(tmpf)
normPath(tmpf) ## note the 'real' location (i.e., symlink resolved)

getRelative(tmpf, prjCache) ## relative path
getRelative(tmpf, tmp2) ## relative path

makeRelative(tmpf, tmp2) ## abs path; tmpf and normPath(tmp2) don't share common path
makeRelative(tmpf, prjCache) ## abs path; tmpf and normPath(tmp2) don't share common path
makeRelative(normPath(tmpf), prjCache) ## rel path; share common path when both normPath-ed

unlink(tmp1, recursive = TRUE)
unlink(tmp2, recursive = TRUE)

---

Harmonize all forms of call

Description

This will convert all known (imagined) calls so that they have the same canonical format i.e., rnorm(n = 1, mean = 0, sd = 1)

Usage

harmonizeCall(callList, .callingEnv, .functionName = NULL)

Arguments

callList	A named list with elements call, usesDots and FUNorig
.callingEnv	The calling environment where Cache was called from
.functionName	A possible function name. If omitted, then it will be deduced from the callList and may be inaccurate.

Value

A named list. We illustrate with the example rnorm(1). The named list will have the original callList (call (the original call, without quote), FUNorig, the original value passed by user to FUN, and usesDots which is a logical indicating whether the ... are used), and appended with new_call (the harmonized call, with the function and arguments evaluated, e.g., (function (n, mean = 0, sd = 1) .Call(C_rnorm, n, mean, sd))(1)), func_call, the same harmonized call with neither function nor arguments not evaluated (e.g., rnorm(1)), func which will be function or method definition function (n, mean = 0, sd = 1) .Call(C_rnorm, n, mean, sd), and .functionName, which will be the function name as a character string (rnorm) either directly passed from the user's .functionName or deduced from the func_call.

---

Has a cached object has been updated?

Description

Has a cached object has been updated?

Usage

isUpdated(x)

Arguments

x	cached object

Value

logical

Examples

# An object not returned from Cache is not "updated"
isUpdated(1:10)

---

Keep original geometries of sf objects

Description

When intersections occur, what was originally 2 polygons features can become LINESTRING and/or POINT and any COLLECTIONS or ⁠MULTI-⁠ versions of these. This function evaluates what the original geometry was and drops any newly created different geometries. For example, if a POLYGON becomes a COLLECTION of MULTIPOLYGON, POLYGON and POINT geometries, the POINT geometries will be dropped. This function is used internally in postProcessTo().

Usage

keepOrigGeom(newObj, origObj)

Arguments

newObj	The new, derived sf object
origObj	The previous, object whose geometries should be used.

Value

The original newObj, but with only the type of geometry that entered into the function.

---

Hardlink, symlink, or copy a file

Description

Attempt first to make a hardlink. If that fails, try to make a symlink (on non-windows systems and symlink = TRUE). If that fails, copy the file.

Usage

linkOrCopy(
  from,
  to,
  symlink = TRUE,
  overwrite = TRUE,
  verbose = getOption("reproducible.verbose", 1)
)

Arguments

from, to	Character vectors, containing file names or paths. to can alternatively be the path to a single existing directory.
symlink	Logical indicating whether to use symlink (instead of hardlink). Default FALSE.
overwrite	Logical. Only used when a link cannot be made and the file is copied instead: passed to file.copy(), so TRUE replaces an existing to.
verbose	Numeric, -1 silent (where possible), 0 being very quiet, 1 showing more messaging, 2 being more messaging, etc. Default is 1. Above 3 will output much more information about the internals of Caching, which may help diagnose Caching challenges. Can set globally with an option, e.g., ⁠options('reproducible.verbose' = 0) to reduce to minimal⁠

Value

This function is called for its side effects, which will be a file.link is that is available or file.copy if not (e.g., the two directories are not on the same physical disk).

Note

Use caution with files-backed objects (e.g., rasters). See examples.

Author(s)

Alex Chubaty and Eliot McIntire

See Also

file.link(), file.symlink(), file.copy().

Examples

tmpDir <- file.path(tempdir(), "symlink-test")
tmpDir <- normalizePath(tmpDir, winslash = "/", mustWork = FALSE)
dir.create(tmpDir)

f0 <- file.path(tmpDir, "file0.csv")
write.csv(iris, f0)

d1 <- file.path(tmpDir, "dir1")
dir.create(d1)
write.csv(iris, file.path(d1, "file1.csv"))

d2 <- file.path(tmpDir, "dir2")
dir.create(d2)
f2 <- file.path(tmpDir, "file2.csv")

## create link to a file
linkOrCopy(f0, f2)
file.exists(f2) ## TRUE
identical(read.table(f0), read.table(f2)) ## TRUE

## deleting the link shouldn't delete the original file
unlink(f0)
file.exists(f0) ## FALSE
file.exists(f2) ## TRUE

if (requireNamespace("terra", quietly = TRUE)) {
  ## using spatRasters and other file-backed objects
  f3a <- system.file("ex/test.grd", package = "terra")
  f3b <- system.file("ex/test.gri", package = "terra")
  r3a <- terra::rast(f3a)
  f4a <- file.path(tmpDir, "raster4.grd")
  f4b <- file.path(tmpDir, "raster4.gri")
  linkOrCopy(f3a, f4a) ## hardlink
  linkOrCopy(f3b, f4b) ## hardlink
  r4a <- terra::rast(f4a)

  isTRUE(all.equal(r3a, r4a)) # TRUE

  ## cleanup
  unlink(tmpDir, recursive = TRUE)
}

---

List a Google Drive folder, using a mirror listing when remapped

Description

A remap-aware alternative to googledrive::drive_ls() for listing a Google Drive folder's files. When the folder id is in the reproducible.urlRemap manifest as a directory remap (a type = "dir" row; see makeUrlRemap()), the files are enumerated from the public bucket listing with no Google authentication. Otherwise it falls back to googledrive::drive_ls() (which authenticates as usual). It is meant as a near drop-in for the common as.data.table(drive_ls(url)) pattern: filter the returned names, then pass the corresponding urls to prepInputs().

Usage

listGoogleDriveFolder(
  url,
  pattern = NULL,
  verbose = getOption("reproducible.verbose", 1)
)

Arguments

url	A Google Drive folder URL or bare id.
pattern	Optional regular expression; only files whose name matches are kept (applied to both the mirror and Drive paths).
verbose	Numeric verbosity level.

Value

A data.table with columns name (file name), id (the Google Drive file id when listed from Drive; NA on the mirror path), and url – a URL to hand to prepInputs(): the public mirror URL when remapped, otherwise the Drive file URL. A 0-row table when the folder is empty/unreachable.

See Also

makeUrlRemap() for the directory-remap manifest; buckethost::makeMirrorManifest(directories = TRUE) to build one.

Examples

# A directory-remap manifest maps a Drive *folder* id to a bucket prefix-listing
# URL; build it with buckethost::makeMirrorManifest(directories = TRUE), or by
# hand (a `type = "dir"` row):
manifest <- data.frame(
  filename = "2020",
  url = paste0("https://object-arbutus.cloud.computecanada.ca/",
               "predictiveecology/?prefix=SCANFI_v2/2020/&delimiter=/"),
  id = "15T4HIFeqzwp0TuOuxmYoexuXdLFnCZBi",
  type = "dir"
)
remap <- makeUrlRemap(manifest)  # the "dir" row populates the folder-listing map

# Then, with `options(reproducible.urlRemap = manifest)` set, a call such as
#   files <- listGoogleDriveFolder(
#     "https://drive.google.com/drive/folders/15T4HIFeqzwp0TuOuxmYoexuXdLFnCZBi")
# lists that folder from the public mirror with no drive_auth(), returning
# name/id/url columns to feed to prepInputs(); without a manifest it falls back
# to googledrive::drive_ls() (which authenticates as usual).

---

Create a list with names from object names

Description

This is a convenience wrapper around ⁠a <- 1; newList <- list(a); names(newList) <- "a"⁠.

Usage

listNamed(...)

Arguments

...	Any elements to add to a list, as in base::list

Details

This will return a named list, where names are the object names, captured internally in the function and assigned to the list. If a user manually supplies names, these will be kept (i.e., not overwritten by the object name).

Examples

a <- 1
b <- 2
d <- 3
(newList <- listNamed(a, b, dManual = d)) # "dManual" name kept

---

Build a URL remap function from a manifest

Description

Convenience constructor for the reproducible.urlRemap option (see preProcess()). Given a data.frame with (at least) columns filename and url, it returns a function ⁠function(url, filename)⁠ suitable for options(reproducible.urlRemap = ...). The returned function matches on the basename of the resolved filename: when a manifest row's filename matches, its url is returned, so the download is redirected there (and, if that URL supports HTTP Range requests, the parallel download path applies). When there is no match it returns NULL, so the original URL is kept.

Usage

makeUrlRemap(manifest)

Arguments

manifest

A data.frame (or data.table) with at least the character columns filename and url. filename is matched against the basename of the file being downloaded. An optional id column (Google Drive file id) is matched secondarily, by the id parsed from the Drive url, when no filename is available (e.g. an unauthenticated Drive download). An optional type column ("file"/"dir") marks directory-remap rows (a Drive folder id -> bucket prefix-listing URL), used to enumerate a Drive folder from the public bucket without authentication.

Details

An optional id column (also accepted as googledriveId, googledrive_id, driveId or gid) holding the Google Drive file id enables a secondary match by id: when the resolved filename is unavailable — e.g. an unauthenticated session that cannot read a Drive file's metadata — the id is parsed from the Drive url and matched against this column. This lets a download be redirected to the (public) mirror with no authentication at all.

An optional type column ("file"/"dir") enables directory remaps: a "dir" row maps a Google Drive folder id (id column) to a bucket prefix-listing URL (its url). When preProcess() downloads such a folder, it enumerates the folder's files without googledrive::drive_ls(), so listing the folder needs no authentication. The dir row is answered from the manifest's own file rows whose url lies directly under that prefix (the ⁠?prefix=⁠ value, or the url itself); the bucket listing is fetched only when no file row matches, so a bucket that denies anonymous listing still works. Rows without a type column (or with type = "file") are ordinary file remaps, unchanged. buckethost::makeMirrorManifest(directories = TRUE) emits such a manifest.

The manifest itself — and the responsibility for keeping it current — lives with the user (for example, a community-maintained mirror manifest); reproducible hard-codes no mirror URLs.

Calling makeUrlRemap() yourself is optional: you can also assign the manifest data.frame (or a CSV path/URL) directly to the option, e.g. options(reproducible.urlRemap = read.csv("manifest.csv")), and reproducible will build (and cache) the remap function internally. See the urlRemap entry in reproducibleOptions().

Value

A function of ⁠(url, filename)⁠ returning a replacement URL, or NULL to keep the original.

See Also

preProcess() for the reproducible.urlRemap option.

Examples

manifest <- data.frame(
  filename = "SCANFI_att_biomass_2010_v2_20260119.tif",
  url = paste0(
    "https://object-arbutus.cloud.computecanada.ca/predictiveecology/",
    "SCANFI_v2/2010/SCANFI_att_biomass_2010_v2_20260119.tif"
  )
)
remap <- makeUrlRemap(manifest)

# matched by basename, whatever the original url was
remap("https://drive.google.com/file/d/abc123/view",
      "SCANFI_att_biomass_2010_v2_20260119.tif")
remap("https://example.com/other.tif", "other.tif") # NULL -- keep original

# to apply it to every download in a session:
#   options(reproducible.urlRemap = remap)

---

Remove quote and determine if call uses ...

Description

Minor cleaning up of the FUN and ... to be used subsequently. This does only very minor things as it is run even if useCache = FALSE, i.e., even if the Cache is skipped.

Usage

matchCall2(definition, call, envir, envir2 = parent.frame(), FUN)

Arguments

definition	a function, by default the function from which match.call is called. See details.
call	an unevaluated call to the function specified by definition, as generated by call.
envir	an environment, from which the ... in call are retrieved, if any.
envir2	Environment. The environment where matchCall2 was called.
FUN	Either a function (e.g., rnorm), a function call (e.g., rnorm(1)), or an unevaluated function call (e.g., using quote()).

Value

A named list with call (the original call, without quote), FUNorig, the original value passed by user to FUN, and usesDots which is a logical indicating whether the ... are used.

---

Merge two cache repositories together

Description

All the cacheFrom artifacts will be put into cacheTo repository. All userTags will be copied verbatim, including accessed, with 1 exception: date will be the current Sys.time() at the time of merging. The createdDate column will be similarly the current time of merging.

Usage

mergeCache(
  cacheTo,
  cacheFrom,
  drvTo = getDrv(getOption("reproducible.drv", NULL)),
  drvFrom = getDrv(getOption("reproducible.drv", NULL)),
  connTo = NULL,
  connFrom = NULL,
  verbose = getOption("reproducible.verbose")
)

## S4 method for signature 'ANY'
mergeCache(
  cacheTo,
  cacheFrom,
  drvTo = getDrv(getOption("reproducible.drv", NULL)),
  drvFrom = getDrv(getOption("reproducible.drv", NULL)),
  connTo = NULL,
  connFrom = NULL,
  verbose = getOption("reproducible.verbose")
)

Arguments

cacheTo	The cache repository (character string of the file path) that will become larger, i.e., merge into this
cacheFrom	The cache repository (character string of the file path) from which all objects will be taken and copied from
drvTo	The database driver for the cacheTo.
drvFrom	The database driver for the cacheFrom
connTo	The connection for the cacheTo. If not provided, then a new one will be made from drvTo and cacheTo
connFrom	The database for the cacheFrom. If not provided, then a new one will be made from drvFrom and cacheFrom
verbose	Numeric, -1 silent (where possible), 0 being very quiet, 1 showing more messaging, 2 being more messaging, etc. Default is 1. Above 3 will output much more information about the internals of Caching, which may help diagnose Caching challenges. Can set globally with an option, e.g., ⁠options('reproducible.verbose' = 0) to reduce to minimal⁠

Value

The character string of the path of cacheTo, i.e., not the objects themselves.

---

Use message with a consistent use of verbose

Description

This family has a consistent use of verbose allowing messages to be turned on or off or verbosity increased or decreased throughout the family of messaging in reproducible.

Usage

messageDF(
  df,
  round,
  colour = NULL,
  colnames = NULL,
  indent = NULL,
  verbose = getOption("reproducible.verbose"),
  verboseLevel = 1,
  appendLF = TRUE
)

messagePrepInputs(
  ...,
  appendLF = TRUE,
  verbose = getOption("reproducible.verbose"),
  verboseLevel = 1
)

messagePreProcess(
  ...,
  appendLF = TRUE,
  verbose = getOption("reproducible.verbose"),
  verboseLevel = 1
)

messageCache(
  ...,
  colour = getOption("reproducible.messageColourCache"),
  verbose = getOption("reproducible.verbose"),
  verboseLevel = 1,
  appendLF = TRUE
)

messageQuestion(..., verboseLevel = 0, appendLF = TRUE)

.messageFunctionFn(
  ...,
  appendLF = TRUE,
  verbose = getOption("reproducible.verbose"),
  verboseLevel = 1
)

messageColoured(
  ...,
  colour = NULL,
  indent = NULL,
  hangingIndent = TRUE,
  verbose = getOption("reproducible.verbose", 1),
  verboseLevel = 1,
  appendLF = TRUE
)

Arguments

df	A data.frame, data.table, matrix
round	An optional numeric to pass to round
colour	Any colour that can be understood by cli
colnames	Logical or NULL. If TRUE, then it will print column names even if there aren't any in the df (i.e., they will) be V1 etc., NULL will print them if they exist, and FALSE which will omit them.
indent	An integer, indicating whether to indent each line
verbose	Numeric, -1 silent (where possible), 0 being very quiet, 1 showing more messaging, 2 being more messaging, etc. Default is 1. Above 3 will output much more information about the internals of Caching, which may help diagnose Caching challenges. Can set globally with an option, e.g., ⁠options('reproducible.verbose' = 0) to reduce to minimal⁠
verboseLevel	The numeric value for this ⁠message*⁠ call, equal or above which verbose must be. The higher this is set, the more unlikely the call will show a message.
appendLF	logical: should messages given as a character string have a newline appended?
...	Any character vector, passed to paste0(...)
hangingIndent	Logical. If there are ⁠\n⁠, should there be a handing indent of 2 spaces. Default is TRUE

Details

• messageDF uses message to print a clean square data structure.

• messageColoured allows specific colours to be used.

• messageQuestion sets a high level for verbose so that the message always gets asked.

Value

Used for side effects. This will produce a message of a structured data.frame.

---