drive_get, drive_ls, drive_download, drive_upload and the others, in the cloud cache, CacheGeo(), prepInputs() and the tile download) are run again after a brief failure: a timeout, DNS or connection error, HTTP 5xx, or failed token refresh, with a wait of 2 s times the attempt number. A 403 or 404 and any other error are raised at once. The default of reproducible.gdriveAuthRetries is now 6 (was 2) and it sets the number of retries of every Drive call, so a project no longer needs to set it; 0 turns retrying off. The cloud cache no longer calls retry(): one mechanism, so a non-transient failure now raises its own error at once, not after five attempts.Cache(useCloud = TRUE) reads an entry from a Google Drive folder shared "Anyone with the link" without a Google sign-in, which a batch run could not do: the cloud path called googledrive directly, which tried to authenticate. It now reaches Drive as prepInputs() does (a loaded token, then a configured identity, then anonymously). With no identity a miss is computed and saved locally, and the upload and folder creation are skipped with one message. Interactive sessions and folders given by name keep googledrive's own sign-in.logUrlAccess() records a download made without prepInputs()/preProcess() in the download ledger (prepInputsLog(); inside Cache() also as tags on the entry, replayed on a hit). CacheGeo()/CacheGeoWrite() now record the Google Drive ledger file they read or write with it. ?prepInputsLog now describes the whole ledger: what is recorded, the cache tags and replay, the reproducible.urlLog modes, and why a ledger can miss an access.Cache() of a file-backed SpatRaster now also stores the companion files that exist beside its file (.aux.xml, .ovr, .msk; reproducible.sidecarCandidates), and restores a deleted one beside the file that is still there. With recent terra/GDAL a GeoTIFF's categories live only in the .tif.aux.xml, so a cache hit after the original files (or just the .aux.xml) were deleted returned a raster with no categories. Filenames() is unchanged.terraPath, which holds spat_<hash>_<pid>_<x>.tif) is restored under the cache, not into that directory (completes #601). The anchor branch of remapFilenames() had no foreign-tempdir check, so the object was backed by a dead worker's scratch file that a clean-up deleted ("[readStart] file does not exist"). Entries already in the cache are fixed without re-caching.CacheGeo() with the old arguments returns what the old CacheGeo() returned again: NULL, not a 0-row sf, when it finds no rows for domain (or there is no ledger file) and writes nothing; the whole ledger after a write, not only the rows for domain; and, with no ledger file and action = "nothing", the rows FUN returns (not written). fireSense_dataPrepFit tests the result with is.data.frame(), so an ELF with no ledger row was taken as fitted and the run stopped. Calls with the new arguments are unchanged.CacheGeo() with the old arguments (targetFile, domain, ...) chooses rows as the old CacheGeo() did again: every row that intersects domain, one that only touches it included, and none unless together they cover it (with bufferOK = TRUE, after a 10 km buffer). The rebuild had turned bufferOK = TRUE into a sliver tolerance of 2.5% of the ledger's extent, so a narrow polygon's own row was dropped: fireSense_dataPrepFit got 0 rows for ELF 13.1 and stopped. Calls that give match are unchanged.CacheGeoWrite() (and CacheGeo() writes) to a Google Drive ledger lost rows that other writers had added. The fetch before the merge went through prepInputs(), which kept a local copy that matched its own CHECKSUMS.txt (or the destinationPathShared copy), so a writer merged onto an old ledger even with no other process running; it is now downloaded directly. And only a shared-disk ledger was locked, so two processes could both read, then both push, and the later push dropped the other's rows (FireSense, 2026-10-08). The whole read, merge and push is now held under a file lock for a Drive ledger (in the user's cache directory, so all of a user's processes on one machine share it) and for a local-only ledger (beside the file) too.prepInputs()/preProcess(): on Windows without httr2, a download no longer corrupts a binary file such as SQLite. The download.file() fallback now uses mode = "wb" (#635).CacheGeo() with the old arguments stopped with "area is required" when an object for FUN in ... had a name that is the start of ledger, area, compute, match or tolerance (e.g. le = le, as in fireSense_spreadFit and fireSense_ignitionFit): R partially matched it to that argument, which came before .... ... now comes first in CacheGeo(), so those arguments must be named in full; unnamed arguments are still ledger, area, compute, ... in that order.CacheGeo() and CacheGeoWrite() return after a write hold copies of the new rows' list-column objects: .geoMerge() (R/cacheGeoFamily.R) passed them through as.data.frame() on a data.table, which copy()s. A copied xgboost model is a new booster, equal but not identical to the one FUN/compute returned (fireSense_ignitionFit tests that it is). The merge now uses setDF(), so nothing is copied.dlFun is first evaluated once in an environment with targetFile and destinationPath as prepInputs() uses them (the shared folder when reproducible.destinationPathShared is set), the ... values, and .callingEnv as parent. It used to be re-run over the calling frames until its names resolved; that search remains as the fallback if the first evaluation fails.studyAreaName() has a new argument notSupplied. A module that falls back to a hash of its study area when the user did not set .studyAreaName can write P(sim)$.studyAreaName <- studyAreaName(sim$studyArea, notSupplied = ".studyAreaName") and get one message naming the setting, the object hashed (sim$studyArea) and the hash, instead of writing that message itself. The default (NULL) is silent, as before.CacheGeoLedger(), CacheGeoRead() and CacheGeoWrite(), and CacheGeo() rebuilt on them (one help page, ?CacheGeoLedger). A ledger is one .rds file of spatial rows, local or shared through a Google Drive folder, a shared-disk folder or (read-only) a URL. CacheGeoRead() always returns an sf object (0 rows if nothing matches; list-columns kept) for match = "intersects", "covers" or "within", with a tolerance that drops slivers; it never writes, creates a folder or uploads, and needs no Google login for a file shared "Anyone with the link". CacheGeoWrite() replaces the rows with the same key (or geometry) or appends, re-reads the remote just before writing, pushes only if the content changed, and merges again if another writer got in between; writers to a shared-disk folder also hold a file lock. CacheGeo(ledger, area, compute) runs compute(area) only for an area the ledger does not cover, and its errors are no longer swallowed. The old arguments (targetFile, domain, FUN, action, ...) are still accepted, with one message per session. A ledger is no longer sorted by polygonID version; rows stay in key order. A list-column that holds an xgboost model is kept through write, read and upsert (the rows are never passed through as.data.table() or copy()).Cache() (R/cache.R, returnInClassOfInput() in R/cache-helpers.R) now returns a cache hit that is an sf or SpatVector in the class of the first sf/SpatVector argument of the call. Because sf and SpatVector share a cacheId (reproducible.digestVersion >= 4), a call first cached with one class used to return that class when rerun with the other. The cacheId is unchanged.targetFile (e.g. an ESRI x.gdb/ left by an earlier failed run) counted as present: Checksums() (R/checksums.R) gives every directory the checksum "dir", so it matched its "dir" row in CHECKSUMS.txt whatever it held. preProcess() then skipped download, extraction and the copy from destinationPathShared, and returned the empty directory. A "dir" row now passes only when the files recorded under it exist (or, with none recorded, when the directory is not empty). linkOrCopy() also no longer warns "already exists" for a destination directory that is already there.linkOrCopy() (R/preProcess.R), given a directory, created an empty directory at the destination, linked none of its files, and still reported "Hardlinked ... no copy/copies made". A targetFile that is a directory, such as an ESRI file geodatabase (x.gdb/), could then arrive empty when preProcess() linked it from destinationPathShared. It now links every file inside the directory.formatCheck() (R/DBI.R) errored with "argument is of length zero" when no cache file existed yet and reproducible.cacheSaveFormat was unset, as in a fresh subprocess while another process is still writing the entry (NULL == "check"). It now falls back to rds, as it already did for "check".prepInputs()/preProcess() with a dlFun that fails no longer saves the error object at targetFile (R/download.R, downloadRemote()); it stops with the dlFun error message and writes nothing, so the next call does not pick up the error as data.
prepInputs()/preProcess() with a dlFun that fails no longer saves the error object at targetFile (R/download.R, downloadRemote()); it stops with the dlFun error message and writes nothing. A quoted-call dlFun is now evaluated once, with targetFile and destinationPath as prepInputs() uses them (the shared store when reproducible.destinationPathShared is set); only an object-not-found failure falls back to searching the calling frames, as before.
.wrap() of a file-backed SpatRaster (R/exportedMethods.R, wrapSpatRaster()) only recorded the file's location, so a wrapped object saved with saveRDS could not be restored once the original file was gone. A new copyFiles = TRUE (passed through ...) copies the backing file(s) into cacheOutputs/ under cachePath, the place .unwrap() reads from, using the same helper Cache() uses. The default is unchanged, so Cache() does not copy twice.
The first path argument of .wrap(), .unwrap() and their methods is now filebackedPath (it was cachePath): it anchors and stores file-backed objects whether or not Cache() is involved. cachePath still works, silently for now; a deprecation message will follow once SpaDES.core and clusters use filebackedPath. Packages that define .wrap/.unwrap methods (e.g., SpaDES.core's simList methods) should rename their formal; Cache() now passes filebackedPath.
postProcessTo() has a rasterize argument. With TRUE, or a list of arguments for terra::rasterize() (e.g. list(field = "value", fun = "max")), a Vector from is cropped and projected as usual, then rasterized onto projectTo (or to), which must be Gridded; masking and writing are done on the raster, and a SpatRaster is returned.cropTo() with a raster cropTo in another CRS no longer reprojects the whole raster to get its extent. That took minutes for a 5000 x 5000 raster, and terra's grid snapping left the extent up to one cell too large or too small. It now projects 201 points along each edge of the extent, which matches the true footprint. A new help section, "Cropping to a raster in another CRS" in ?postProcessTo, compares the methods.reproducible.leaveOnDisk (default TRUE) now sets terraOptions(todisk = TRUE) during postProcessTo(). It used to set memfrac = 0 whenever memfrac was at terra's default 0.5, which made terra::project() about 15 times slower (34 minutes instead of under a minute for one SCANFI study area). memfrac is no longer set to 0.postProcessTo() warns when terraOptions(memfrac) is below 0.1 (including 0) and uses memfrac = 0.1 for the call, because lower values make terra very slow. To keep rasters out of memory, set terraOptions(memfrac = 0.1, todisk = TRUE).type = "dir" row in the reproducible.urlRemap manifest is now answered from the manifest's own file rows under that folder's prefix, so listGoogleDriveFolder() and Drive-folder downloads work with buckets that deny anonymous ListBucket. The bucket listing is used only when no file row matches.CacheGeo(action = "update"/"append") stopped with "ALTLIST classes must provide a Set_elt method" when the ledger already had a row and a list-column held an xgboost model (fireSense_ignitionFit's fits): the existing and new rows were joined through as.data.table(), whose copy() fails on that object. They are now joined with rbindlist() on the data.frames as they are.postProcessTo(writeTo = , gdal = ) (and writeTo()) now honour a gdal argument of GDAL creation options (e.g. c("INTERLEAVE=BAND", "TILED=YES")) for the written file; it was ignored and NUM_THREADS=1 hardcoded. NUM_THREADS=1 is still added unless the caller sets NUM_THREADS.urlLog() with a default method that returns the URL ledger kept as
reproducible.url* Cache tags as a table (one row per cache entry and url; columns chosen with
which, sorted by last use). It replaces showCache(userTags = "reproducible.url"), which also
matched argument digests containing "url". Other packages, such as SpaDES.core, add methods.Replacing a file that other processes may be reading no longer leaves its path missing.
linkOrCopy() unlinked the destination and then linked or copied onto it, and prepInputs(purge = 7)
moved the local copies aside for the whole download, so a second process reading the file (two
FireSense workers on one input, 2026-09-29) stopped with "The file does not exist".
linkOrCopy() now makes the new file under a temporary name beside the destination and renames it over
(on Windows, where an existing destination can block the rename, the old file is removed first).
purge = 7 fetches into a scratch directory inside destinationPath and swaps each file in that way,
leaving the old copies untouched if the download fails; a Google Drive folder purge = 7 replaces its
destinationPathShared copies the same way instead of unlinking them.
The header comment of R/urlLog.R said the default reproducible.urlLog = NULL keeps tags only and no
in-memory session log. The code fills the in-memory log in the default and TRUE modes, as ?prepInputsLog
says; the comment now matches. Comments only, no change in behaviour.
?prepInputs starts with a short Features table linking each feature to its argument, option or section.
?reproducibleOptions no longer says the default reproducible.urlLog = NULL keeps no in-session list.
Cache(cacheId = ) skips its "cacheId passed to override automatic digesting" message when the
cacheId carries the attribute cacheIdAnnounced = TRUE, for callers that have already said which
entry they are using (SpaDES.core's cacheChaining). A plain cacheId still prints it.Cache() no longer writes a copy of every file-backed raster beside its original when it memoises
a result (reproducible.useMemoise = TRUE). The memoised entry is now the wrapped object, whose
rasters point at the cache's own files, on every path (a new result, a disk hit, and
loadFromCache()), and a memoise hit restores files the way a disk hit does. Two processes that
shared an output folder used to write the same <name>_1.tif copy and corrupt it.Copy() of a file-backed SpatRaster now follows its documentation: with no filebackedDir the
copy goes to a fresh temporary directory, NULL makes no copy, and a directory takes the copy.
It never writes beside the original.nextNumericName() cut a file name at its first dot, so a name with a dot in it
(x_4.2.2.tif) always got _1.publishFile()). The destination is
a copy, never a hard link, so overwriting it in place cannot change the cache's file.SpatRaster that uses some of its file's layers keeps their order, and a layer used
more than once (a stack sampled with replacement) is restored as many times; the order used to
come back reversed and repeats were dropped.options("reproducible.cacheSaveFormat") unset (the default since 3.2.1.9046), a Cache()
hit on a qs2 entry rewrote its tag file (<cacheId>.dbFile.qs2) with saveRDS(). The next hit
could not read it and failed with "qs is required but not yet installed". The tag file is now
written in the format its file name says. A tag file already damaged this way is now read as
rds and saved back as qs2 on its next hit, and loadFile() no longer asks for qs when qs
is only a fallback candidate.CacheGeo(action = "update", ...) returned the existing row and never evaluated FUN when
domain was already covered by an existing entry, so a refit (e.g. fireSense_SpreadFit with
refitExisting = TRUE) was silently discarded. FUN is now evaluated and the matching row(s)
replaced, keyed on the polygonID column when present on both sides, otherwise on exact
geometry equality. This is step 1 of the planned CacheGeo redesign.options("reproducible.cacheSaveFormat") now defaults to NULL instead of "rds". A session
that never set the option was indistinguishable from one that explicitly chose "rds", so
reading an existing qs2 cache entry (via loadFromCache() or a Cache() hit) silently
converted it to .rds on disk. With the option unset, an entry is now read in whatever format
it is already stored in and is never converted; new entries are still saved as .rds. Setting
the option explicitly ("rds", "qs2", ...) keeps the previous behaviour, including converting
an existing entry to the requested format.swapCacheFileFormat() re-saved an entry in its new cacheSaveFormat without carrying over its
existing tags, because its callers never passed userTags; saveToCache() then defaulted to a
single "otherFunctions" tag, and the old entry (with its real tags) was removed. A format swap
(e.g. changing options(reproducible.cacheSaveFormat = ...)) now carries the entry's existing tags
forward, on both the file-backed and DBI cache backends. On the DBI backend, the swap also deleted
the newly-saved row along with the old one, because rmFromCache() there deletes by cacheId only
(no per-format column); the old entry is now removed before the new one is saved..wrap() no longer requires terra to cache a data.table: it shared the terra-only branch that
deep-copies a data.table, so Cache() of a data.table failed where terra is not installed.
With reproducible.useMemoise = TRUE, Cache() memoised the object it returned, not a copy, on a
first run. Changing the result in place (e.g. a data.table with set() or :=, or a later event
changing a simList) changed what the next memoised hit returned. Every memoise write now goes through
memoiseAssign(), which stores makeMemoisable() of the object.
reproducible.useMemoise = TRUE, loadFromCache() and Cache() store different forms of an
object in the shared memoise environment, and each read only its own. A Cache() hit after
loadFromCache() returned a simList as a plain list; loadFromCache() after a Cache() hit
returned a SpatRaster still packed. Both now read through one function, memoiseGet(), which
handles either form. SpaDES.core's cacheChaining jump calls loadFromCache() between Cache() hits.saveToCache() no longer memoises. Cache() overwrote that entry straight away, so it was an
unwrap on every cache miss for nothing.CacheGeo() reads a local targetFile again when its content has changed. The read was cached on the
file's name alone, so after another job appended to the file, every later call returned the first
version. It is now keyed on the file's md5, as a Google Drive file already was. useCache = FALSE now
also reaches that cached read; before, only options(reproducible.useCache = FALSE) switched it off.CacheGeo() can update a .gpkg targetFile. A .gpkg names its geometry column geom, and the append
step used $geometry, so it failed; checkNameHasGeom() also broke the sf object when renaming the column.
Before the fix above, the test never reached this: the .gpkg read returned the cached .rds read instead.prepInputs() no longer fetches the sf "guessing a driver" web page when the caller supplied both the file
and fun. For a file extension it did not know, it looked the extension up (with rvest, over the network)
even though nothing was left to guess, so a call such as fireSenseUtils' lightning data (.txt files read by
an explicit fun) made rvest look required.clearCache() removes an entry's file-backed files again. They are stored as <cacheId>_<name>, but since the
speedup in 3e5b888b (2025-11-03) clearCache() recognised an entry's files only as <cacheId>.<ext>. Entries that
also tag their files (a plain Cache() call) were still cleaned through the tags; others, such as a cached SpaDES
simList, left their rasters behind.linkOrCopy() (used by Cache() to store file-backed objects) no longer empties a file that is already
in place. When some files could not be hard-linked (e.g., on another filesystem), the copy fallback also
received a file that was being kept because its destination already held it. If that destination was a
hard link of the source, file.copy() truncated both to 0 bytes: a cached terra raster and its cache copy
were emptied, and the job then failed with "[rast] cannot open this file as a SpatRaster". The link
results were one element shorter than the file list and were recycled over it.Cache() call nested inside another is now counted as nested whether or not any userTags are
present. The nesting level was only incremented when some userTags existed, and a non-skipped
outer Cache() supplies one (outerFunction:<name>) but a skipped one does not. So under an outer
Cache(useCache = FALSE) or useCache = 0 without userTags, an inner Cache(useCache = TRUE)
was cached, while the same call with userTags on the outer call was skipped; numeric useCache
levels were miscounted the same way. Both now follow the documented rule: a nested TRUE takes the
outermost call's useCache. Behaviour change: code that relied on an untagged
Cache(useCache = FALSE) leaving inner Cache() calls on will now see them skipped.Cache(useCache = "always") caches a call even when options(reproducible.useCache = FALSE) --
which is what SpaDES.core's spades.useCache = "eventsOnly" sets for a run -- and even when it
is nested inside a Cache() call whose useCache is FALSE. Otherwise it is TRUE: the same
key, hit or miss as usual. It is for a package's own calls whose repetition is known to be
expensive, such as LandR's forest-land inputs, which fireSense prepared once per dataYear.
There is no off switch; delete the entry to recompute. It does not transfer: a Cache() nested
inside an "always" call keeps its own useCache, and an inner TRUE inherits TRUE. Before,
"always" (like any unrecognised string) happened to behave much the same, by falling through
to TRUE, but nested calls inherited the string.Cache() call in a session and then once per 50 skipped calls, and says so. With e.g. SpaDES.core's
options(spades.useCache = "eventsOnly") every module-internal Cache() call is skipped, and the
message printed hundreds of times per run, burying other output. verbose >= 2 still shows every one.try2()/tryCatch2() wrappers and the undocumented, non-exported
reproducible.useTry option that selected between their two branches. They date from a period of
debugging Rcpp/cache corruption; errors that were formerly trapped and inspected now propagate, so
failures surface at their origin. The removed stopRcppError() helper, which rephrased a digest
failure as "... is corrupt ... resolved by restarting the R session", is gone with it: a corrupt
object to be digested now raises the underlying error directly.dealWithCacheRecoveryErrors(), which the above narrowed: the Filenames() trigger is no longer
reachable in normal operation, leaving the try(output[1]) one. The tests pin the surviving
trigger, both error signatures it matches, and the no-op cases.accessed tag written on every cache hit, and tags written by callers such as SpaDES.core's
cacheChaining. Every write to a tag file (.addTagsRepo(), .updateTagsRepo(), saveDBFileSingle(),
rmFromCache(), and the format rewrite in showCache()) now holds <cacheId>.tags.lock.Cache(), the shared-input stash and tag writes now lock through one function, acquireLockFile(). A
process waiting on a Cache() lock takes it as soon as it is released, instead of re-checking every 2.5 s.useCache = FALSE, a Cache() call in the Cache(fn(args)) or fn(args) |> Cache() form
that also carried an argument Cache() does not have (e.g. a typo such as .omitArgs = for
omitArgs = ) failed with could not find function "FUN": the bypass took the stray argument
as a sign of the Cache(fn, args...) form and called the already-evaluated result. It now calls
only a function; anything else is returned as the result, as when caching is on.reproducible.showCachePreWarm now defaults to interactive() instead of TRUE. The pre-warm
only pays off when a later showCache() in the same session finds it finished, which is what
a person at the console waits for; a batch run (Rscript, R CMD BATCH, callr or future
workers, knitr) rarely does. Set the option or R_REPRODUCIBLE_SHOWCACHE_PREWARM to turn it on
there. An R session started with R --interactive, as the SpaDES.project queue workers are,
is interactive, so those workers still switch it off explicitly.SpatRaster that had been produced in a temporary
directory belonging to another process -- terra's tempdir when it is not this
session's tempdir() (e.g. one scratch disk shared by many workers), or another R
session's Rtmp* -- is no longer restored to that path just because the file still
exists. The file there is named for the process that produced it, so the next clean-up
of that process's temp files removed it from under the live object ("[project] cannot
create dataset from source"). Such rasters now come back under the cache. A raster in
this session's own tempdir() is still restored to its original path.The showCache() pre-warm process (prepopulateCacheAsync(), and Cache(showSimilar = TRUE))
now exits when its scan is done. It sent its result back through the fork's pipe, and an
attached fork stays alive until the parent collects it (a result larger than the pipe buffer
even blocks in the write). A batch job that never calls showCache() again never collects,
so every such job kept an idle ~400 MB process for its whole run, and the process outlived
the job if the job was killed. The fork is now detached: it writes its result to a temporary
file and exits, and the next showCache() call reads that file.
prepInputs()'s overwrite now applies only to the writeTo file, as its help page
said. It was also passed to preProcess(), where it never caused a re-download (a file
matching CHECKSUMS.txt is kept either way) but did decide whether a damaged local copy
could be replaced: under the default overwrite = FALSE, re-running a call over such a
file stopped with "already exists ... Use overwrite = TRUE?". A file that fails its
checksum is now downloaded again and replaced whatever overwrite is, with a message
naming the local copies (both, when reproducible.destinationPathShared is set).
When overwrite is not supplied, prepInputs() replaces its own writeTo output, so
re-running a call (e.g., after a Cache miss from a changed module set) no longer stops
with "already exists and overwrite = FALSE". An explicit overwrite = FALSE still
stops. Seen through LandR::prepInputs_SCANFI_LCC_FAO() in Biomass_borealDataPrep.
purge = 7 now downloads again. It sets aside every local copy of the call's files --
in destinationPath and, with reproducible.destinationPathShared, in the shared stash --
drops their CHECKSUMS.txt entries, and downloads, extracts and links them as on a first
run; the old copies are put back if the download fails. Before, it only dropped the
entries, which were then rebuilt from the files on disk, so nothing was downloaded and a
damaged file was accepted. The checksum-mismatch error now names every local copy and
suggests purge = 7 instead of deleting files by hand.
A Google Drive folder url now decides which files to fetch by CHECKSUMS.txt, like a
single file: a file whose entry matches is kept, a missing or mismatched one is fetched,
and purge = 7 fetches them all. It used to go by file name only, with overwrite as the
switch to fetch again.
preProcess(overwrite) no longer does anything and is deprecated: it is accepted, with a
one-time message pointing to prepInputs(overwrite) and purge = 7.
Cache(useCloud = TRUE) now accepts a Google Drive folder URL that ends at the folder id,
the form Drive's address bar gives. checkAndMakeCloudFolderID() took the id out with a
pattern that needed at least one character after it, so
https://drive.google.com/drive/folders/<33-character id> became a 32-character id. That
still looked like an id, so Drive was asked for a folder that does not exist and the call
stopped with "File not found". A URL with ?usp=... after the id was unaffected, which is
why this went unnoticed. The id is now read with the same helper prepInputs() uses for
Drive URLs.
linkOrCopy() no longer replaces a target that already holds byte-identical content.
It unlinked every existing target before linking, so the same bytes were given a new
inode and every other path hardlinked to the old one was stranded as a private copy.
This bit hardest through extractFromArchive(), which unpacks to a temp directory and
then moves the results to exdir -- and exdir is destinationPathShared itself
whenever the lookup has redirected destinationPath there. Each extraction therefore
destroyed the copy every other destination was sharing, and the next one destroyed that.
Measured on a 15-worker run: CA_FAO_forest_2019.tif collected three inodes in twenty
minutes, each orphaned at nlink = 1. Four sequential destinations now hold one inode
with nlink 2, 3, 4, 5; before, the stash inode changed and the count reset.
linkOrCopy() no longer calls file.link() with zero-length input, which errors with
no files to link from rather than returning logical(0). A set consisting only of
directories reached it; inside extractFromArchive() the error was swallowed and
retried until the extraction fallbacks ran out, surfacing as a misleading
Please install.packages('archive').
preProcess() now lets only one process at a time fill destinationPathShared for a
given input, so concurrent callers share one copy instead of each keeping their own.
Filling the stash for one input takes several steps -- read CHECKSUMS.txt, download,
extract, hardlink in, append the rows -- and nothing serialised them. Every process read
the stash before any of them had written it, so every process downloaded and extracted
its own copy and kept it; a process could also read the stash mid-write and match a
CHECKSUMS.txt row whose file was not linked in yet, failing with
No archive exists with filename: <stash>/x.zip. Measured with 15 concurrent consumers
of one archive: 2 of 15 succeeded, onto 2 inodes; now 15 of 15 succeed onto 1 inode. On
a 15-worker run, one 0.78 GB input had accumulated 86 paths across 36 inodes. The lock
is keyed on the input, so unrelated inputs never wait on each other, and it times out
(reproducible.stashLockTimeout, default one hour) rather than ever blocking a run.
destinationPathShared now links files that came out of an archive, instead of leaving
every destination with its own copy. The shared stash is scoped per archive by
.sharedDirsFor(), and runChecksums() reads back from that scoped directory, but
appendChecksumsTable() wrote its copy of the rows to the unscoped shared root -- where
the read side never looks. An archive-derived file was therefore never found in the
stash, so preProcess() re-downloaded it and then reached the "already exists at
<stash>" branch of downloadRemote(): an error under the default
overwrite = FALSE, and a private re-extraction that replaced the stash under
overwrite = TRUE. Either way the hardlink count never climbed past 2. Found on a run
with ~70 study areas sharing one stash, where hardlinking was saving 70% overall but
148 archive-derived files still accounted for 588 GB of duplicate copies -- e.g.
CA_FAO_forest_2019.tif present as 71 copies across 50 inodes.
whyNoCacheHit() answers, after the fact, the question a cache miss always
raises: which input changed. Every cache entry already records the hash of
each element it digested, addressed by its path into the call's arguments, so
the answer is a comparison rather than an investigation. Give it the entry the
surprising run wrote -- or nothing, for the most recent one -- and it names the
differing elements against the closest previous call for the same function:
Cache entry 7100d202c2d3ad3c (.inputObjects_canClimateData)
Closest previous call: 034ea09a1fbd24bc (2026-09-06 22:17:33)
1 of 44 digested elements differ:
sim.params.canClimateData..studyAreaName [differs]
It reads the repository only, so it needs neither the objects nor a re-run and
works on entries written before anything looked wrong. It complements
options(reproducible.showSimilar = TRUE), which reports at call time and
stops descending at reproducible.showSimilarDepth: this descends as far as
the recorded paths go, so a difference nested inside a parameter list is named
in full instead of appearing as "possible, unknown, differences in a nested
list".
Cache(dryRun = TRUE) now returns, invisibly, the element-by-element digest of
the call that would have run, instead of NULL. Passed to whyNoCacheHit(),
it answers "why will this miss?" before the expensive call runs:
dr <- Cache(myFun, args, dryRun = TRUE) # digests only; runs and saves nothing
whyNoCacheHit(dr)
showSimilar's "differences in a nested list deeper than N" message now names
whyNoCacheHit(), which has no depth limit, so the approximate answer points
at the exact one.
whyNoCacheHit(since = ) explains a whole run rather than one call, which is
the form a pipeline needs: in SpaDES the Cache() calls belong to modules,
so there is no call of yours to put dryRun on and no cacheId to hand over.
Every entry written after since (optionally up to before) is compared with
the closest earlier call of the same function, and the summary groups the
differences by element, so "13 entries differ on .studyAreaName" is one line
rather than thirteen.
whyNoCacheHitOnce() is the cache equivalent of debugonce(), and the
easiest way to use any of this in a pipeline. Arm it, run the thing that
recomputed when you expected it not to, and the first call that fails to reuse
the cache -- and has an earlier call of the same function to compare against
-- prints which elements differ and stops. The arming is then spent, so the
next run is unaffected and there is nothing to remember to switch off. A miss
with nothing to compare against is new work, not a slip, and does not spend the
arming. whyNoCacheHitOnce(k) fires only when the closest earlier call differs
in at most k elements; whyNoCacheHitOnce(cachePath = ) confines it to one
repository, so a miss in an unrelated cache does not spend it; and
whyNoCacheHitOnce(FALSE) cancels.
It costs nothing while a run is reusing the cache: the check sits after both hit paths, so a call that finds its result never reaches it and no repository read happens on its behalf.
New option reproducible.stopOnCacheMiss is the sticky form of the same
check, for keeping it on across a whole session.
Cost, measured on a 37,000-entry repository with
reproducible.useMemoise = TRUE: 64 s for the first check in a session (the
cold repository read), then 0.3-0.6 s per later check of the same function and
~2 s the first time each other function is seen. The per-function reads are
held for the session and dropped whenever anything is written, so they cannot
answer with a stale view; forgetCacheLookups() drops them by hand.
postProcessTo() registers its terraOptions(memfrac) restore with
add = TRUE. Without it that on.exit() replaced any restore already registered
in the same call, so the other option it changes (memmax) could be left altered
process-wide. Only the order in which the two branches happen to run was keeping
that from biting.if (!useDBI()), so with the DBI backend no
lock was taken and two concurrent workers reaching the same uncached key both
computed it. SQLite's own locking cannot prevent that: the function call happens
between two independent database operations and no transaction spans them, so WAL
and busy_timeout keep the database consistent without deduplicating the
computation. No deadlock is introduced, since this package holds no transactions,
so a database lock never outlives a single statement and every caller takes the
file lock first.googledrive::drive_deauth().
That call is process-wide, so one network blip during reproducible's first Drive access
left the whole R session anonymous, and every later direct googledrive::drive_ls() by
another package (LandR's SCANFI folder listing, for one) failed with "Does not exist" on
a folder that exists. After a genuine 403/404 denial the deauthorisation still happens,
and the configured email identity is now put back once reproducible's own read is over.alsoExtract = "similar" beside a plain file url no longer fetches archive-named
siblings (rasterTest.zip next to rasterTest.tif). A companion is a .tfw, .prj,
.aux.xml; an archive sharing the stem is different data, and once fetched it was
handed down the pipeline as an archive to extract from -- which is what has kept
LandR's R-CMD-check red since 2026-08-24 on machines without 7-Zip. Archive siblings
are still returned when the target itself is an archive.oauth2.googleapis.com, DNS, a
timeout, a 5xx) no longer demotes the request to anonymous access, which on a private
file ended in a misleading 404. Such failures are retried (reproducible.gdriveAuthRetries,
default 2, with backoff); a 403/404 denial is still final. The eventual
"Could not access the Google Drive resource" error now says which authenticated
attempts were made and why each failed..listFilesInArchive() no longer fails with "the condition has length > 1" on the
system-tool path. .checkForSimilar() passes several guessed candidate archives
(<name>.rar, .tar, .zip) when a download has no archive extension; the .rar
first element routed to the 7-Zip/unrar branch, which handed the whole vector to
.testForArchiveExtract(). Only reached when the archive package is not
installed, which is why it broke every LandR CI job (and development since
2026-08-29) while passing on developer machines. Candidates that are not on disk
now return NULL before any tool probe, and the probe sees the first candidate only.reproducible.destinationPathShared: files extracted from an archive are now
stored under <shared>/<archive stem>/ (with the archive and their own
CHECKSUMS.txt) instead of flat in the shared root. The flat layout keyed
extracted files by their archive-relative name alone, so two archives with the
same inner path -- every ClimateNA tile zip contains Year_1991MSY/CMD_sm.asc --
shared one file: preProcess() redirected into the shared root, found the name
in the root CHECKSUMS.txt, skipped extraction, and hardlinked the first
archive's file into every later request. An archive already sitting flat in a
pre-existing shared root is migrated into its scoped directory on first use, so
downloads are still reused; flat extracted files are left alone and no longer
consulted for archive contents. (#585)linkOrCopy() no longer errors when from and to contain a self-referential
pair alongside real ones. A file already at its destination cannot be linked or
copied onto itself -- file.copy() raises
"file can not be copied both 'from' and 'to'" -- and the guard against this was
!all(normPath(to) %in% normPath(from)). That is set membership rather than
pairwise, and all-or-nothing, while the operation itself is element-wise: a call
mixing self-referential pairs with real ones fell straight through and errored,
taking the legitimate copies down with it. It could also skip real work, when a
to happened to equal some unrelated from. Self-reference is now decided per
element; those elements are reported and skipped, and the rest proceed.
Reached in practice through prepInputs()/preProcess() when some needed files
are already in getOption("reproducible.inputPaths") and others are not.unwrapSpatRaster() restores a file-backed SpatRaster to the directory it was
produced in, rather than to file.path(cachePath, basename(f)). It had been
unlinking the original file before computing the destination, which made
remapFilenames()'s "is the original location still valid" test
(is_absolute_path(x) && file.exists(x)) fail on the very files just deleted.
For a raster with no resolvable anchor that test always failed, so the
destination fell back to the cache root with the cacheId dropped -- and two
cached calls whose rasters shared a basename silently overwrote one another
there, with one call returning the other's data. The unlink now clears only the
computed destination, after it is known.attributesReassign() no longer uses setdiff(), which was ~44% of
SpaDES.core::loadSimList(parse = FALSE) on a large simList. Same result,
~21% faster load.padYears() pads a c(start, end) pair of years to a common width, taken from
the end year, so the two sort together. Wraps paddedFloatToChar(), whose
padL default is the same formula.preProcess() writes the .hash sidecar on the first download, rather than
only on a later call that could re-verify the file locally. It records the
host's content hash (md5/sha1/sha256) and the ETag when both are offered:
the hash pins the bytes, the ETag is what the server can answer
If-None-Match against. Hosts whose ETag is not a content hash — including
raw.githubusercontent.com — previously got no sidecar at all, so every call
re-downloaded.getRemoteMetadata() asks the remote even when targetFile is supplied, so
the ETag and size are available in that case too. Previously that argument
skipped the request entirely, leaving remoteHash undefined and erroring./ to _, so example.com/my_data/sub_dir and example.com/my/data/sub/dir
shared one file. Names stay human-readable; sidecars written by earlier
versions are still found.preProcessCheckURLs(): walk a directory, ask each recorded URL whether
it has changed, and report. redownload decides what happens to anything
that has: report only ("no", the default), fetch it now ("immediate"), or
remove its sidecar so the next preProcess() fetches it
("nextPreProcess"). Partial matches work, e.g. "imm", "next".
This is the intended way to re-check — a deliberate action rather than
options(reproducible.checkRemoteHash = TRUE), which is easy to set once and
forget. See ?preProcessCheckURLs.options(reproducible.checkRemoteHash = TRUE) revalidates such files with a
conditional If-None-Match request: one round-trip, no download. If the
remote is unreachable the local file is used. The default is still to trust
the sidecar. See ?reproducibleOptions.prepInputs() no longer errors on Linux distributions without apt.
The check for RAR support shelled out to apt, which exists only on
Debian/Ubuntu; system(..., intern = TRUE) errors rather than returning
non-zero when a command is absent, so this failed on Fedora, RHEL, Arch and
SUSE. CRAN's r-devel-linux-x86_64-fedora-{clang,gcc} reported it as a check
ERROR. It now asks 7z itself (7z i), which answers the real question
("does this 7z support RAR?") on every platform and needs no package manager.detectActiveCores() no longer errors where ps is unavailable, and a
missing unrar no longer fails silently: both now check for the binary with
Sys.which() before calling it.brew install p7zip) alongside Debian/Ubuntu and RHEL/Fedora.testthat::is_testing() and
NOT_CRAN != "true"), the test is skip()-ped rather than erroring. Where
NOT_CRAN == "true" (local, CI) failures still error, so regressions are not
masked; outside tests, behaviour is unchanged.PredictiveEcology/reproducible.R CMD check. All \dontrun{} and
\donttest{} wrappers have been removed, so no example is exempt from being
checked. Examples that reach a third-party server are instead gated on
NOT_CRAN, keeping CRAN's machines off remote resources while local and CI
runs still exercise them. The prepInputs() example now uses a 15 kB
ecozones archive shipped in inst/ex (and mirrored as a release asset)
rather than repeatedly downloading a 1.44 MB file from a federal server;
prepInputs went from 7.1 s to 0.07 s and no example exceeds CRAN's 5 s
threshold (#437).prepInputs()/preProcess() record every file and web
access; view with prepInputsLog()/getUrlLog(). Includes targetFile,
archive and alsoExtract columns, per-Cache url frames, and cache-hit
replay. On by default; options(reproducible.urlLog = FALSE) disables.reproducible.timeout default raised to 12000.reproducible.urlRemap, accepting a manifest
data.frame or CSV path, with optional id (Google Drive) and type columns.showSimilar is now cloud-aware under useCloud.reproducible.digestVersion selects the hashing scheme; version 4 (the
new default) makes sf/SpatVector digests platform-stable and realizes
deferred-string ALTREP vectors before hashing.prepInputsCOG()): reads only the
spatial window of interest via /vsicurl/, with terra's native progress bar.reproducible.preDigestDump, improved showSimilar output,
and a one-time hint pointing at getUrlLog()/showCache().CacheGeo() handles multiple overlapping (not just contained) domains.lifecycle badges.Parallelism is now controlled by three options, one per resource —
reproducible.parallel.cores (CPU-bound work, i.e. tiling),
reproducible.parallel.download (concurrent ranged download connections) and
reproducible.parallel.upload (concurrent Google Drive uploads). The network
options are deliberately no longer derived from the core count, and are not
capped by mc.cores; CPU work still is. reproducible.parallel.streams and
reproducible.parallel.maxConnections are deprecated but still honoured, with
the new options taking precedence when both are set.
reproducible.parallel.cores = 1L additionally disables the background
showCache() pre-warm fork, which costs one extra process.
Five functions are no longer exported — internetExists(),
downloadFile(), loadFile(), checkAndMakeCloudFolderID() and
prepInputsCOG(). All are implementation details with better-maintained
equivalents elsewhere (e.g. curl::has_internet(), curl::curl_download())
or no use outside this package; they continue to work internally.
writeFuture() is defunct, and the future-backed background download
and cache-write paths are removed. Both were off by default and effectively
untested. reproducible.futurePlan is retained because downstream packages
set it, but has no effect as of 3.2.0.
options("reproducible.gdalwarp") is removed; it is no longer needed.
options("reproducible.cachePath") is no longer pre-set — it resolves lazily
on first use.
postProcessTo() is faster, uses less memory on large rasters, and keeps
categorical (factor) rasters categorical.
Tiling no longer deadlocks after a terra::writeRaster(). A default
writeRaster() allocates a GDAL thread pool of one thread per core that is
never released; forking such a process (as prepInputs(numTiles = ) did via
mclapply()) left the children holding GDAL mutexes no surviving thread could
release, and they hung indefinitely. The trigger is the child asking GDAL
for a worker pool, not the parent having one, so writing the tiles with
NUM_THREADS=1 is sufficient on its own — it holds regardless of what the
caller did beforehand, or which version of \pkg{terra} they run.
A failed parallel worker no longer silently drops tiles or uploads.
mclapply() marks every value a dead worker was given as failed, and those
results were not inspected — so a single failing worker quietly produced a
half-complete set (observed as 2 of 4 tiles reaching Google Drive). The failed
slice is now retried serially, and an error is raised if that also fails.
The last large third-party download is gone from the tests. test-gis.R
fetched a ~40 MB national fire-point archive from a federal server, on a URL
that has since been renamed upstream and now 404s. It is replaced by a 2.8 kB
25-point fixture shipped in inst/ex and hosted as a release asset.
prepInputs(numTiles = ) now works without geodata configured. It builds
its tile grid from GADM boundaries, but called geodata::gadm() with no
path, which errors on any machine where geodata::geodata_path() has never
been set. The download location is now resolved explicitly, preferring
somewhere persistent: geodata_path(), then
reproducible.destinationPathShared, then reproducible.inputPath (under
tempdir()). If GADM cannot be retrieved at all, a warning now says so
and names the fixed extent used instead, rather than failing or silently
tiling the wrong area.
Setting mc.cores no longer throttles downloads. Concurrent connections
defaulted to parallelly::availableCores() - 1, and because availableCores()
honours mc.cores, capping CPU forking silently reduced downloads to a single
connection.
Changing options(reproducible.cacheSaveFormat) no longer invalidates the
cache. On the file-backed backend every entry was silently recomputed and
both old and new files left on disk; entries are now migrated to the new
format. Two related defects on the same path are also fixed (loadFile()
erroring with object 'fe' not found when given format=, and the
format-retry loop giving up after the first candidate).
The background showCache() pre-warm fork is no longer leaked. It is
spawned only when it will be consumed, and never under a DBI backend;
previously long sessions accumulated forks and OOM-killed CI runners. New
advanced option reproducible.showCachePreWarm.
DBI backend: tag values containing a single quote no longer break inserts;
.addTagsRepo()/.updateTagsRepo() accept a named list of connections; and
URL logging is no longer silently disabled under useDBI(TRUE).
Google Drive: public files no longer trigger an interactive OAuth prompt;
authentication uses a verified fallback chain; failures report the actual
cause; and Cache(useCloud = TRUE) no longer pages the entire cloud-cache
folder. A best-effort cloud upload no longer aborts the run.
File-backed objects (e.g. terra SpatRaster) are restored portably
across machines via the new reproducible.fileBackedAnchors.
Platform-specific: .archiveExtractBinary() no longer invokes apt on
macOS, nor errors on Windows when more than one archive binary is installed.
Single-stream downloads no longer hang for certain servers; parallel ranged downloads are used only where the server supports them.
Assorted message and reporting fixes in messageColoured()/cliCol(),
getRelative(), and direct preProcess() calls.
test-destinationPathShared.R) on file systems
whose inode numbers exceed .Machine$integer.max: the inode helper now
compares inode numbers as strings rather than coercing to integer. No
user-facing change; the package code is unaffected.prepInputsCOG: new fast-path inside prepInputs for remote tiled GeoTiff
files (including Cloud Optimized GeoTiffs). When the url is HTTP(S) and at
least one of to, cropTo, or maskTo is supplied, only the spatial window
of interest is fetched via GDAL's /vsicurl/ — no full-file download. The
windowed SpatRaster is returned to the normal postProcess pipeline for
crop/mask/write. The fast-path can be disabled with
options(reproducible.useCOG = FALSE).The options reproducible.inputPaths and reproducible.inputPathsRecursive
have been renamed to reproducible.destinationPathShared and
reproducible.destinationPathSharedRecursive respectively (matching the prepInputs
naming family). The old names remain fully functional as backwards-compatible
aliases: if reproducible.destinationPathShared is NULL and reproducible.inputPaths
is set, the old value is used automatically (with a deprecation message).
Update your code by replacing options(reproducible.inputPaths = ...) with
options(reproducible.destinationPathShared = ...) at your convenience.
alsoExtract in prepInputs/preProcess now accepts regex patterns in
addition to exact filenames. For example,
alsoExtract = "CMD_sm|CMD_sp" will extract all archive members whose
name matches that regular expression. The expansion is performed against the
archive's file listing using grep(): if an element is a literal match it is
kept as-is; otherwise it is treated as a pattern. Special sentinel values
("similar", "none", NA) are not affected. The expansion happens before
file extraction whether or not the archive was already present before the call.
preProcess now skips downloading when a local copy already exists and
matches the remote version, even if it was never recorded in CHECKSUMS.txt
for the current destinationPath. There is also a new <cacheId>.hash file
placed alongside the cached repository files, that is a simpler mechanism than
the CHECKSUMS.txt, i.e., one file, one hash. This was implemented because there
were too many edge cases that were difficult to handle when there is a single
CHECKSUMS.txt file. Nevertheless, the CHECKSUMS.txt file is still used if
it is present, and the <>..hash file is absent, so a user can manually
place a known CHECKSUMS.txt file into a directory as "the canonical version".
The new pp_remote_hash_check stage fetches
remote metadata (ETag / content-length for HTTP; md5Checksum / size for Google
Drive) and compares against the local file. A stored .hash file is checked
first; if absent, file-size equality is used as a fast proxy and the hash is
persisted for future runs. This fixes the common cluster pattern where an archive
lives in options("reproducible.inputPaths") but has not yet been checksummed
for the current run-specific destinationPath.
Design note — two-layer caching strategy: The remote hash check and the
local CHECKSUMS.txt are complementary, not competing. Remote metadata (ETags,
content-length, Google Drive md5Checksum) solves the bootstrapping problem
— confirming a file is correct before it has ever been checksummed locally.
CHECKSUMS.txt then takes over for all subsequent runs: it requires no network
round-trip, works on compute nodes without internet access, and is
content-addressable (survives URL changes). Note that HTTP ETags are not
universally content-hash-based — many servers derive them from inode + mtime,
making cross-server or post-migration comparisons unreliable — so the remote
check is intentionally a best-effort shortcut rather than a replacement for the
local checksum record.
showCache (when useDBI = FALSE) now has lazy memory caching. This is relevant
for very large caches (e.g., >10,000 entries).
Cache() now lazy-spawns the showCache async background scan against the
cachePath the call actually uses. With the lazy spawn, the first Cache() call
in simInit/spades kicks off the fork, which then runs to completion in
parallel with the simulation; subsequent showCache() calls return in
~1 second. The spawn helper is idempotent (~10 us per call).
New exported helper prepopulateCacheAsync(cachePath) lets workflows
kick off the async scan explicitly (e.g. early in setupProject()) so
the fork has even more wall-clock time to complete.
Lots of new unit tests for new features, and to cover edge cases that were slipping through.
Cache(omitArgs = TRUE) now drops every captured argument from the cache
digest, so the digest depends only on FUN itself (the function value
-- including its body, so source edits still bust the cache) and
.cacheExtra. Useful when a developer wants the cache key to be
insensitive to the function's inputs and pin freshness via .cacheExtra
(e.g. a quoted reference to runtime state) instead of enumerating every
input or every argument to omit. Character-vector
omitArgs = c("a", "b") still works as before.
downloadRemote: the "before" snapshot of destinationPath
taken before evaluating dlFun now uses recursive = TRUE to match the
"after" snapshot. Previously, the non-recursive snapshot omitted files
that were already present in subdirectories of destinationPath, so the
setdiff() of after vs. before classified those pre-existing subdirectory
files as newly created. They then propagated as downloadResults$destFile,
triggered the "already exists at prepInputs
call into the same reproducible.inputPaths/reproducible.destinationPathShared).
Cache(useCloud = ...) now accepts two character values, intended for
separating developer and user roles when sharing a cloud-cache folder:
"push" is equivalent to TRUE (developer role -- bidirectional;
downloads on a cloud hit, uploads on a miss); "pull" is read-only (user
role -- downloads on a cloud hit, never uploads). When "pull" is set
and the local cache already has the object, the Google Drive listing is
not fetched at all (the cloud is consulted only after a local miss). An
invalid character value now errors at the front door of Cache() rather
than silently being treated as FALSE.
prepInputs/.guessAtTargetAndFun: no longer auto-selects OS-injected archive
metadata when targetFile is unspecified. Previously, a Mac-created zip
containing both foo.shp and __MACOSX/._foo.shp could pick the
AppleDouble copy and fail to load. Filtering covers macOS (__MACOSX/*,
._* AppleDouble, .DS_Store) and Windows (Thumbs.db, desktop.ini).
An explicit targetFile pointing at one of these is still honored.
pp_remote_hash_check: now skips URLs that are not http:// or https://.
Previously, file:// URLs would attempt a HEAD-style metadata fetch and then
call makeRemoteHashFile, whose URL-to-filename mapping only strips the
https?:// prefix; on Windows the resulting .hash filename retained file:
and the drive-letter colon, which is not a legal Windows path character.
prepInputs/preProcess: when called with dlFun only (no url,
targetFile, or archive), the file produced by dlFun is now treated as
the source of truth. Previously, runChecksums would still scan
getOption("reproducible.inputPaths") and, with no canonical filename to
look up, Checksums() listed every file in the stash and matched any of
them against the stash's CHECKSUMS.txt. A non-empty match silently
redirected destinationPath to the stash and caused prepInputs to load
an unrelated file (e.g., a previously stashed shapefile instead of the
GADM .rds produced by geodata::gadm()).
prepInputs/preProcess: dlFun = pkg::fn(args) (a function call passed
directly, without quote()-wrapping) is once again kept as a deferred call
object instead of being eagerly evaluated. The previous fix for
dlFun = if (cond) fn else NULL broke this canonical usage by forcing the
lazy promise for all non-quote() expressions. The new logic keeps
function-call expressions (fn(args), pkg::fn(args), pkg:::fn(args))
deferred while still evaluating control-flow expressions and bare symbols.
prepInputsCOG now requires a GeoTiff-style URL extension (.tif, .tiff,
.cog, .gtiff) before attempting a /vsicurl/ read. Previously, any
HTTP(S) URL combined with a spatial subsetting argument would trigger the
fast-path, producing a confusing GDAL Error 4 ... not recognized as being in a supported file format warning when the URL pointed to an archive
(e.g. .zip, .tar.gz).
lockFile now uses checkPath(..., create = TRUE) instead of a silent
dir.create(..., showWarnings = FALSE) when creating the lock-file directory,
so a missing or unwritable cache directory (e.g., on a network filesystem)
produces a clear error rather than a confusing filelock failure.
Fix .listFilesInArchive incorrectly returning an empty file list for zip
archives where archive::archive() reports size = 0 for every entry
(a known metadata-reading issue with certain compression variants such as
Deflate64). When all reported sizes are zero but the archive file itself is
non-empty, all paths are now included rather than being filtered out.
Fix spurious "More than one possible files to load" message (and "Picking the
last one") printed by preProcess even when fun = NA. When the user
explicitly passes fun = NA (meaning: do not load the file into R),
.guessAtTargetAndFun now returns immediately without inspecting or
messaging about the extracted file list.
Fix pp_remote_hash_check incorrectly treating a direct .tif (or other
non-archive) download as an archive when the remote hash matched. The stage
was unconditionally setting ctx$archive <- localFile, which caused
downstream pp_extract to run 7z/unzip on the plain raster file.
Fix: only set ctx$archive when .isArchive(localFile) is non-NULL.
Fix spurious preProcess could not extract the files from the archive error
when files were already present on disk (e.g. extracted earlier in the same
call or found via reproducible.inputPaths). In extractFromArchive, result
was computed from the checkSums table before .checkSumsUpdate() was called,
so freshly-extracted files had no prior entry and NROW(result) == 0 forced the
re-extraction branch even though all(isOK) was TRUE. Fix: compute result
after .checkSumsUpdate() so it reflects current disk state. The error was
non-fatal (caught by the surrounding try()) but printed an alarming message
and wasted effort attempting a zero-file extraction.
Fix Google Drive download failed: HTTP 401 Unauthorized error that occurred
mid-session when downloading multiple tiles via prepInputsWithTiles. The raw
access_token string extracted from the gargle/googledrive token expired
(1-hour TTL) while tiles were being downloaded. Fix: force a gargle token
refresh (via Token2.0$refresh()) before each download_resumable_httr2 call,
and retry once on 401 with a fresh token for both the httr2 and curl code paths.
Fix object 'fun' not found error when Cache(prepInputs, ..., fun = fun, ...) is called
with fun as a local variable name. substitute(fun) captured the symbol rather than
the value; the symbol was then evaluated in the wrong frame (Cache's internal frame, not the
user's). Fix: force the R promise directly (funCaptured <- fun) so resolution happens in
the frame where the promise was created (the user's frame), regardless of call depth.
Fix filelock::lock() "Permission denied" error under high parallelism (30+ workers).
Three root causes: (1) deleting the .lock file after unlock() broke mutex correctness —
workers blocked on fcntl(F_SETLKW) held the old inode's lock while a fresh caller
created a new inode and acquired its own "lock" simultaneously; (2) stale .lock files
owned by root (from a prior sudo/root run) caused EACCES at open(O_RDWR|O_CREAT);
(3) the tryCatch matched on "Permission denied" which is locale-dependent (varies with
LC_MESSAGES) — now matches on "Cannot open lock file" (filelock's fixed C-level prefix).
Fix: stop deleting lock files after release; wrap filelock::lock() in tryCatch with
a 5-attempt retry loop; match on the locale-independent error prefix.
postProcess: when 2 large polygon datasets were provided (from and to), the pre-cropping step
failed as the buffer was not applied. This has been fixed and the buffer now scales with
the size of the polygons.
qs removed as an option for CacheSaveFormat. The user can stay with
declaring options(reproducible.CacheSaveFormat = "qs"), but it will use qs2. qs
is being removed from CRAN;terra::project issues;archive package is not installed or is not
able to deal with the compression algorithm (e.g., Deflate64 from Windows);showCache now uses a custom memoising internally. Large Cache repositories (>500GB) were
slow to showCache. Because it is memoised, this only affects 2nd and subsequent calls
in an R session. The first will still be slower.CacheGeo handles a few more cases;isGoogleDriveDirectory handles more cases correctly (i.e., works now if it is a Google ID);Cache so it is simpler and more robust. The main function is now 200
lines, instead of almost 700; internals all cleaner and maintainable;reproducible.leaveOnDisk which is only relevant for postProcess with
objects that are sometimes disk-backed and sometimes memory-based (like SpatRaster or Raster).
The default terra and raster behaviour (which creates unpredictable behaviour, with
the transient compute context being the trigger for one behaviour or another)
for these was creating unnecessarily slow postProcessing by bringing the objects
to memory, sometimes, and this in turn led to unnecessarily slow Cache behaviour because
terra::wrap is very slow for large SpatRaster objects. See ?reproducibleOptions;showCache (when useDBI() is FALSE) now uses a type of internal memoising, so it
is much faster for large cache databases, after a first time called.dot functions and are now exported
e.g., for use in other packages;postProcess pipeline where terra functions are used. Now all
functions are memory-safe, so will not bring the data to memory;Cache previously would bring some objects to memory that don't need to, e.g., SpatRaster from
terra. These are now left on disk, as part of the Cache pipeline;prepInputsWithTiles: new function. prepInputs can now pass through a different sub-function,
prepInputsWithTiles, which can deal with (i.e., upload and download) remote files that
are tiled. See ?prepInputsWithTiles;cacheChaining; in cases where there are >1 Cache call within a single
function, the cacheChaining will digest the containing function (via sys.function(-1))
to determine whether it is stable between calls. If it is unchanged, then a series of
Cache calls can be eligible for chaining, meaning where each subsequent Cache call
refrains from digesting an object that was the outcome of a prior Cache call, within
the same -- and unchanged from the previous time -- function. This can dramatically
speed up computations when Cache needs to digest large objects and the digest step
takes a long time;format replaces cacheSaveFormat as an argument so individual Cache calls can switch backend;
this can be useful when e.g., qs (which tends to be faster and smaller files) does not work
for all types of objects e.g., xgboost.CacheGeo added new cases that are able to be used.Cache;digest of the arguments. These changes are not backwards compatible. Details next.terra::SpatRaster class;digest changes include the following fixes:
CacheDigest which is used within Cache did not digest the names of the
list of arguments passed. This did not affect .robustDigest of a normal list, which
keeps the names intact.cacheId in the filename; now they will have the
cacheId prefixed on the file (so they sort alongside the main cache file). This
cacheId prefix is removed on recovery from the cache, overwriting
any files with the same name.functionName from a function had several edge cases did not work; these now workoptions(reproducible.digestV3 = FALSE). This will keep the behaviour where
lists are digested without their names for CacheDigest and Cache.
This will not affect the file-backed objects changes described above, which will ignore this
option.useMemoise would work with file-backed objects, but only if the file-backed object
did not change after the caching (the pointer to the file was intact, but the file changed).
Now, memoising will copy file-backed information from disk
each time it "retrieves a file-backed object from memory". This will result in slower
memoising than previously. However, it will be robust to downstream changes to the file.purgeChecksums to allow user to manually purge a file from CHECKSUMS.txtreproducible.useCacheV3: default = TRUE to use the new Cache source codereproducible.digestV3: default = TRUE to use the new Cache digest algorithmsmaskTo can now use a SpatRaster for the maskPackedStatExtent class, releasing it for terra; reproducible
now uses a PackedStatExtent2; this will eventually be replaced by the terra
PackedStatExtent when this conflict is removed;.robustDigest method for "character" no longer will evaluate character strings as files, by
default. A user can force the old behaviour with options(reproducible.testCharacterAsFile = TRUE)).
This created unwanted, and inexplicable hanging of a computer, e.g.,
in a data.frame with thousands of rows of a character vector that represent filenames that
existed, but their content was not expected to be digested;
it would take possibly hours to digest. To digest files, user must explicitly coerce to
"Path" with asPath(x), or fs::as_fs_path as the previous hanging behaviour was surprising
and could not be easily diagnosed;url in prepInputs can now point to a directory; use alsoExtract to pick files by regular expression;remapFileNames();terra::project() arguments use_gdal and by_util through projectTo() to terra::project();preProcess, supplying a user_agent (which happens automatically within the function) would cause the download to fail; now there is some redundancy within dlGeneric that will retry without a user_agent if it detects this issue;cli; instead of custom messaging functions;crayon dependency;httr --> convert to use httr2 for some pieces; transition not complete;cacheId, e.g., in Cache(..., cacheId = "myCacheItem"), myCacheItem was not used. Fixed.prepInputs(..., fun = sf::st_read) now works as expected ... like prepInputs(..., fun = "sf::st_read")postProcessTo that use sf::gdal_utils directly. These are still experimental and will only be activated with options("reproducible.gdalwarp" = TRUE)gdalMask has changed default for "touches". Now has equivalent for terra::mask(..., touches = TRUE), using "-wo CUTLINE_ALL_TOUCHED=TRUE"gdalProject now uses 2 threads, setting "-wo NUM_THREADS=2"; can be changed by user with options("reproducible.gdalwarpThreads" = X); see ?reproducibleOptionsgdal* functions now address datatype issuesgdal* defaults to FLT8S if datatype not passedmakeRelative, makeAbsolute and similar have been created to ease many issues encountered in preProcessshowSimilar (e.g., options(reproducible.showSimilar = 1)) now preferentially shows the most recent item in cache if there are several with equivalent matching.Cache and prepInputs families; functions are highlighted with a different colour; indent level reflects nesting of both Cache and prepInputs, so it is easier to identify which message goes with which function call.preProcess is a lot faster now for large numbers of files; uses CHECKSUMS more effectively and fewer timesretry now captures its expr so it doesn't need a quote; is like try now.showSimilar mechanisms now returns the most recent, if there are >1 similar that are equivalently similargoogledrive for e.g., large files on spotting connections, instructions for using gdown are providedshowCache, clearCache now have extra arguments fun, cacheId, and ... now can take any arbitrary tag = value pair. The cacheId argument will be very fast if a user is not using useDBI() is FALSE..wrap and .unwrap can now deal with SpatVectorCollection (a terra class that does not have a wrap/unwrap method in terra)spooky or fastdigest were not stable for integers and factors. There is now a work around in .robustDigest that stabilizes these by expanding them from their ALTREP representation first. Since they will be saved and recovered anyway, this will have little effect..wrap and .unwrap are becoming more mature and can handle many more classes effectively. Methods can still be written, if needed.cacheSaveFormat = "qs", which previously was not reliable especially for environments. With all recent changes to .wrap and .unwrap, these appear stable now and should be able to be used for environments.switchDataType can now correctly switch between gdal formats and terrafastdigest was removed from CRAN and so is removed from here.SpatRaster objectsisUpdated() to determine whether a cached object has been updated;makeRelative() is now exported for use downstream (e.g., SpaDES.core);getRelative() and normPathRel() for improved symlink handling (#362);Cache with the function named instead of just cacheIdprepInputs: minor changesChecksums dealt with, so fewer unneeded downloadswrapSpatRaster (wrap for file-backed spatRaster objects) fixes for more edge casespostProcessTo can now use sf::gdal_utils for the case of from is a gridded object and to is a polygon vector. This appears to be between 2x and 10x faster in tests.postProcessTo does a pre-crop (with buffer) to make the projectTo faster. When both from and to are vector objects, this pre-crop appears to create slivers in some cases. This step is now skipped for these cases.Cache can now deal with unnamed functions, e.g., Cache((function(x) x)(1)). It will be refered to as "headless".terra would fail if internet was unavailable, even when internet is not necessary, due to needing to retrieve projection information. Many cases where this happens will now divert to use sf.Cache can now skip calculating objSize, which can take a non-trivial amount of time for large, complicated objects; see reproducibleOptions()Filenames for some classes returned ""; now returns NULL so character vectors are only pointers to filesquick argument is specified was failing, always creating the same object; fixed with #PR368useDBI was incorrectly used if a user had set the option prior to package loading. Now works as expected.preProcess deals better with more cases of nested paths in archives.inputPathsgetRVersion() <= "XXX"assessDataType for categorical (factor) Raster and SpatRasterround with R > 4.3.1; now a primitive, that does method dispatch. Failure was identified with unit tests, by Luke Tierney who was making the change in base::round..unwrap call, and missing check in preProcess, when targetFilePath was NULL.Copy & new .wrap, .unwrap generics and methods to wrap classes that don't save well to disk as is. This uses the name similar to terra::wrap, but with slight differences internally to allow for SpatRaster objects who are file-backed and must have their files moved when they are unwrapped.loadFiles updated for more caseswithr throughout testing for cleaning upFilename added, including for Path classCache(..., useCloud = TRUE) had many cases that were not working; known cases are now working. Also, now file from file-backed cases are now placed inside the cacheOutputs folder rather than inside a separate folder (used to be "rasters")reproducible.useFuture now defaults to "multisession"data.table development branch (#314)data.table::setattr to deal with "modified compiler constants" issue that was detected during CRAN checkspreProcess failed when googledrive url filename could be found, but destinationPath was not "."normPath had different behaviour on *nix-alikes and Windows. Now it is the same.SpatRaster objects if saved to a specific, non relative (to getwd()) path would not be recovered correctly (#316)prepInputs and family.Cache via options(reproducible.useDBI = FALSE) is single data files with the same basename as the cached object, i.e., with the same cacheId in the file name. This is a replacement for RSQLite and will likely become the default in the next release. This approach makes cloud caching easier as all metadata are available in small binary files for each cached object. This is simpler, faster and creates far fewer package dependencies (now 11 recursive; before 27 recursive). If a user has DBI and RSQLite installed, then the backend will default to use these currently, i.e., the previous behaviour. The user can change the backend without loss of Cache data.raster and sp to Suggests; no more internal functions use these. User can still work with Raster and sp class objects as before.preProcess can now handle Google docs files, if type = ... is passed.postProcess now uses terra and sf internally (with #253) throughout the family of postProcess functions. The previous *Input and *Output functions now redirect to the new *To* functions. These are faster, more stable, and cover vastly more cases than the previous *Inputs family. The old backends no longer work as before.raster to terra: maxFn, minFn, rasterRead.dealWithClass and .dealWithClassOnRecovery are now exported generics, with several methods here, notably, list, environment, defaultraster to terra transition (e.g. studyAreaName can deal with SpatVector)prepInputs now deals with archives that have sub-folder structure are now dealt with correctly in all examples and tests esp. #181.prepInputs can now deal with .gdb files. Though, it is limited to sf out of the box, so e.g., Raster layers inside gdb files are not supported (yet?). User can pass fun = NA to not try to load it, but at least have the .gdb file locally on disk.hardLinkOrCopy now uses linkOrCopy(symlink = FALSE); more cases dealt with especially nested directory structures that do not exist in the to.terra and sf.preProcess had multiple changes. The following now work: archives with subfolders, archives with subfolders with identical basenames (different dirnames), gdb files, other files where targetFile is a directory.preProcess for minor efficiency gains, edge cases, code cleaningCacheGeo that weaves together prepInputs and Cache to create a geo-spatial caching. See help and examples.maskTo now allows touches arg for terra::maskSpatial class is also "fixed" in fixErrorsInprepInputs and preProcess now capture dlFun, so user can pass unquoted dlFunCopy method for SpatRaster, with and without file-backingCache(..., useCloud = TRUE) reworked so appears to be more robust than previously.maskTo now works even if to is larger than fromnetCDF works with prepInputs; thanks to user nbsmokee with PR #300.prepInputs and family, the user will have to install terra and sf at a minimum.terra, sf are in Suggestsfasterize, fpCompare, magrittrSuggests: raster, sp, rlangreproducible no longer installs DBI, nor does it use RSQLite. All cache repositories database files will be in binary individual files in the cacheOutputs file. If a user has DBI and a SQLite engine, then the previous behaviour will be used.reproducible.useNewDigestAlgorithm is not longer an option as the old algorithms do not work reliably.assessDataTypeGDAL(), clearStubArtifacts(),digestRasterLayer2(); evalArgsOnly(); .getSourceURL(); .getTargetCRS(); .checkSums(), .groupedMessage(); .checkForAuxililaryFiles()option("reproducible.polygonShortcut") removed.basename renamed to basename2Cache was incorrectly dealing with environment and environment-like objects. Since some objects, e.g., Spat* objects in terra, must be wrapped prior to saving, environments must be scanned for these classes of objects prior to saving. This previously only occurred for list objects;SpaDES.core, there were some cases where the Cache was failing as it could not find the module name;postProcess (using raster and sp) to postProcessTo, some cases are falling through the cracks; these have being addressed.Cache now captures the first argument passed to it without evaluating it, so Cache(rnorm(1)) now works as expected.Cache now works with base pipe |> (with R >= 4.1).FUN passed to Cache, there should be no problems with previous cache databases being successfully recovered.Cache internals so that digesting is more accurate, as the correct methods for functions are more accurately found, objects within functions are more precisely evaluated.() in DESCRIPTION for functions;\value in .Rd files for exported methods (structure, the class, the output meaning);postProcess now also checks resolution when assessing whether to projectprepInputs has an internal Cache call for loading the object into memory; this was incorrectly evaluating all files if there were more than one file downloaded and extracted. This resulted in cases, e.g. shapefiles, being considered identical if they had the identical geometries, even if their data were different. This is fixed now as it uses the digest of all files extracted.digestPathContent from Cacheoptions("reproducible.useGDAL") is now deprecated; the package is moving towards terra.postProcessTo to deal with changes in GDAL/PROJ/GEOS (#253; @rsbivand)gdalUtilities, gdalUtils, and rgeos from Suggestsraster and terra, because previous versions were causing collisions.terra and sf are used throughoutprepInputs can now take fun as a quoted expression on x, the object loaded by dlFun in preProcesspreProcess arg dlFun can now be a quoted expressionobjSize; now is primarily a wrapper around lobstr::obj_size, but has an option to get more detail for lists and environments..robustDigest now deals explicitly with numerics, which digest differently on different OSs. Namely, they get rounded prior to digesting. Through trial and error, it was found that setting options("reproducible.digestDigits" = 7) was sufficient for all known cases. Rounding to deeper than 7 decimal places was insufficient. There are also new methods for language, integer, data.frame (which does each column one at a time to address the numeric issue)postProcess called postProcessTo. This will eventually replace postProcess as it is much faster in all cases and simpler code base thanks to the fantastic work of Robert Hijmans (terra) and all the upstream work that terra relies onCache call. If there is a delay after this message, then it is the code following the Cache call that is (silently) slow.retry can now return a named list for the exprBetween, which allows for more than one object to be modified between retries..robustDigest was removing Cache attributes from objects under many conditions, when it should have left them there. It is unclear what the issues were, as this would likely not have impacted Cache. Now these attributes are left on.data.table objects appear to not be recovered correctly from disk (e.g., from Cache repository. We have added data.table::copy when recovering from Cache repositoryclearCache and cc did not correctly remove file-backed raster files (when not clearing whole CacheRepo); this may have resulted in a proliferation of files, each a filename with an underscore and a new higher number. This fix should eliminate this problem.getGDALVersion() (#239)maskInputs() when not passing rasterToMatch.isna.SpatialFix when using postProcess.quosurelwgeom now a suggested packageterra class objects can now be correctly saved and recovered by CachefixErrors can now distinguish testValidity = NA meaning don't fix errors and testValidity = FALSE run buffering which fixes many errors, but don't test whether there are any invalid polygons first (maybe slow), or testValidity = TRUE meaning test for validity, then if some are invalid, then run buffer.reproducible.useNewDigestAlgorithm = 2 which will have user visible changes. To keep old behaviour, set options(reproducible.useNewDigestAlgorithm = 1)options(reproducible.showSimilar) is set. It is now more compact e.g., 3 lines instead of 5.sf methods to studyAreaNameCache returns; i.e., a 2nd time through a Cache would return a cached copy, when some of the arguments were different. It occurred for when the differences were in unnamed arguments only.reproducible will be slowly changing the defaults for vector GIS datasets from the sp package to the sf package.
There is a large user-visible change that will come (in the next release), which will cause prepInputs to read .shp files with sf::st_read instead of raster::shapefile, as it is much faster. To change now, set options("reproducible.shapefileRead" = "sf::st_read")
fun in prepInputs for shapefiles (.shp) is now sf::st_read if the system has sf installed. This can be overridden with options("reproducible.shapefileRead" = "raster::shapefile"), and this is indicated with a message at the moment this is occurring, as it will cause different behaviour.quick argument in Cache can now be a character vector, allowing individual character arguments to be digested as character vectors and others to be digested as files located at the specified path as represented by the character vector.objSize previously included objects in namespaces, baseenv and emptyenv, so it was generally too large. Now uses the same criteria as pryr::object_sizeunzip missing (thanks to @CeresBarros #202)7z.exe on Windows if the object is larger than 2GB, if can't find unzip.fun argument in prepInputs and family can now be a quoted expression.archive argument in prepInputs can now be NA which means to treat the file downloaded not as an archive, even if it has a .zip file extensionprepInputspostProcess especially for very large objects (>5GB tested). Previously, it was running many fixErrors calls; now only calls fixErrors on fail of the proximate call (e.g., st_crop or whatever)retry now has a new argument exprBetween to allow for doing something after the fail (for example, if an operation fails, e.g., st_crop, then run fixErrors, then return back to st_crop for the retry)Cache now has MUCH better nested levels detection, with messaging... and control of how deep the Caching goes seems good, via useCache = 2 will only Cache 2 levels in...archive argument in prepInputs family can now be NA ... meaning do not try to unzip even if it is a .zip file or other standard archive extensiongdb.zip files (e.g., a file with a .zip extension, but that should not be opened with an unzip-type program) can now be opened with prepInputs(url = "whateverUrl", archive = NA, fun = "sf::st_read")fun argument in prepInputs can now be a quoted function call.preProcess now does a better job with large archives that can't be correctly handled with the default zip and unzip with R, by trying system2 calls to possible 7z.exe or other options on Linux-alikes.Copy generic no longer has fileBackedDir argument. It is now passed through with the .... This was creating a bug with some cases where fileBackedDir was not being correctly executed.fixErrors() now better handles sf polygons with mixed geometries that include points.CachewriteOutputs.Raster attempted to change datatype of Raster class objects using the setReplacement dataType<-, without subsequently writing to disk via writeRaster. This created bad values in the Raster* object. This now performs a writeRaster if there is a datatype passed to writeOutputs e.g., through prepInputs or postProcess.updateSlotFilename has many more tests.prepInputs(..., fun = NA) now is the correct specification for "do not load object into R". This essentially replicates preProcess with same arguments.Copy did not correctly copy RasterStacks when some of the RasterLayer objects were in memory, some on disk; raster::fromDisk returned FALSE in those cases, so Copy didn't occur on the file-backed layer files. Using Filenames instead to determine if there are any files that need copying.options("reproducible.useNewDigestAlgorithm" = 2)options("reproducible.polygonShortcut" = FALSE) as there were still too many edge cases that were not covered.RasterStack objects with a single file (thus acting like a RasterBrick) are now handled correctly by Cache and prepInputs families, especially with new options("reproducible.useNewDigestAlgorithm" = 2), though in tests, it worked with default alsoRSQLite now uses a RNG during dbAppend; this affected 2 tests (#185).rgeospaddedFloatToChar to reproducible from SpaDES.core.%>% code from magrittr to allow the cached alternative, %C%. With new magrittr pipe now in compiled source code, more of the legacy code is required here.reproducible.messageColourPrepInputs for all prepInputs functions; reproducible.messageColourCache for all Cache functions; and reproducible.messageColourQuestion for questions that require user input. Defaults are cyan, blue and green respectively. These are user-visible colour changes.Cache cases where a file.link is used instead of saving.options(reproducible.verbose = 0) will turn off almost all messaging.postProcess and family now have filename2 = NULL as the default, so not saved to disk. This is a change.verbose is now an argument throughout, whose default is getOption(reproducible.verbose), which is set by default to 1. Thus, individual function calls can be more or less verbose, or the whole session via option.RasterStack objects were not correctly saved to disk under some conditions in postProcess - fixedpostProcess now uses a simpler single call to gdalwarp, if available, for RasterLayer class to accomplish cropInputs, projectInputs, maskInputs, and writeOutputs all at once. This should be faster, simpler and, perhaps, more stable. It will only be invoked if the RasterLayer is too large to fit into RAM. To force it to be used the user must set useGDAL = "force" in prepInputs or postProcess or globally with options("reproducible.useGDAL" = "force")postProcess when using the new gdalwarp, has better persistence of colour table, and NA values as these are kept with better reliabilityCache now works as expected (e.g., with parallel processing, it will avoid collisions) with SQLite thanks to suggestion here: https://stackoverflow.com/a/44445010Raster class objects to account for more of the metadata (including the colortable). This will change the digest value of all Raster layers, causing re-run of CacheRequire, pkgDep, trimVersionNumber, normPath, checkPath that were moved to Require package. For backwards compatibility, these are imported and reexportedfile.move used to rename/copy files across disks (a situation where file.rename would fail)DBI type functions now have default cachePath of getOption("reproducible.cachePath")Cache(prepInputs, ... on a file-backed Raster* class object now gives the non-Cache repository folder as the filename(returnRaster). Previously, the return object would contain the cache repository as the folder for the file-backed Raster*backports, memoise, quickPlot, R.utils, remotes, tools, and versions; moved to Suggests: fastdigest, gdalUtils, googledrive, httr, qs, rgdal, sf, testthat; added: Require. Now there are 12 non-base packages listed in Imports. This is down from 31 prior to Ver 1.0.0.file.link not file.symlink for saveToCache. This would have resulted in C Stack overflow errors due to missing original file in the file.symlinkunzip when extracting large (>= 4GB) files (#145, @tati-micheletti)projectInputs when converting to longlat projections, setMinMax for gdalwarp resultsFilenames now consistently returns a character vector (#149)file.link does not succeed.raster) are updated.options('reproducible.cacheSaveFormat') on the fly; cache will look for the file by cacheId and write it using options('reproducible.cacheSaveFormat').
If it is in another format, Cache will load it and resave it with the new format. Experimental still.Copy methods for refClass objects, SQLite and moved environment method into ANY as it would be dispatched for unknown classes that inherit from environment, of which there are many and this should be interceptedRequire can now handle minimum version numbers, e.g., Require("bit (>=1.1-15.2)"); this can be worked into downstream tools. Still experimental.file.link or file.symlink if an existing Cache entry with identical output exists and it is large (currently 1e6 bytes); this will save disk space.elapsedTimeDigest, elapsedTimeFirstRun, and elapsedTimeLoad, respectively.preProcess). Includes 2 new functions, tempdir2 and tempfile2 for use with reproducible packagereproducible.tempPath, which is used for the new control of temporary files. Defaults to file.path(tempdir(), "reproducible"). This feature was requested to help manage large amounts of temporary objects that were not being easily and automatically cleaneddrv and conn; user may need to manually call movedCache if cache is not responding correctly.
File-backed Rasters are automatically updated with new paths.Raster* will have their filenames updated on the fly during a Cache recovery.
User doesn't need to do anything.postProcess now will perform simple tests and skip cropInputs and projectInputs with a message if it can, rather than using Cache to "skip". This should speed up postProcess in many cases.Cache has change. Now, cacheId is shown in all cases, making it easier to identify specific items in the cache.Copy only creates a temporary directory for filebacked rasters; previously any Copy command was creating a temporary directory, regardless of whether it was neededcropInputs.spatialObjects had a bug when object was a large non-Raster class.cropInputs may have failed due to "self intersection" error when x was a SpatialPolygons* object; now catches error, runs fixErrors and retries crop.
Great reprex by @tati-micheletti. Fixed in commit 89e652ef111af7de91a17a613c66312c1b848847 .Filenames bugfix related to RasterBrickprepInputs does a better job of keeping all temporary files in a temporary folder; and cleans up after itself better.prepInputs now will not show message that it is loading object into R if fun = NULL (#135).options("reproducible.useDBI" = FALSE)DBI package directly, without archivist. This has much improved speed.options("reproducible.cacheSaveFormat"). This can be either rds (default) or qs. All cached objects will be saved with this format. Previously it was rda.qs::qsave. In many cases, this has much improved speed and file sizes compared to rds; however, testing across a wide range of conditions will occur before it becomes the default.... because Cache is now much faster, the default is to turn memoising off, via options("reproducible.useMemoise" = FALSE).
In cases of large objects, memoising should still be faster, so user can still activate it, setting the option to TRUE.postProcess arg useGDAL can now take "force" as the default behaviour is to not use GDAL if the problem can fit into RAM and sf or raster tools will be faster than GDAL toolsuseCloud argument in Cache and family has slightly modified functionality (see ?Cache new section useCloud) and now has more tests including edge cases, such as useCloud = TRUE, useCache = 'overwrite'.
The cloud version now will also follow the "overwrite" command.archivist; moved to Suggests.bitops, dplyr, fasterize, flock, git2r, lubridate, RcppArmadillo, RCurl and tidyselect. Some of these went to Suggests.postProcess calls that use GDAL made more robust (including #93).dplyr as a direct dependency. It is still an indirect dependency through DiagrammeRreproducible.showSimilarDepth allows for a deeper assessment of nested lists for differences between the nearest cached object and the present object. This greater depth may allow more fine tuned understanding of why an object is not correctly cachingoptions("reproducible.futurePlan") to something other than FALSE, then it will show download progress if the file is "large".googledrive v 1.0.0 (#119)pkgDep2, a new convenience function to get the dependencies of the "first order" dependencies.useCache, used in many functions (incl Cache, postProcess) can now be numeric, a qualitative indicator of "how deep" nested Cache calls should set useCache = TRUE -- implemented as 1 or 2 in postProcess currently. See ?CachepkgDep was becoming unreliable for unknown reasons. It has been reimplemented, much faster, without memoising. The speed gains should be immediately noticeable (6 second to 0.1 second for pkgDep("reproducible"))retry to use exponential backoff when attempting to access online resources (#121)useCloud and cloudFolderID. This is a new approach to cloud caching.
It has been tested with file backed RasterLayer, RasterStack and RasterBrick and all normal R objects.
It will not work for any other class of disk-backed files, e.g., ff or bigmatrix, nor is it likely to work for R6 class objects.Cache, i.e., useCache and cloudFolderIDdownloadData from Google Drive now protects against HTTP2 error by capturing error and retrying.
This is a curl issue for interrupted connections.rcnst errors on R-devel, tested using devtools::check(env_vars = list("R_COMPILE_PKGS"=1, "R_JIT_STRATEGY"=4, "R_CHECK_CONSTANTS"=5))cacheRepo: getArtifact, getCacheId, getUserTagsretry, a new function, wraps try with an explicit attempt to retry the same code upon error.
Useful for flaky functions, such as googldrive::drive_download which sometimes fails due to curl HTTP2 error.Rcpp functionality as the functions were no longer faster than their R base alternatives.prepInputs was not correctly passing useCachecropInputs was reprojecting extent of y as a time saving approach, but this was incorrect if studyArea is a SpatialPolygon that is not close to filling the extent. It now reprojects studyArea directly which will be slower, but correct. (#93)CHECKSUMS.txt should now be ordered consistently across operating systems (note: base::order will not succeed in doing this --> now using .orderDotsUnderscoreFirst)cloudSyncCache has a new argument: cacheIds. Now user can control entries by cacheId, so can delete/upload individual objects by cacheIdpostProcess family for sf class objectscloudCache bugfixes for more casestibble from Imports as it's no longer being used%>% pipe that was long ago deprecated. User should use %C% if they want a pipe that is Cache-aware. See examples.options descriptions now in reproducible, see ?reproducibleOptionscacheRepo and options("reproducible.cachePath") can take a vector of paths. Similar to how .libPaths() works for libraries, Cache will search first in the first entry in the cacheRepo, then the second etc. until it finds an entry. It will only write to the first entry.options("reproducible.useCache" = "devMode"). The point of this mode is to facilitate using the Cache when functions and datasets are continually in flux, and old Cache entries are likely stale very often. In devMode, the cache mechanism will work as normal if the Cache call is the first time for a function OR if it successfully finds a copy in the cache based on the normal Cache mechanism. It differs from the normal Cache if the Cache call does not find a copy in the cacheRepo, but it does find an entry that matches based on userTags. In this case, it will delete the old entry in the cacheRepo (identified based on matching userTags), then continue with normal Cache. For this to work correctly, userTags must be unique for each function call. This should be used with caution as it is still experimental.options("reproducible.useNewDigestAlgorithm" = FALSE). There is a message of this change on package load.cloud* functions, especially cloudCache which allows sharing of Cache among collaborators. Currently only works with googledriveassessDataType to consolidate assessDataTypeGDAL and assessDataType into single function (#71, @ianmseddy)cc: new function -- a shortcut for some commonly used options for clearCache()prepInputs to handle .rar archives, on systems with correct binaries to deal with them (#86, @tati-micheletti)fastdigest::fastdigest as it is not return the identical hash across operating systemsprepInputs on GIS objects that don't use raster::raster to load object were skipping postProcess. Fixed.prepInputs would cause virtually all entries in CHECKSUMS.txt to be deleted. 2 cases where this happened were identified and corrected.data.table class objects would give an error sometimes due to use of attr(DT). Internally, attributes are now added with data.table::setattr to deal with this.gdalwarp from prostProcess now correctly matches extent (#73, @tati-micheletti)preProcess (#92, @tati-micheletti)remotes to Imports and removed devtoolsNew value possible for options(reproducible.useCache = 'overwrite'), which
allows use of Cache in cases where the function call has an entry in the cacheRepo,
will purge it and add the output of the current call instead.
New option reproducible.inputPaths (default NULL) and reproducible.inputPathsRecursive
(default FALSE), which will be used in prepInputs as possible directory sources
(searched recursively or not) for files being downloaded/extracted/prepared.
This allows the using of local copies of files in (an)other location(s) instead
of downloading them. If local location does not have the required files,
it will proceed to download so there is little cost in setting this option.
If files do exist on local system, the function will attempt to use a hardlink before making a copy.
dlGoogle() now sets options(httr_oob_default = TRUE) if using Rstudio Server.
Files in CHECKSUMS now sorted alphabetically.
Checksums can now have a CHECKSUMS.txt file located in a different place than the destinationPath
Attempt to select raster resampling method based on raster type if no method supplied (#63, @ianmseddy)
projectInputs
new function assessDataTypeGDAL, used in postProcess, to identify smallest datatype for large Raster* objects passed to GDAL system call
Raster objects, enact gdalwarp system call if raster::canProcessInMemory(x,4) = FALSE for faster and memory-safe processingRaster objects, including factor rastersextractFromArchive for large (>2GB) zip files.
In the R help manual, unzip fails for zip files >2GB.
This uses a system call if the zip file is too large and fails using base::unzip.raster::getData issues.Cache() when deeply nested, due to grep(sys.calls(), ...) that would take long and hang.preProcess(url = NULL) (#65, @tati-micheletti)clearCache (#67), especially for large Raster objects that are stored as binary R files (i.e., .rda)raster package changes in development version of raster packageraster::projectRaster.robustDigest now does not include Cache-added attributespreProcess() (#68, @tati-micheletti)future to Suggests.future for Cache saving to SQLite database, via options("reproducible.futurePlan"), if the future package is installed. This is FALSE by default.do.call function is Cached, previously, it would be labelled in the database as do.call. Now it attempts to extract the actual function being called by the do.call. Messaging is similarly changed.reproducible.ask, logical, indicating whether clearCache should ask for deletions when in an interactive sessionprepInputs, preProcess and downloadFile now have dlFun, to pass a custom function for downloading (e.g., "raster::getData")prepInputs will automatically use readRDS if the file is a .rds.prepInputs will return a list if fun = "base::load", with a message; can still pass an envir to obtain standard behaviour of base::load.clearCache - new argument ask.assessDataType, used in postProcess, to identify smallest datatype for Raster* objects, if user does not pass an explicit datatype in prepInputs or postProcess (#39, @CeresBarros).git2r update (@stewid, #36)..prepareRasterBackedFile -- now will postpend an incremented numeric to a cached copy of a file-backed Raster object, if it already exists. This mirrors the behaviour of the .rda file. Previously, if two Cache events returned the same file name backing a Raster object, even if the content was different, it would allow the same file name. If either cached object was deleted, therefore, it would cause the other one to break as its file-backing would be missing.spades.XXX and should have been reproducible.XXX.copyFile did not perform correctly under all cases; now better handling of these cases, often sending to file.copy (slower, but more reliable)extractFromArchive needed a new Checksum function call under some circumstancesextractFromArchive -- when dealing with nested zips, not all args were passed in recursively (#37, @CeresBarros)prepInputs -- arguments that were same as Cache were not being correctly passed internally to Cache, and if wrapped in Cache, it was not passed into prepInputs. Fixed..prepareFileBackedRaster was failing in some cases (specifically if it was inside a do.call) (#40, @CeresBarros).Cache was failing under some cases of Cache(do.call, ...). Fixed.Cache -- when arguments to Cache were the same as the arguments in FUN, Cache would "take" them. Now, they are correctly passed to the FUN.preProcess -- writing to checksums may have produced a warning if CHECKSUMS.txt was not present. Now it does not.new functions:
convertPaths and convertRasterPaths to assist with renaming moved files.prepInputs -- new features
alsoExtract now has more options (NULL, NA, "similar") and defaults to extracting all files in an archive (NULL).postProcess altogether if no studyArea or rasterToMatch. Previously, this would invoke Cache even if there was nothing to postProcess.copyFile correctly handles directory names containing spaces.makeMemoisable fixed to handle additional edge cases.new functions:
prepInputs to aid in data downloading and preparation problems, solved in a reproducible, Cache-aware way.postProcess which is a wrapper for sequences of several other new functions (cropInputs, fixErrors, projectInputs, maskInputs, writeOutputs, and determineFilename)downloadFile can handle Google Drive and ftp/http(s) fileszipCache and mergeCachecompareNA does comparisons with NA as a possible value e.g., compareNA(c(1,NA), c(2, NA)) returns FALSE, TRUECache -- new features:
showSimilar, verbose which can help with debugginguseCache which allows turning caching on and off at a high level (e.g., options("useCache"))cacheId which allows user to hard code a result from a CachedigestPathContent --> quick, compareRasterFileLength --> lengthCache function calls, unless explicitly set on the inner functionsuserTags added automatically to cache entries so much more powerful searching via showCache(userTags="something")checksums now returns a data.table with the same columns whether write = TRUE or write = FALSE.
clearCache and showCache now give messages and require user intervention if request to clearCache would be large quantities of data deleted
memoise::memoise now used on 3rd run through an identical Cache call, dramatically speeding up in most cases
new options: reproducible.cachePath, reproducible.quick, reproducible.useMemoise, reproducible.useCache, reproducible.useragent, reproducible.verbose
asPath has a new argument indicating how deep should the path be considered when included in caching (only relevant when quick = TRUE)
New vignette on using Cache
Cache is parallel-safe, meaning there are tryCatch around every attempt at writing to SQLite database so it can be used safely on multi-threaded machines
bug fixes, unit tests, more imports for packages e.g., stats
updates for R 3.6.0 compact storage of sequence vectors
experimental pipes (%>%, %C%) and assign %<%
several performance enhancements
mergeCache: a new function to merge two different Cache repositories
memoise::memoise is now used on loadFromLocalRepo, meaning that the 3rd time Cache() is run on the same arguments (and the 2nd time in a session), the returned Cache will be from a RAM object via memoise. To stop this behaviour and use only disk-based Caching, set options(reproducible.useMemoise = FALSE) .
Cache assign -- %<% can be used instead of normal assign, equivalent to lhs <- Cache(rhs).
new option: reproducible.verbose, set to FALSE by default, but if set to true may help understand caching behaviour, especially for complex highly nested code.
all options now described in ?reproducible.
All Cache arguments other than FUN and ... will now propagate to internal, nested Cache calls, if they are not specified explicitly in each of the inner Cache calls.
Cached pipe operator %C% -- use to begin a pipe sequence, e.g., Cache() %C% ...
Cache arg sideEffect can now be a path
Cache arg digestPathContent default changed from FALSE (was for speed) to TRUE (for content accuracy)
New function, searchFull, which shows the full search path, known alternatively as "scope", or "binding environments". It is where R will search for a function when requested by a user.
Uses memoise::memoise for several functions (loadFromLocalRepo, pkgDep, package_dependencies, available.packages) for speed -- will impact memory at the expense of speed.
New Require function
require on those 20 packages, but require does not check for dependencies and deal with them if missing: it just errors. This speed should be fast enough for many purposes.remove dplyr from Imports
Add RCurl to Imports
change name of digestRaster to .digestRaster
digestRaster affecting in-memory rastersrgdal to SuggestsSpaDES package.