A project often reads data from many places: files on the web, Google Drive, other people’s folders. Sooner or later someone asks “what does this project need?”: to cite the data, to check a source has not changed, or to run the project on a machine with no internet.
reproducible answers that question with a download ledger.
Every call to prepInputs() or preProcess()
writes one record: the address, the file it was meant to give, and where
it was put. A call with no url that reads a file already on
disk is recorded too, with local:<path> in place of
the address. This vignette shows how to read the ledger, how it behaves
with Cache(), and how to go from the ledger to the list of
files to copy to another machine.
prepInputsLog() returns every access of the current R
session. clearUrlLog() empties it. The examples here use
local files given as file:// urls, so they run without
internet; a https:// or Google Drive url is recorded the
same way.
library(reproducible)
src <- file.path(tempdir(), "sources")
dir.create(src, showWarnings = FALSE)
writeLines("a,b\n1,2", file.path(src, "table.csv"))
writeLines("x,y\n3,4", file.path(src, "other.csv"))
url1 <- paste0("file://", file.path(src, "table.csv"))
url2 <- paste0("file://", file.path(src, "other.csv"))
inputs <- file.path(tempdir(), "inputs")
clearUrlLog()
tab <- prepInputs(url = url1, targetFile = "table.csv", destinationPath = inputs, fun = NA)
data.table::rbindlist(prepInputsLog(), fill = TRUE)[, c("fn", "url", "targetFile", "destinationPath")]
#> fn url targetFile
#> <char> <char> <char>
#> 1: prepInputs file:///tmp/RtmpJaUC3V/sources/table.csv table.csv
#> destinationPath
#> <char>
#> 1: /tmp/RtmpJaUC3V/inputsA url is recorded when it is used, whether or not a download was
needed: the second time the file is already in
destinationPath, but the project still uses it.
A file read with no url is recorded as
local: and its path. These are inputs someone put on the
disk by hand; they have no source to download from, so they must be
copied.
writeLines("id\n1", file.path(inputs, "byHand.csv"))
clearUrlLog()
byHand <- prepInputs(targetFile = "byHand.csv", destinationPath = inputs, fun = NA)
led <- data.table::rbindlist(prepInputsLog(), fill = TRUE)
led[, c("fn", "url")]
#> fn url
#> <char> <char>
#> 1: prepInputs (local) local:/tmp/RtmpJaUC3V/inputs/byHand.csv
led[!startsWith(url, "local:")] # only the remote sources
#> Empty data.table (0 rows and 8 cols): time,fn,url,targetFile,archive,alsoExtract...Cache()On a cache hit the cached function does not run, so nothing is
downloaded. The ledger would miss the data if it relied on the download.
Instead, Cache() stores the urls read inside it as tags on
the cache entry, and replays them on every hit. The ledger of a run is
therefore the same whether its steps were computed or came from the
cache.
options(reproducible.cachePath = file.path(tempdir(), "cache"))
getTables <- function(u1, u2, dp) {
list(prepInputs(url = u1, targetFile = "table.csv", destinationPath = dp, fun = NA),
prepInputs(url = u2, targetFile = "other.csv", destinationPath = dp, fun = NA))
}
clearUrlLog()
out <- Cache(getTables(url1, url2, inputs)) # computed
clearUrlLog()
out <- Cache(getTables(url1, url2, inputs)) # a cache hit: nothing runs, the urls are replayed
data.table::rbindlist(prepInputsLog(), fill = TRUE)[, c("fn", "url", "targetFile", "cacheId")]
#> fn url targetFile
#> <char> <char> <char>
#> 1: prepInputs file:///tmp/RtmpJaUC3V/sources/table.csv table.csv
#> 2: prepInputs file:///tmp/RtmpJaUC3V/sources/other.csv other.csv
#> cacheId
#> <char>
#> 1: 889b4577c214bf5f
#> 2: 889b4577c214bf5fThis holds for nested calls too. If an outer Cache() is
computed and an inner Cache() inside it is a hit, the inner
urls are passed to the outer entry, so a later hit of the outer entry
replays them.
The tags stay in the cache after the session ends.
urlLog() reads them for a whole cache folder: the data
every cached step of a project has used.
urlLog(getOption("reproducible.cachePath"),
which = c("function", "url", "targetFile", "destinationPath"))
#> function url targetFile
#> <char> <char> <char>
#> 1: getTables file:///tmp/RtmpJaUC3V/sources/table.csv table.csv
#> 2: getTables file:///tmp/RtmpJaUC3V/sources/other.csv other.csv
#> destinationPath
#> <char>
#> 1: /tmp/RtmpJaUC3V/inputs
#> 2: /tmp/RtmpJaUC3V/inputsSpaDES.core keeps one ledger per run: urlLog(sim) gives
the records of that simList, with the module and event that
read each url. The records also reach prepInputsLog().
Event caching, .inputObjects caching and
cacheChaining all replay the tags, as above.
The ledger only sees prepInputs() and
preProcess() (and CacheGeo()). Code that
downloads with download.file(),
googledrive::drive_download() or a web-service package
should say so with logUrlAccess(), once per access, just
before the download. Inside Cache() it is tagged and
replayed like any other record.
getStation <- function(id) {
u <- paste0("https://example.com/stations/", id, ".csv")
logUrlAccess(u, targetFile = paste0(id, ".csv"))
# download.file(u, ...) would go here
data.frame(id = id)
}
clearUrlLog()
st <- getStation("A1")
data.table::rbindlist(prepInputsLog(), fill = TRUE)[, c("fn", "url")]
#> fn url
#> <char> <char>
#> 1: getStation https://example.com/stations/A1.csv?prepInputsLog lists everything that is recorded and the
few cases that are not (for example a cache entry made with
options(reproducible.urlLog = FALSE)).
To run a project on a machine without internet, copy the files it reads. Two tools give that list, and they answer different questions.
The ledger names every input
prepInputs() read: each url with the
destinationPath and targetFile it was saved
as, and each local file (local:<path>). Copy those
files (and archive, if one was given) into the same
destinationPath on the other machine, and
prepInputs() finds them there without downloading. This
list is complete even when the run came from the cache, because hits
replay their urls.
## from the cache; for one session, data.table::rbindlist(prepInputsLog(), fill = TRUE)
led <- urlLog(getOption("reproducible.cachePath"),
which = c("url", "targetFile", "destinationPath"))
led <- led[!is.na(targetFile) & !is.na(destinationPath)]
unique(file.path(led$destinationPath, led$targetFile))
#> [1] "/tmp/RtmpJaUC3V/inputs/table.csv" "/tmp/RtmpJaUC3V/inputs/other.csv"filesUsedTrace() (Linux, needs
strace) runs a script and returns every existing file under
the given folders that the run opened: inputs, cache entries, module
code. It shows exactly what this run read, but no more. On a
cache hit the raw inputs behind the entry are never opened, so they are
not in the list. If the cache key then changes on the other machine, the
step is recomputed and needs those raw inputs.
For a machine without internet, copy both: the traced files (cache entries and anything read locally), and the files the ledger names (the raw inputs, so a cache miss there can still be recomputed). Or copy the project’s whole input and cache folders, if they are not too large.