Software Tools to Collect and Use Provenance in R

doi:10.6073/pasta/c622edd9114927f407cd55adb323fee7 Software Tools to Collect and Use Provenance in R Emery Boose https://orcid.org/0000-0003-4820-0231 Aaron Ellison https://orcid.org/0000-0003-4151-6081 Elizabeth Fong Matthew Lau https://orcid.org/0000-0003-3758-2406 Barbara Lerner Thomas Pasquier https://orcid.org/0000-0001-6876-1306 Margo Seltzer https://orcid.org/0000-0002-2165-4658 2024 English

The software tools that scientists use to process and analyze data are typically optimized for performance and ease of use. Few if any such tools are designed to capture and record the details of what happens as the tool performs its task. This detailed information, and more generally the history of an item of data from its creation to its present state, is known as provenance. Provenance has the potential to make science more transparent, reliable, and reproducible. This project focused on collecting and using provenance for scripts written in the R statistical language, which is widely used by ecologists and environmental scientists for data analysis and visualization. Our tools include a provenance collector (rdtLite), which collects provenance as an R script executes (or during a console session), as well as other tools that use the collected provenance to document and visualize the execution or to support activites such as script debugging. The R packages included here are also available on CRAN. For more details, see the project website on GitHub (https://end-to-end-provenance.github.io).

analytical tools modeling LTER controlled vocabulary disturbance LTER core area Harvard Forest HFR LTER USA HFR default

This dataset is released to the public under Creative Commons CC0 1.0 (No Rights Reserved). Please keep the dataset creators informed of any plans to use the dataset. Consultation with the original investigators is strongly encouraged. Publications and data products that make use of the dataset should include proper acknowledgement.

Creative Commons Zero v1.0 Universal https://spdx.org/licenses/CC0-1.0.html CC0-1.0 https://harvardforest.fas.harvard.edu/exist/apps/datasets/showData.html?id=hf091 Global. Coordinates based on WGS84 datum. -180 +180 +90 -90 complete Information Manager Harvard Forest

324 North Main Street Petersham MA 01366 USA

(978) 724-3302 hf-im@lists.fas.harvard.edu Harvard Forest

324 North Main Street Petersham MA 01366 USA

(978) 724-3302 (978) 724-3595 https://harvardforest.fas.harvard.edu

1. rdtLite collects provenance as an R script executes (or during a console session) and saves it in extended PROV-JSON format to the file prov.json. By default, this file is written to the R session temporary directory (to meet CRAN requirements) and is overwritten in subsequent executions of the same script (or console session), but you can choose to save it elsewhere and to save time-stamped versions if desired. Simple data values are automatically saved in the prov.json file. Complex data values (e.g. R lists or data frames) may optionally be saved (wholly or in part) as separate snapshot files. 2. provDebugR uses the provenance collected by rdtLite to support time-traveling debugging of an R script without the need to set breakpoints or insert print statements and rerun the script. 3. provExplainR uses the provenance collected by rdtLite from two different executions of a script to help explain why the script results differ. 4. provGraphR creates an adjacency matrix from the provenance object created by provParseR. The adjacency matrix can then be used to quickly traverse the provenance graph. This package supports other packages and is not intended to be used directly. 5. provParseR facilitates access to the provenance information collected by rdtLite. The prov.parse function accepts this information as a string or file in extended PROV-JSON format and returns it as an R object. Access functions then extract the desired information from this object and returns it as a data frame. This package supports other packages and is not intended to be used directly. 6. provSummarizeR creates a concise high-level summary of the provenance collected by rdtLite, including information about computing environment, loaded libraries, sourced scripts, and inputs and outputs. 7. provTraceR uses the provenance collected by rdtLite for a single R script or a series of R scripts to identify input files, output files, and exchanged files based on file hash values. 8. provViz provides an R interface to a visualization tool, written in Java, that allows you to view and query the provenance graph directly. You will need to have Java installed for this to work. Note: the R packages included here have been renamed for archival purposes. Please rename the package file after downloading to remove the archival prefix before installing as an R package (e.g. rename "hf091-01-rdtLite_1.4.tar.gz" to "rdtLite_1.4.tar.gz").

Harvard Forest Long-Term Ecological Research Harvard Forest

324 North Main Street Petersham MA 01366 USA

(978) 724-3302 (978) 724-3595 https://harvardforest.fas.harvard.edu https://ror.org/059cpzx98 pointOfContact The Harvard Forest Long-Term Ecological Research (LTER) program examines ecological dynamics in the New England region resulting from natural disturbances, environmental change, and human impacts. National Science Foundation LTER grants: DEB-8811764, DEB-9411975, DEB-0080592, DEB-0620443, DEB-1237491, DEB-1832210. hf091-01-rdtLite_1.4.tar.gz rdtLite v. 1.4 hf091-01-rdtLite_1.4.tar.gz 368269 7f1229f80b8c93159edb337d758c9909 tar.gz R package https://harvardforest.fas.harvard.edu/data/p09/hf091/hf091-01-rdtLite_1.4.tar.gz script hf091-02-provDebugR_1.0.1.tar.gz provDebugR v. 1.0.1 hf091-02-provDebugR_1.0.1.tar.gz 91110 cef27d4b0dc60f61f464b291b98d2526 tar.gz R package https://harvardforest.fas.harvard.edu/data/p09/hf091/hf091-02-provDebugR_1.0.1.tar.gz script hf091-03-provExplainR_1.1.1.tar.gz provExplainR v. 1.1.1 hf091-03-provExplainR_1.1.1.tar.gz 44552 112a656b2fd05f36f97cb5efb2f233ee tar.gz R package https://harvardforest.fas.harvard.edu/data/p09/hf091/hf091-03-provExplainR_1.1.1.tar.gz script hf091-04-provGraphR_1.0.1.tar.gz provGraphR v. 1.0.1 hf091-04-provGraphR_1.0.1.tar.gz 24132 30edec6eb55d88f3ed56063fab7e9e49 tar.gz R package https://harvardforest.fas.harvard.edu/data/p09/hf091/hf091-04-provGraphR_1.0.1.tar.gz script hf091-05-provParseR_1.0.tar.gz provParseR v. 1.0 hf091-05-provParseR_1.0.tar.gz 57502 5055ea270449431a9b1b50763c499184 tar.gz R package https://harvardforest.fas.harvard.edu/data/p09/hf091/hf091-05-provParseR_1.0.tar.gz script hf091-06-provSummarizeR_1.5.1.tar.gz provSummarize v. 1.5.1 hf091-06-provSummarizeR_1.5.1.tar.gz 26021 9847833b38cd69d67942cdf6d3c592d9 tar.gz R package https://harvardforest.fas.harvard.edu/data/p09/hf091/hf091-06-provSummarizeR_1.5.1.tar.gz script hf091-07-provTraceR_1.0.tar.gz provTraceR v. 1.0 hf091-07-provTraceR_1.0.tar.gz 20232 41170c2260392d8364e67d143b5927c7 tar.gz R package https://harvardforest.fas.harvard.edu/data/p09/hf091/hf091-07-provTraceR_1.0.tar.gz script hf091-08-provViz_1.0.9.tar.gz provViz v. 1.0.9 hf091-08-provViz_1.0.9.tar.gz 3163921 c48cff5b6eff1f03b71bbdc4f0c6800f tar.gz R package https://harvardforest.fas.harvard.edu/data/p09/hf091/hf091-08-provViz_1.0.9.tar.gz script