class: center, middle, inverse, title-slide # Automated Assessment
and Standards Alignment
of R Packages ### Mark Padgham & Noam Ross
rOpenSci & EcoHealth Alliance
Münster, Germany & New York, USA --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # rOpenSci: Transforming science through open data, software & reproducibility "*We help develop R packages for the sciences via community driven learning, review and maintenance of contributed software in the R ecosystem*" --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # rOpenSci & Software Peer Review "packages contributed by the community undergo a transparent, constructive, non adversarial and open review process." --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # rOpenSci & Software Peer Review: Scope Current scope includes software for: .left-column[ - data retrieval - data extraction - data munging - data deposition - data validation and testing - workflow automation - version control - citation management and bibliometrics ] .right-column[ - scientific software wrappers - field and laboratory reproducibility tools - database software bindings - geospatial data - text analysis ] --- class: center, top background-image: url(data:image/png;base64,#img/SloanLogo-1B-SMALL-Gold-Blue.png) background-size: contain background-position: 50% 50% -- # Expanding Software Peer Review -- .fontBIG[ Expand Scope<br>to include<br>Statistical Software ] --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # Statistical Software Peer Review ## What is statistical software? .left-column[ - Bayesian and Monte Carlo - Regression and Supervised Learning - Dimensionality Reduction, Clustering, and Unsupervised Learning - Exploratory Data Analysis (EDA) and Summary Statistics ] .right-column[ - Time Series Analyses - Machine Learning - Spatial Analyses - ~~Wrapper Packages~~ - ~~Network Analysis~~ - ~~Probability Distributions~~ - ~~Workflow Support~~ ] --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # Statistical Software Peer Review ## Our approach - General and category-specific standards - Standards compliance documented within code - Automated assessment tools --- class: left, top background-image: url(data:image/png;base64,#img/package-flow.png) background-size: contain background-position: 50% 50% --- class: center, middle, inverse # autotest ## github.com/ropensci-review-tools ## search: "ropensci autotest" --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package: Why? ## rOpenSci Software Peer Review - Reviewers often the first "external" users of submitted packages - Quickly uncover bugs, distracting reviews from higher-level focus on software quality - A tool to help uncover bugs prior to submission should increase software quality --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package: How? - A form of "mutation testing" - Extracts documented example code to identify all inputs to, and outputs from, all functions - Mutates those inputs - Also matches input and output types to descriptions --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package 1/3: Set up a package to test: ```r path <- file.path (tempdir (), "demo") usethis::create_package (path) fs::dir_tree (path) ``` ``` ## /tmp/Rtmp0PM0ND/demo ## ├── DESCRIPTION ## ├── NAMESPACE ## └── R ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package 2/3: Add a function: ```r #' my_function #' #' @param x An input #' @return Something else #' @export my_function <- function (x) { return (x + 1) } ``` - save to `./R/myfn.R` - run `roxygen2::roxygenise()` to create documentation (`/man`) file --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package 3/3: `autotest` it: ```r library (autotest) a <- autotest_package (path, test = TRUE) ``` ``` ## ## ── autotesting demo ── ## ``` - Returns a table with 1 row and 9 columns --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package 3/3: `autotest` it: ```r library (autotest) a <- autotest_package (path, test = TRUE) print (a) ``` ``` ## type test_name fn_name parameter parameter_type ## 1 warning fn_without_example my_function <NA> <NA> ## operation ## 1 Identify functions without documented examples ## content test yaml_hash ## 1 This function has no documented example TRUE <NA> ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package 3/3: `autotest` it: ```r library (autotest) a <- autotest_package (path, test = TRUE) a$content ``` ``` ## [1] "This function has no documented example" ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ```r #' my_function #' #' @param x An input #' @return Something else #' @export my_function <- function (x) { return (x + 1) } ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ```r #' my_function #' #' @param x An input #' @return Something else #' @export #' @examples #' y <- my_function (x = 1) my_function <- function (x) { return (x + 1) } ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ```r library (autotest) a <- autotest_package (path, test = TRUE) ``` ``` ## ## ── autotesting demo ## ✔ [1 / 1]: myfunction ``` ```r a$content ``` ``` ## [1] Parameter [x] is not specified as integer, yet ## only used as such; please use '1L' for integer, ## or 1.0 for non-integer values. ## [2] Parameter [x] of function [my_function] is only used ## as a single numeric value, but responds to vectors of ## length > 1 ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ```r #' my_function #' #' @param x An input #' @return Something else #' @export #' @examples #' y <- my_function (x = 1) my_function <- function (x) { return (x + 1) } ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ```r #' my_function #' #' @param x An input #' @return Something else #' @export #' @examples #' y <- my_function (x = 1L) my_function <- function (x) { checkmate::assertInt (x) return (x + 1) } ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ```r library (autotest) a <- autotest_package (path, test = TRUE) ``` ``` ## ## ── autotesting demo ## ✔ [1 / 1]: myfunction ``` ```r a$content ``` ``` ## [1] Parameter [x] permits unrestricted integer inputs, ## yet does not document this; please add 'unrestricted' ## to parameter description. ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ```r #' my_function #' #' @param x An input #' @return Something else #' @export #' @examples #' y <- my_function (x = 1L) my_function <- function (x) { checkmate::assertInt (x) return (x + 1) } ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ```r #' my_function #' #' @param x An unrestricted integer input #' @return Something else #' @export #' @examples #' y <- my_function (x = 1L) my_function <- function (x) { checkmate::assertInt (x) return (x + 1) } ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ```r library (autotest) a <- autotest_package (path, test = TRUE) ``` ``` ## ## ── autotesting demo ## ✔ [1 / 1]: myfunction ``` ```r a ## NULL ``` -- ### When `autotest` works, it should do nothing! --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ## Which tests are run? `test = FALSE` ```r a <- autotest_package (path, test = FALSE) a ``` ``` ## # A tibble: 9 x 9 ## type test_name fn_name parameter parameter_type operation content test ## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <lgl> ## 1 dummy int_range my_func… x single integer Ascertain … Should e… TRUE ## 2 dummy int_as_nu… my_func… x single integer Integer va… (Should … TRUE ## 3 dummy single_pa… my_func… x single integer Length 2 v… Should t… TRUE ## 4 dummy return_su… my_func… (return … (return objec… Check that… <NA> TRUE ## 5 dummy return_va… my_func… (return … (return objec… Check that… <NA> TRUE ## 6 dummy return_de… my_func… (return … (return objec… Check whet… <NA> TRUE ## 7 dummy return_cl… my_func… (return … (return objec… Compare cl… <NA> TRUE ## 8 dummy par_is_do… my_func… x <NA> Check that… <NA> TRUE ## 9 dummy par_match… my_func… x <NA> Check that… <NA> TRUE ## # … with 1 more variable: yaml_hash <chr> ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ## Which tests are run? `test = FALSE` ```r a <- autotest_package (path, test = FALSE) a$test_name ``` ``` ## [1] "int_range" "int_as_numeric" ## [3] "single_par_as_length_2" "return_successful" ## [5] "return_val_described" "return_desc_includes_class" ## [7] "return_class_matches_desc" "par_is_documented" ## [9] "par_matches_docs" ``` --- class: center, middle, inverse # autotesting a real package --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package A real example: `autotest`ing the `stats` package ```r a <- autotest_package (package = "stats", functions = "var", test = FALSE) # default ``` - `package` argument can be name of an installed package, or path to local source directory - `functions` argument can restrict tests to specified functions only --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package A real example: `autotest`ing the `stats` package ```r a <- autotest_package (package = "stats", functions = "var", test = FALSE) # default ``` ``` ## ## ── autotesting stats ## ✔ [1 / 4]: var ## ✔ [2 / 4]: cor ## ✔ [3 / 4]: cov ## ✔ [4 / 4]: cov ``` ```r dim (a) ``` ``` ## [1] 150 9 ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package A real example: `autotest`ing the `stats` package ```r a <- autotest_package (package = "stats", functions = "var", test = TRUE) # takes 20-30 seconds ``` ``` ## ## ── autotesting stats ## ✔ [1 / 4]: var ## ✔ [2 / 4]: cor ## ✔ [3 / 4]: cov ## ✔ [4 / 4]: cov ``` ```r dim (a) ``` ``` ## [1] 15 9 ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package A real example: `autotest`ing the `stats` package ``` ## # A tibble: 15 x 9 ## type test_name fn_name parameter parameter_type operation content test ## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <lgl> ## 1 warni… par_is_de… var use <NA> Check tha… "Example… TRUE ## 2 warni… par_is_de… cov y <NA> Check tha… "Example… TRUE ## 3 diagn… single_ch… cor use single charac… upper-cas… "is case… TRUE ## 4 diagn… single_ch… cor method single charac… upper-cas… "is case… TRUE ## 5 diagn… single_ch… cor use single charac… upper-cas… "is case… TRUE ## 6 diagn… single_ch… cor method single charac… upper-cas… "is case… TRUE ## 7 diagn… single_ch… cov use single charac… upper-cas… "is case… TRUE ## 8 diagn… single_ch… cov method single charac… upper-cas… "is case… TRUE ## 9 diagn… single_ch… cov use single charac… upper-cas… "is case… TRUE ## 10 diagn… single_ch… cov method single charac… upper-cas… "is case… TRUE ## 11 diagn… single_ch… cov use single charac… upper-cas… "is case… TRUE ## 12 diagn… single_ch… cov method single charac… upper-cas… "is case… TRUE ## 13 diagn… vector_to… var x vector Convert v… "Functio… TRUE ## 14 diagn… vector_to… var x vector Convert v… "Functio… TRUE ## 15 diagn… vector_to… var y vector Convert v… "Functio… TRUE ## # … with 1 more variable: yaml_hash <chr> ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package A real example: `autotest`ing the `stats` package - Use `DT` package to inspect results! ```r a <- autotest_package (package = "stats", functions = "var", test = TRUE) # takes 20-30 seconds DT::datatable (a) ``` --- class: left, top background-image: url(data:image/png;base64,#img/stats-var-TRUE.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package A real example: `autotest`ing the `stats` package ```r a <- autotest_package (package = "stats", functions = "var", test = TRUE) ``` - `stats` package is algorithmially robust - But `autotest` reveals a few gaps in documentation! --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # Statistical Software Peer Review ## Our approach - General and category-specific standards - Standards compliance documented within code - ~~Automated assessment tools~~ --- class: center, middle, inverse # Statistical Standards ## Online Book Contains all General and Category-Specific Standards `stats-devguide.ropensci.org` --- class: left, top background-image: url(data:image/png;base64,#img/devguide-stds-G1.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/devguide-stds-G2.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/devguide-stds-G3.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/devguide-stds-G4.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/devguide-stds-BS1.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/devguide-stds-BS2.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/devguide-stds-BS3.png) background-size: contain background-position: 50% 50% --- class: center, middle, inverse # Statistical Standards ## 40 General Standards &<br>265 Category-Specific Standards<br>in 7 Categories ## Four New Categories in Progress --- class: left, top background-image: url(data:image/png;base64,#img/package-flow.png) background-size: contain background-position: 50% 50% --- class: center, middle, inverse # The 'srr' package ## `srr` = "Software Review Roclets" (A "roclet" is a documentation engine,<br>or "doclet", for the **R** language.) ## github.com/ropensci-review-tools ## search: "ropensci srr" --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package: How? ```r #' Function to add two numbers #' #' @rdtitle Addtwo #' @param x First input #' @param y Second input #' @return Sum of x and y #' @export addtwo <- function (x, y) { return (x + y) } ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package: How? ```r #' Function to add two numbers #' #' @rdtitle Addtwo #' @param x First input #' @param y Second input #' @return Sum of x and y #' @export #' #' @srrstats {G1.0} Primary citation for this function follows #' @references Smith & Jones (2021) A new addition algorithm addtwo <- function (x, y) { return (x + y) } ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package: How? - `srr` introduces new tags for documenting standards: - `@srrstats` for adherence to standards - `@srrstatsNA` for non-applicable standards - `@srrstatsTODO` for standards yet to be addressed - Standard command to update package documentation automatically generates a summary report of standards compliance on screen --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package: How? ```r roxygen2::roxygenise(path) ``` ``` ## ℹ Loading demo ## ## ── rOpenSci Statistical Software Standards ─────────────────────────────────────────────────────────────────────────── ## ## ── @srrstats standards: ## • [G1.0] in function 'addtwo()' on line#11 of file [R/test.R] ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package: How? - One function dumps full text of standards in file `R/srr-stats-standards.R` - All are initially tagged `@srrstatsTODO` - Standards are addressed by changing tag to `@srrstats` and moving to location within code where standard is addressed --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package: How? In package directory: ```r srr_stats_roxygen (category = "ml") ``` ``` ## ✔ Downloaded general standards ## ✔ Downloaded ml standards ## ℹ Roxygen2-formatted standards written to [srr-stats-standards.R] ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package: How? ``` ## [1] "#' srr_stats" ## [2] "#'" ## [3] "#' All of the following standards initially have `@srrstatsTODO` tags." ## [4] "#' These may be moved at any time to any other locations in your code." ## [5] "#' Once addressed, please modify the tag from `@srrstatsTODO` to `@srrstats`," ## [6] "#' or `@srrstatsNA`, ensuring that references to every one of the following" ## [7] "#' standards remain somewhere within your code." ## [8] "#' (These comments may be deleted at any time.)" ## [9] "#'" ## [10] "#' @srrstatsVerbose TRUE" ## [11] "#'" ## [12] "#' @srrstatsTODO {G1.0} *Statistical Software should list at least one primary reference from published academic literature.* " ## [13] "#' @srrstatsTODO {G1.1} *Statistical Software should document whether the algorithm(s) it implements are:* - *The first implementation of a novel algorithm*; or - *The first implementation within **R** of an algorithm which has previously been implemented in other languages or contexts*; or - *An improvement on other implementations of similar algorithms in **R***. " ## [14] "#' @srrstatsTODO {G1.2} *Statistical Software should include a* Life Cycle Statement *describing current and anticipated future states of development.* " ## [15] "#' @srrstatsTODO {G1.3} *All statistical terminology should be clarified and unambiguously defined.* " ## [16] "#' @srrstatsTODO {G1.4} *Software should use [`roxygen2`](https://roxygen2.r-lib.org/) to document all functions.*" ## [17] "#' @srrstatsTODO {G1.4a} *All internal (non-exported) functions should also be documented in standard [`roxygen2`](https://roxygen2.r-lib.org/) format, along with a final `@noRd` tag to suppress automatic generation of `.Rd` files.* " ## [18] "#' @srrstatsTODO {G1.5} *Software should include all code necessary to reproduce results which form the basis of performance claims made in associated publications.* " ## [19] "#' @srrstatsTODO {G1.6} *Software should include code necessary to compare performance claims with alternative implementations in other R packages.* " ## [20] "#' @srrstatsTODO {G2.0} *Implement assertions on lengths of inputs, particularly through asserting that inputs expected to be single- or multi-valued are indeed so.*" ## [21] "#' @srrstatsTODO {G2.0a} Provide explicit secondary documentation of any expectations on lengths of inputs" ## [22] "#' @srrstatsTODO {G2.1} *Implement assertions on types of inputs (see the initial point on nomenclature above).*" ## [23] "#' @srrstatsTODO {G2.1a} *Provide explicit secondary documentation of expectations on data types of all vector inputs.*" ## [24] "#' @srrstatsTODO {G2.2} *Appropriately prohibit or restrict submission of multivariate input to parameters expected to be univariate.*" ## [25] "#' @srrstatsTODO {G2.3} *For univariate character input:*" ## [26] "#' @srrstatsTODO {G2.3a} *Use `match.arg()` or equivalent where applicable to only permit expected values.*" ## [27] "#' @srrstatsTODO {G2.3b} *Either: use `tolower()` or equivalent to ensure input of character parameters is not case dependent; or explicitly document that parameters are strictly case-sensitive.*" ## [28] "#' @srrstatsTODO {G2.4} *Provide appropriate mechanisms to convert between different data types, potentially including:*" ## [29] "#' @srrstatsTODO {G2.4a} *explicit conversion to `integer` via `as.integer()`*" ## [30] "#' @srrstatsTODO {G2.4b} *explicit conversion to continuous via `as.numeric()`*" ## [31] "#' @srrstatsTODO {G2.4c} *explicit conversion to character via `as.character()` (and not `paste` or `paste0`)*" ## [32] "#' @srrstatsTODO {G2.4d} *explicit conversion to factor via `as.factor()`*" ## [33] "#' @srrstatsTODO {G2.4e} *explicit conversion from factor via `as...()` functions*" ## [34] "#' @srrstatsTODO {G2.5} *Where inputs are expected to be of `factor` type, secondary documentation should explicitly state whether these should be `ordered` or not, and those inputs should provide appropriate error or other routines to ensure inputs follow these expectations.* " ## [35] "#' @srrstatsTODO {G2.6} *Software which accepts one-dimensional input should ensure values are appropriately pre-processed regardless of class structures.* " ## [36] "#' @srrstatsTODO {G2.7} *Software should accept as input as many of the above standard tabular forms as possible, including extension to domain-specific forms.* " ## [37] "#' @srrstatsTODO {G2.8} *Software should provide appropriate conversion or dispatch routines as part of initial pre-processing to ensure that all other sub-functions of a package receive inputs of a single defined class or type.*" ## [38] "#' @srrstatsTODO {G2.9} *Software should issue diagnostic messages for type conversion in which information is lost (such as conversion of variables from factor to character; standardisation of variable names; or removal of meta-data such as those associated with [`sf`-format](https://r-spatial.github.io/sf/) data) or added (such as insertion of variable or column names where none were provided).* " ## [39] "#' @srrstatsTODO {G2.10} *Software should ensure that extraction or filtering of single columns from tabular inputs should not presume any particular default behaviour, and should ensure all column-extraction operations behave consistently regardless of the class of tabular data used as input.* " ## [40] "#' @srrstatsTODO {G2.11} *Software should ensure that `data.frame`-like tabular objects which have columns which do not themselves have standard class attributes (typically, `vector`) are appropriately processed, and do not error without reason. This behaviour should be tested. Again, columns created by the [`units` package](https://github.com/r-quantities/units/) provide a good test case.*" ## [41] "#' @srrstatsTODO {G2.12} *Software should ensure that `data.frame`-like tabular objects which have list columns should ensure that those columns are appropriately pre-processed either through being removed, converted to equivalent vector columns where appropriate, or some other appropriate treatment such as an informative error. This behaviour should be tested.* " ## [42] "#' @srrstatsTODO {G2.13} *Statistical Software should implement appropriate checks for missing data as part of initial pre-processing prior to passing data to analytic algorithms.*" ## [43] "#' @srrstatsTODO {G2.14} *Where possible, all functions should provide options for users to specify how to handle missing (`NA`) data, with options minimally including:*" ## [44] "#' @srrstatsTODO {G2.14a} *error on missing data*" ## [45] "#' @srrstatsTODO {G2.14b} *ignore missing data with default warnings or messages issued*" ## [46] "#' @srrstatsTODO {G2.14c} *replace missing data with appropriately imputed values*" ## [47] "#' @srrstatsTODO {G2.15} *Functions should never assume non-missingness, and should never pass data with potential missing values to any base routines with default `na.rm = FALSE`-type parameters (such as [`mean()`](https://stat.ethz.ch/R-manual/R-devel/library/base/html/mean.html), [`sd()`](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/sd.html) or [`cor()`](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/cor.html)).*" ## [48] "#' @srrstatsTODO {G2.16} *All functions should also provide options to handle undefined values (e.g., `NaN`, `Inf` and `-Inf`), including potentially ignoring or removing such values.* " ## [49] "#' @srrstatsTODO {G3.0} *Statistical software should never compare floating point numbers for equality. All numeric equality comparisons should either ensure that they are made between integers, or use appropriate tolerances for approximate equality.* " ## [50] "#' @srrstatsTODO {G3.1} *Statistical software which relies on covariance calculations should enable users to choose between different algorithms for calculating covariances, and should not rely solely on covariances from the `stats::cov` function.*" ## [51] "#' @srrstatsTODO {G3.1a} *The ability to use arbitrarily specified covariance methods should be documented (typically in examples or vignettes).* " ## [52] "#' @srrstatsTODO {G4.0} *Statistical Software which enables outputs to be written to local files should parse parameters specifying file names to ensure appropriate file suffices are automatically generated where not provided.* " ## [53] "#' @srrstatsTODO {G5.0} *Where applicable or practicable, tests should use standard data sets with known properties (for example, the [NIST Standard Reference Datasets](https://www.itl.nist.gov/div898/strd/), or data sets provided by other widely-used R packages).*" ## [54] "#' @srrstatsTODO {G5.1} *Data sets created within, and used to test, a package should be exported (or otherwise made generally available) so that users can confirm tests and run examples.* " ## [55] "#' @srrstatsTODO {G5.2} *Appropriate error and warning behaviour of all functions should be explicitly demonstrated through tests. In particular,*" ## [56] "#' @srrstatsTODO {G5.2a} *Every message produced within R code by `stop()`, `warning()`, `message()`, or equivalent should be unique*" ## [57] "#' @srrstatsTODO {G5.2b} *Explicit tests should demonstrate conditions which trigger every one of those messages, and should compare the result with expected values.*" ## [58] "#' @srrstatsTODO {G5.3} *For functions which are expected to return objects containing no missing (`NA`) or undefined (`NaN`, `Inf`) values, the absence of any such values in return objects should be explicitly tested.* " ## [59] "#' @srrstatsTODO {G5.4} **Correctness tests** *to test that statistical algorithms produce expected results to some fixed test data sets (potentially through comparisons using binding frameworks such as [RStata](https://github.com/lbraglia/RStata)).*" ## [60] "#' @srrstatsTODO {G5.4a} *For new methods, it can be difficult to separate out correctness of the method from the correctness of the implementation, as there may not be reference for comparison. In this case, testing may be implemented against simple, trivial cases or against multiple implementations such as an initial R implementation compared with results from a C/C++ implementation.*" ## [61] "#' @srrstatsTODO {G5.4b} *For new implementations of existing methods, correctness tests should include tests against previous implementations. Such testing may explicitly call those implementations in testing, preferably from fixed-versions of other software, or use stored outputs from those where that is not possible.*" ## [62] "#' @srrstatsTODO {G5.4c} *Where applicable, stored values may be drawn from published paper outputs when applicable and where code from original implementations is not available*" ## [63] "#' @srrstatsTODO {G5.5} *Correctness tests should be run with a fixed random seed*" ## [64] "#' @srrstatsTODO {G5.6} **Parameter recovery tests** *to test that the implementation produce expected results given data with known properties. For instance, a linear regression algorithm should return expected coefficient values for a simulated data set generated from a linear model.*" ## [65] "#' @srrstatsTODO {G5.6a} *Parameter recovery tests should generally be expected to succeed within a defined tolerance rather than recovering exact values.*" ## [66] "#' @srrstatsTODO {G5.6b} *Parameter recovery tests should be run with multiple random seeds when either data simulation or the algorithm contains a random component. (When long-running, such tests may be part of an extended, rather than regular, test suite; see G4.10-4.12, below).*" ## [67] "#' @srrstatsTODO {G5.7} **Algorithm performance tests** *to test that implementation performs as expected as properties of data change. For instance, a test may show that parameters approach correct estimates within tolerance as data size increases, or that convergence times decrease for higher convergence thresholds.*" ## [68] "#' @srrstatsTODO {G5.8} **Edge condition tests** *to test that these conditions produce expected behaviour such as clear warnings or errors when confronted with data with extreme properties including but not limited to:*" ## [69] "#' @srrstatsTODO {G5.8a} *Zero-length data*" ## [70] "#' @srrstatsTODO {G5.8b} *Data of unsupported types (e.g., character or complex numbers in for functions designed only for numeric data)*" ## [71] "#' @srrstatsTODO {G5.8c} *Data with all-`NA` fields or columns or all identical fields or columns*" ## [72] "#' @srrstatsTODO {G5.8d} *Data outside the scope of the algorithm (for example, data with more fields (columns) than observations (rows) for some regression algorithms)*" ## [73] "#' @srrstatsTODO {G5.9} **Noise susceptibility tests** *Packages should test for expected stochastic behaviour, such as through the following conditions:*" ## [74] "#' @srrstatsTODO {G5.9a} *Adding trivial noise (for example, at the scale of `.Machine$double.eps`) to data does not meaningfully change results*" ## [75] "#' @srrstatsTODO {G5.9b} *Running under different random seeds or initial conditions does not meaningfully change results* " ## [76] "#' @srrstatsTODO {G5.10} *Extended tests should included and run under a common framework with other tests but be switched on by flags such as as a `<MYPKG>_EXTENDED_TESTS=1` environment variable.*" ## [77] "#' @srrstatsTODO {G5.11} *Where extended tests require large data sets or other assets, these should be provided for downloading and fetched as part of the testing workflow.*" ## [78] "#' @srrstatsTODO {G5.11a} *When any downloads of additional data necessary for extended tests fail, the tests themselves should not fail, rather be skipped and implicitly succeed with an appropriate diagnostic message.*" ## [79] "#' @srrstatsTODO {G5.12} *Any conditions necessary to run extended tests such as platform requirements, memory, expected runtime, and artefacts produced that may need manual inspection, should be described in developer documentation such as a `CONTRIBUTING.md` or `tests/README.md` file.*" ## [80] "#' @srrstatsTODO {ML1.0} *Documentation should make a clear conceptual distinction between training and test data (even where such may ultimately be confounded as described above.)*" ## [81] "#' @srrstatsTODO {ML1.0a} *Where these terms are ultimately eschewed, these should nevertheless be used in initial documentation, along with clear explanation of, and justification for, alternative terminology.*" ## [82] "#' @srrstatsTODO {ML1.1} *Absent clear justification for alternative design decisions, input data should be expected to be labelled \"test\", \"training\", and, where applicable, \"validation\" data.*" ## [83] "#' @srrstatsTODO {ML1.1a} *The presence and use of these labels should be explicitly confirmed via pre-processing steps (and tested in accordance with **ML7.0**, below).*" ## [84] "#' @srrstatsTODO {ML1.1b} *Matches to expected labels should be case-insensitive and based on partial matching such that, for example, \"Test\", \"test\", or \"testing\" should all suffice.* " ## [85] "#' @srrstatsTODO {ML1.2} *Training and test data sets for ML software should be able to be input as a single, generally tabular, data object, with the training and test data distinguished either by* - *A specified variable containing, for example, `TRUE`/`FALSE` or `0`/`1` values, or which uses some other system such as missing (`NA`) values to denote test data); and/or* - *An additional parameter designating case or row numbers, or labels of test data.* " ## [86] "#' @srrstatsTODO {ML1.3} *Input data should be clearly partitioned between training and test data (for example, through having each passed as a distinct `list` item), or should enable an additional means of categorically distinguishing training from test data (such as via an additional parameter which provides explicit labels). Where applicable, distinction of validation and any other data should also accord with this standard.* " ## [87] "#' @srrstatsTODO {ML1.4} *Training and test data sets, along with other necessary components such as validation data sets, should be stored in their own distinctly labelled sub-directories (for distinct files), or according to an explicit and distinct labelling scheme (for example, for database connections). Labelling should in all cases adhere to **ML1.1**, above.* " ## [88] "#' @srrstatsTODO {ML1.5} *ML software should implement a single function which summarises the contents of test and training (and other) data sets, minimally including counts of numbers of cases, records, or files, and potentially extending to tables or summaries of file or data types, sizes, and other information (such as unique hashes for each component).* " ## [89] "#' @srrstatsTODO {ML1.6} *ML software which does not admit missing values, and which expects no missing values, should implement explicit pre-processing routines to identify whether data has any missing values, and should generally error appropriately and informatively when passed data with missing values. In addition, ML software which does not admit missing values should:*" ## [90] "#' @srrstatsTODO {ML1.6a} *Explain why missing values are not admitted.*" ## [91] "#' @srrstatsTODO {ML1.6b} *Provide explicit examples (in function documentation, vignettes, or both) for how missing values may be imputed, rather than simply discarded.*" ## [92] "#' @srrstatsTODO {ML1.7} *ML software which admits missing values should clearly document how such values are processed.*" ## [93] "#' @srrstatsTODO {ML1.7a} *Where missing values are imputed, software should offer multiple user-defined ways to impute missing data.*" ## [94] "#' @srrstatsTODO {ML1.7b} *Where missing values are imputed, the precise imputation steps should also be explicitly documented, either in tests (see **ML7.2** below), function documentation, or vignettes.*" ## [95] "#' @srrstatsTODO {ML1.8} *ML software should enable equal treatment of missing values for both training and test data, with optional user ability to control application to either one or both.* " ## [96] "#' @srrstatsTODO {ML2.0} *A dedicated function should enable pre-processing steps to be defined and parametrized.*" ## [97] "#' @srrstatsTODO {ML2.0a} *That function should return an object which can be directly submitted to a specified model (see section 3, below).*" ## [98] "#' @srrstatsTODO {ML2.0b} *Absent explicit justification otherwise, that return object should have a defined class minimally intended to implement a default `print` method which summarizes the input data set (as per **ML1.5** above) and associated transformations (see the following standard).* " ## [99] "#' @srrstatsTODO {ML2.1} *ML software which uses broadcasting to reconcile dimensionally incommensurate input data should offer an ability to at least optionally record transformations applied to each input file.* " ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package: How? ``` ## [1] "#' srr_stats" ## [2] "#'" ## [3] "#' All of the following standards initially have `@srrstatsTODO` tags." ## [4] "#' These may be moved at any time to any other locations in your code." ## [5] "#' Once addressed, please modify the tag from `@srrstatsTODO` to `@srrstats`," ## [6] "#' or `@srrstatsNA`, ensuring that references to every one of the following" ## [7] "#' standards remain somewhere within your code." ## [8] "#' (These comments may be deleted at any time.)" ## [9] "#'" ## [10] "#' @srrstatsVerbose TRUE" ## [11] "#'" ## [12] "#' @srrstatsTODO {G1.0} *Statistical Software should list at least one primary reference from published academic literature.* " ## [13] "#' @srrstatsTODO {G1.1} *Statistical Software should document whether the algorithm(s) it implements are:* - *The first implementation of a novel algorithm*; or - *The first implementation within **R** of an algorithm which has previously been implemented in other languages or contexts*; or - *An improvement on other implementations of similar algorithms in **R***. " ## [14] "#' @srrstatsTODO {G1.2} *Statistical Software should include a* Life Cycle Statement *describing current and anticipated future states of development.* " ``` File has: - 180 lines - 159 standards --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package: How? - Insert standards into package with one line of code - Comply with standards - Update documentation to automatically generate compliance report - Generate hyperlinked `html` version with `srr_report()` function --- class: left, top background-image: url(data:image/png;base64,#img/srr_report.png) background-size: cover background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package: How? - Insert standards into package with one line of code - Comply with standards - Update documentation to automatically generate compliance report - Generate hyperlinked `html` version with `srr_report()` function - Also extensible, with alternative standards<br>substituted by modifying a single URL --- class: left, top background-image: url(data:image/png;base64,#img/package-flow.png) background-size: contain background-position: 50% 50% --- class: left, middle, inverse # Statistical Software Peer Review - Now accepting submissions - Search "rOpenSci statistical software" - All tools available for use now at `github.com/ropensci-review-tools`