class: center, middle, inverse, title-slide # Assessment and Standardization of R Packages for Statistical Analysis by ROpenSci ### Mark Padgham & Noam Ross<br>rOpenSci & EcoHealth Alliance<br>Münster, Germany & New York, USA --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # rOpenSci: Transforming science through open data, software & reproducibility "*We help develop R packages for the sciences via community driven learning, review and maintenance of contributed software in the R ecosystem*" --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # rOpenSci & Software Peer Review "packages contributed by the community undergo a transparent, constructive, non adversarial and open review process." --- class: center, middle, inverse background-image: url(data:image/png;base64,#img/devguide-cover.png) background-size: contain background-position: 50% 50% -- # `devguide.ropensci.org` --- class: center, middle, inverse background-image: url(data:image/png;base64,#img/devguide-ch1.png) background-size: contain background-position: 50% 50% --- class: center, middle, inverse background-image: url(data:image/png;base64,#img/devguide-ch2.png) background-size: contain background-position: 50% 50% --- class: center, middle, inverse background-image: url(data:image/png;base64,#img/devguide-ch3.png) background-size: contain background-position: 50% 50% --- class: center, top, inverse background-image: url(data:image/png;base64,#img/devguide-cover.png) background-size: contain background-position: 50% 50% .red[ recommendations<br>not principles ] -- .red[ semi-structured<br> collection of<br> best practices ] --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # rOpenSci & Software Peer Review: Scope Current scope includes software for: .left-column[ - data retrieval - data extraction - data munging - data deposition - data validation and testing - workflow automation - version control - citation management and bibliometrics ] .right-column[ - scientific software wrappers - field and laboratory reproducibility tools - database software bindings - geospatial data - text analysis ] --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # rOpenSci & Software Peer Review: Scope Current scope ~~includes~~ does NOT include software for: - statistics ... even through R is an<br>"environment for statistical computing" --- class: center, top background-image: url(data:image/png;base64,#img/SloanLogo-1B-SMALL-Gold-Blue.png) background-size: contain background-position: 50% 50% -- # Expanding Software Peer Review -- .fontBIG[ Expand Scope<br>to include<br>Statistical Software ] --- class: left, top, inverse background-image: url(data:image/png;base64,#img/icon_lettering_color_large-dark.png) background-size: contain background-position: 50% 50% # Statistical Software Peer Review - ### &#9989; 1. Define "Statistical Software" - ### &#9989; 2. Develop Standards for Statistical Software - ### &#9989; 3. Develop Tools to Aid Software Development - ### &#9989; 4. Automate System for Software Submissions - ### &#9989; 5. Open Peer Review System to Statistical Software <br><br> (Status: September 2021) --- class: left, top, inverse background-image: url(data:image/png;base64,#img/icon_lettering_color_large-dark.png) background-size: contain background-position: 50% 50% # Statistical Software Peer Review <br> ## `devguide.ropensci.org` +<br>`stats-devguide.ropensci.org` --- class: left, top, inverse background-image: url(data:image/png;base64,#img/icon_lettering_color_large-dark.png) background-size: contain background-position: 50% 50% # Statistical Software Peer Review - ### &#9989; 1. Define "Statistical Software" --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # Statistical Software Peer Review ## What is Statistical Software? - Empirical Analysis - Data: All abstracts from major statistics conferences (JSM, SDSS, ... n ~ 20,000) - Methods: Natural Language Processing &#8594; concept clusters &#8594; network analyses - Results ... --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # Statistical Software Peer Review ## What is Statistical Software? .left-column[ - Bayesian and Monte Carlo - Regression and Supervised Learning - Dimensionality Reduction, Clustering, and Unsupervised Learning - Exploratory Data Analysis (EDA) and Summary Statistics ] .right-column[ - Time Series Analyses - Machine Learning - Spatial Analyses - ~~Wrapper Packages~~ - ~~Network Analysis~~ - ~~Probability Distributions~~ - ~~Workflow Support~~ ] --- class: left, top, inverse background-image: url(data:image/png;base64,#img/icon_lettering_color_large-dark.png) background-size: contain background-position: 50% 50% # Statistical Software Peer Review - ### &#9989; 1. Define "Statistical Software" -- - ### &#9989; 2. Develop Standards for Statistical Software --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # Statistical Software Standards ## Methodology (Specific): - Collate lists of software packages in each category - "Read" code in each package - "Use" each package to provide insight into: - User interfaces - Package design - Documentation - ...? - Take notes on good and bad practices - Translate these into general principles --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # Statistical Software Standards ## Methodology (General): - Translate observed practices into principles - Write these principles as standards - Selectively elevate standards to *General* - Iterate, remove, add, refine, restructure standards --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # Statistical Software Standards ## Results - Input from > 20 people - 40 General Standards - 265 Category-Specific Standards - 7 Categories - Four Further Categories in Development -- - `devguide.ropensci.org`: 27,000 words - `stats-devguide.ropensci.org`: 48,000 words --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # Statistical Software Standards ### [General & Category-Specific Standards](https://stats-devguide.ropensci.org/standards.html)<br>generally divided into standards for: - Documentation - Input Structures - Algorithms - Output Structures - Testing --- class: left, top, inverse background-image: url(data:image/png;base64,#img/icon_lettering_color_large-dark.png) background-size: contain background-position: 50% 50% # Statistical Software Peer Review - ### &#9989; 1. Define "Statistical Software" - ### &#9989; 2. Develop Standards for Statistical Software - ### &#9989; 3. Develop Tools to Aid Software Development --- class: left, top, inverse background-image: url(data:image/png;base64,#img/package-flow.png) background-size: contain background-position: 50% 50% --- class: center, middle, inverse # The 'srr' package ## `srr` = "Software Review Roclets" (A "roclet" is a documentation engine,<br>or "doclet", for the **R** language.) ## github.com/ropensci-review-tools ## search: "ropensci srr" --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package: How? - [Enable standards compliance statements to be included within code itself, at the point(s) at which each standard is actually addressed.](https://ropensci-review-tools.github.io/roreviewapi/static/tsbox_srraad57cea.html) -- - One function dumps full text of standards in file `R/srr-stats-standards.R` - All are initially tagged `@srrstatsTODO` - Standards are addressed by changing tag to `@srrstats` and moving to location within code where standard is addressed --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package: How? - Reports on standards compliance automatically generated locally each time documentation is updated. ```r roxygen2::roxygenise(path) ``` ``` ## ℹ Loading demo ## ## ── rOpenSci Statistical Software Standards ─────────────────────────────────────────────────────────────────────────── ## ## ── @srrstats standards: ## • [G1.0] in function 'addtwo()' on line#11 of file [R/test.R] ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package: Summary - Insert standards into package with one line of code - Comply with standards - Update documentation to automatically generate compliance report - Generate hyperlinked `html` version with `srr_report()` function - Also extensible, with alternative standards<br>substituted by modifying a single URL --- class: left, top, inverse background-image: url(data:image/png;base64,#img/package-flow.png) background-size: contain background-position: 50% 50% --- class: center, middle, inverse # autotest ## github.com/ropensci-review-tools ## search: "ropensci autotest" --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package: Why? ## rOpenSci Software Peer Review - Reviewers often the first "external" users of submitted packages - Quickly uncover bugs, distracting reviews from higher-level focus on software quality - A tool to help uncover bugs prior to submission should increase software quality --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package: How? - A form of "mutation testing" - Extracts documented example code to identify all inputs to, and outputs from, all functions - Mutates those inputs - Also matches input and output types to descriptions --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package Example: `autotest`ing the `stats` package ```r a <- autotest_package (package = "stats", functions = "var", test = FALSE) # default ``` - `package` argument can be name of an installed package, or path to local source directory. - `functions` argument can restrict tests to specified functions only. - `test = FALSE` returns a list of tests without actually performing them. --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package Example: `autotest`ing the `stats` package ```r a <- autotest_package (package = "stats", functions = "var", test = FALSE) # default ``` ``` ## ## ── autotesting stats ## ✔ [1 / 4]: var ## ✔ [2 / 4]: cor ## ✔ [3 / 4]: cov ## ✔ [4 / 4]: cov ``` ```r dim (a) ``` ``` ## [1] 150 9 ``` &#8594; 150 tests on 4 functions --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package Example: `autotest`ing the `stats` package ``` ## type test_name fn_name parameter parameter_type ## 1 dummy int_as_numeric var x integer vector ## 2 dummy vector_custom_class var x vector ## 3 dummy vector_to_list_col var x vector ## 4 dummy negate_logical var na.rm single logical ## 5 dummy subst_int_for_logical var na.rm single logical ## 6 dummy subst_char_for_logical var na.rm single logical ## 7 dummy single_par_as_length_2 var na.rm single logical ## 8 dummy return_successful var (return object) (return object) ## 9 dummy return_val_described var (return object) (return object) ## 10 dummy return_desc_includes_class var (return object) (return object) ## 11 dummy return_class_matches_desc var (return object) (return object) ## 12 dummy par_is_documented var x <NA> ## 13 dummy par_matches_docs var x <NA> ## 14 dummy par_is_documented var na.rm <NA> ## 15 dummy par_matches_docs var na.rm <NA> ## 16 dummy int_as_numeric var y integer vector ## 17 dummy vector_custom_class var y vector ## 18 dummy vector_to_list_col var y vector ## 19 dummy par_is_documented var y <NA> ## 20 dummy par_matches_docs var y <NA> ## 21 dummy int_as_numeric cor x integer vector ## 22 dummy vector_custom_class cor x vector ## 23 dummy vector_to_list_col cor x vector ## 24 dummy int_as_numeric cor y integer vector ## 25 dummy vector_custom_class cor y vector ## 26 dummy vector_to_list_col cor y vector ## 27 dummy single_char_case cor use single character ## 28 dummy single_char_case cor use single character ## 29 dummy random_char_string cor use single character ## 30 dummy single_par_as_length_2 cor use single character ## 31 dummy single_char_case cor method single character ## 32 dummy single_char_case cor method single character ## 33 dummy random_char_string cor method single character ## 34 dummy single_par_as_length_2 cor method single character ## 35 dummy return_successful cor (return object) (return object) ## 36 dummy return_val_described cor (return object) (return object) ## 37 dummy return_desc_includes_class cor (return object) (return object) ## 38 dummy return_class_matches_desc cor (return object) (return object) ## 39 dummy par_is_documented cor x <NA> ## 40 dummy par_matches_docs cor x <NA> ## 41 dummy par_is_documented cor y <NA> ## 42 dummy par_matches_docs cor y <NA> ## 43 dummy par_is_documented cor use <NA> ## 44 dummy par_matches_docs cor use <NA> ## 45 dummy par_is_documented cor method <NA> ## 46 dummy par_matches_docs cor method <NA> ## 47 dummy single_char_case cor use single character ## 48 dummy single_char_case cor use single character ## 49 dummy random_char_string cor use single character ## 50 dummy single_par_as_length_2 cor use single character ## 51 dummy single_char_case cor method single character ## 52 dummy single_char_case cor method single character ## 53 dummy random_char_string cor method single character ## 54 dummy single_par_as_length_2 cor method single character ## 55 dummy return_successful cor (return object) (return object) ## 56 dummy return_val_described cor (return object) (return object) ## 57 dummy return_desc_includes_class cor (return object) (return object) ## 58 dummy return_class_matches_desc cor (return object) (return object) ## 59 dummy par_is_documented cor x <NA> ## 60 dummy par_matches_docs cor x <NA> ## 61 dummy par_is_documented cor use <NA> ## 62 dummy par_matches_docs cor use <NA> ## 63 dummy par_is_documented cor method <NA> ## 64 dummy par_matches_docs cor method <NA> ## 65 dummy rect_as_other cov x data.frame ## 66 dummy rect_as_other cov x data.frame ## 67 dummy rect_compare_dims cov x data.frame ## 68 dummy rect_compare_dims cov x data.frame ## 69 dummy rect_compare_col_names cov x data.frame ## 70 dummy rect_compare_col_names cov x data.frame ## 71 dummy rect_compare_col_structure cov x data.frame ## 72 dummy rect_compare_col_structure cov x data.frame ## 73 dummy extend_rect_class cov x data.frame ## 74 dummy replace_rect_class cov x data.frame ## 75 dummy single_char_case cov use single character ## 76 dummy single_char_case cov use single character ## 77 dummy random_char_string cov use single character ## 78 dummy single_par_as_length_2 cov use single character ## 79 dummy single_char_case cov method single character ## 80 dummy single_char_case cov method single character ## 81 dummy random_char_string cov method single character ## 82 dummy single_par_as_length_2 cov method single character ## 83 dummy return_successful cov (return object) (return object) ## 84 dummy return_val_described cov (return object) (return object) ## 85 dummy return_desc_includes_class cov (return object) (return object) ## 86 dummy return_class_matches_desc cov (return object) (return object) ## 87 dummy par_is_documented cov x <NA> ## 88 dummy par_matches_docs cov x <NA> ## 89 dummy par_is_documented cov use <NA> ## 90 dummy par_matches_docs cov use <NA> ## 91 dummy par_is_documented cov method <NA> ## 92 dummy par_matches_docs cov method <NA> ## 93 dummy rect_as_other cov x data.frame ## 94 dummy rect_as_other cov x data.frame ## 95 dummy rect_compare_dims cov x data.frame ## 96 dummy rect_compare_dims cov x data.frame ## 97 dummy rect_compare_col_names cov x data.frame ## 98 dummy rect_compare_col_names cov x data.frame ## 99 dummy rect_compare_col_structure cov x data.frame ## 100 dummy rect_compare_col_structure cov x data.frame ## 101 dummy extend_rect_class cov x data.frame ## 102 dummy replace_rect_class cov x data.frame ## 103 dummy single_char_case cov use single character ## 104 dummy single_char_case cov use single character ## 105 dummy random_char_string cov use single character ## 106 dummy single_par_as_length_2 cov use single character ## 107 dummy single_char_case cov method single character ## 108 dummy single_char_case cov method single character ## 109 dummy random_char_string cov method single character ## 110 dummy single_par_as_length_2 cov method single character ## 111 dummy return_successful cov (return object) (return object) ## 112 dummy return_val_described cov (return object) (return object) ## 113 dummy return_desc_includes_class cov (return object) (return object) ## 114 dummy return_class_matches_desc cov (return object) (return object) ## 115 dummy par_is_documented cov x <NA> ## 116 dummy par_matches_docs cov x <NA> ## 117 dummy par_is_documented cov use <NA> ## 118 dummy par_matches_docs cov use <NA> ## 119 dummy par_is_documented cov method <NA> ## 120 dummy par_matches_docs cov method <NA> ## 121 dummy rect_as_other cov x data.frame ## 122 dummy rect_as_other cov x data.frame ## 123 dummy rect_compare_dims cov x data.frame ## 124 dummy rect_compare_dims cov x data.frame ## 125 dummy rect_compare_col_names cov x data.frame ## 126 dummy rect_compare_col_names cov x data.frame ## 127 dummy rect_compare_col_structure cov x data.frame ## 128 dummy rect_compare_col_structure cov x data.frame ## 129 dummy extend_rect_class cov x data.frame ## 130 dummy replace_rect_class cov x data.frame ## 131 dummy single_char_case cov use single character ## 132 dummy single_char_case cov use single character ## 133 dummy random_char_string cov use single character ## 134 dummy single_par_as_length_2 cov use single character ## 135 dummy single_char_case cov method single character ## 136 dummy single_char_case cov method single character ## 137 dummy random_char_string cov method single character ## 138 dummy single_par_as_length_2 cov method single character ## 139 dummy return_successful cov (return object) (return object) ## 140 dummy return_val_described cov (return object) (return object) ## 141 dummy return_desc_includes_class cov (return object) (return object) ## 142 dummy return_class_matches_desc cov (return object) (return object) ## 143 dummy par_is_documented cov x <NA> ## 144 dummy par_matches_docs cov x <NA> ## 145 dummy par_is_documented cov use <NA> ## 146 dummy par_matches_docs cov use <NA> ## 147 dummy par_is_documented cov method <NA> ## 148 dummy par_matches_docs cov method <NA> ## 149 warning par_is_demonstrated var use <NA> ## 150 warning par_is_demonstrated cov y <NA> ## operation ## 1 Integer value converted to numeric ## 2 Custom class definitions for vector input ## 3 Convert vector input to list-columns ## 4 Negate default value of logical parameter ## 5 Substitute integer values for logical parameter ## 6 Substitute character values for logical parameter ## 7 Length 2 vector for length 1 parameter ## 8 Check that function successfully returns an object ## 9 Check that description has return value ## 10 Check whether description of return value specifies class ## 11 Compare class of return value with description ## 12 Check that parameter is documented ## 13 Check that documentation matches class of input parameter ## 14 Check that parameter is documented ## 15 Check that documentation matches class of input parameter ## 16 Integer value converted to numeric ## 17 Custom class definitions for vector input ## 18 Convert vector input to list-columns ## 19 Check that parameter is documented ## 20 Check that documentation matches class of input parameter ## 21 Integer value converted to numeric ## 22 Custom class definitions for vector input ## 23 Convert vector input to list-columns ## 24 Integer value converted to numeric ## 25 Custom class definitions for vector input ## 26 Convert vector input to list-columns ## 27 lower-case character parameter ## 28 upper-case character parameter ## 29 random character string as parameter ## 30 Length 2 vector for length 1 parameter ## 31 lower-case character parameter ## 32 upper-case character parameter ## 33 random character string as parameter ## 34 Length 2 vector for length 1 parameter ## 35 Check that function successfully returns an object ## 36 Check that description has return value ## 37 Check whether description of return value specifies class ## 38 Compare class of return value with description ## 39 Check that parameter is documented ## 40 Check that documentation matches class of input parameter ## 41 Check that parameter is documented ## 42 Check that documentation matches class of input parameter ## 43 Check that parameter is documented ## 44 Check that documentation matches class of input parameter ## 45 Check that parameter is documented ## 46 Check that documentation matches class of input parameter ## 47 lower-case character parameter ## 48 upper-case character parameter ## 49 random character string as parameter ## 50 Length 2 vector for length 1 parameter ## 51 lower-case character parameter ## 52 upper-case character parameter ## 53 random character string as parameter ## 54 Length 2 vector for length 1 parameter ## 55 Check that function successfully returns an object ## 56 Check that description has return value ## 57 Check whether description of return value specifies class ## 58 Compare class of return value with description ## 59 Check that parameter is documented ## 60 Check that documentation matches class of input parameter ## 61 Check that parameter is documented ## 62 Check that documentation matches class of input parameter ## 63 Check that parameter is documented ## 64 Check that documentation matches class of input parameter ## 65 Convert [data.frame] to [tibble] ## 66 Convert [data.frame] to [data.table] ## 67 Convert [data.frame] to [tibble] ## 68 Convert [data.frame] to [data.table] ## 69 Convert [data.frame] to [tibble] ## 70 Convert [data.frame] to [data.table] ## 71 Convert [data.frame] to [tibble] ## 72 Convert [data.frame] to [data.table] ## 73 Extend existent class [data.frame] with new class ## 74 Replace class [data.frame] with new class ## 75 lower-case character parameter ## 76 upper-case character parameter ## 77 random character string as parameter ## 78 Length 2 vector for length 1 parameter ## 79 lower-case character parameter ## 80 upper-case character parameter ## 81 random character string as parameter ## 82 Length 2 vector for length 1 parameter ## 83 Check that function successfully returns an object ## 84 Check that description has return value ## 85 Check whether description of return value specifies class ## 86 Compare class of return value with description ## 87 Check that parameter is documented ## 88 Check that documentation matches class of input parameter ## 89 Check that parameter is documented ## 90 Check that documentation matches class of input parameter ## 91 Check that parameter is documented ## 92 Check that documentation matches class of input parameter ## 93 Convert [data.frame] to [tibble] ## 94 Convert [data.frame] to [data.table] ## 95 Convert [data.frame] to [tibble] ## 96 Convert [data.frame] to [data.table] ## 97 Convert [data.frame] to [tibble] ## 98 Convert [data.frame] to [data.table] ## 99 Convert [data.frame] to [tibble] ## 100 Convert [data.frame] to [data.table] ## 101 Extend existent class [data.frame] with new class ## 102 Replace class [data.frame] with new class ## 103 lower-case character parameter ## 104 upper-case character parameter ## 105 random character string as parameter ## 106 Length 2 vector for length 1 parameter ## 107 lower-case character parameter ## 108 upper-case character parameter ## 109 random character string as parameter ## 110 Length 2 vector for length 1 parameter ## 111 Check that function successfully returns an object ## 112 Check that description has return value ## 113 Check whether description of return value specifies class ## 114 Compare class of return value with description ## 115 Check that parameter is documented ## 116 Check that documentation matches class of input parameter ## 117 Check that parameter is documented ## 118 Check that documentation matches class of input parameter ## 119 Check that parameter is documented ## 120 Check that documentation matches class of input parameter ## 121 Convert [data.frame] to [tibble] ## 122 Convert [data.frame] to [data.table] ## 123 Convert [data.frame] to [tibble] ## 124 Convert [data.frame] to [data.table] ## 125 Convert [data.frame] to [tibble] ## 126 Convert [data.frame] to [data.table] ## 127 Convert [data.frame] to [tibble] ## 128 Convert [data.frame] to [data.table] ## 129 Extend existent class [data.frame] with new class ## 130 Replace class [data.frame] with new class ## 131 lower-case character parameter ## 132 upper-case character parameter ## 133 random character string as parameter ## 134 Length 2 vector for length 1 parameter ## 135 lower-case character parameter ## 136 upper-case character parameter ## 137 random character string as parameter ## 138 Length 2 vector for length 1 parameter ## 139 Check that function successfully returns an object ## 140 Check that description has return value ## 141 Check whether description of return value specifies class ## 142 Compare class of return value with description ## 143 Check that parameter is documented ## 144 Check that documentation matches class of input parameter ## 145 Check that parameter is documented ## 146 Check that documentation matches class of input parameter ## 147 Check that parameter is documented ## 148 Check that documentation matches class of input parameter ## 149 Check that parameter usage is demonstrated ## 150 Check that parameter usage is demonstrated ## content test ## 1 (Should yield same result) TRUE ## 2 (Should yield same result) TRUE ## 3 (Should yield same result) TRUE ## 4 (Function call should still work) TRUE ## 5 (Function call should still work unless explicitly prevented) TRUE ## 6 should trigger warning or error TRUE ## 7 Should trigger message, warning, or error TRUE ## 8 <NA> TRUE ## 9 <NA> TRUE ## 10 <NA> TRUE ## 11 <NA> TRUE ## 12 <NA> TRUE ## 13 <NA> TRUE ## 14 <NA> TRUE ## 15 <NA> TRUE ## 16 (Should yield same result) TRUE ## 17 (Should yield same result) TRUE ## 18 (Should yield same result) TRUE ## 19 <NA> TRUE ## 20 <NA> TRUE ## 21 (Should yield same result) TRUE ## 22 (Should yield same result) TRUE ## 23 (Should yield same result) TRUE ## 24 (Should yield same result) TRUE ## 25 (Should yield same result) TRUE ## 26 (Should yield same result) TRUE ## 27 (Should yield same result) TRUE ## 28 (Should yield same result) TRUE ## 29 Should error TRUE ## 30 Should trigger message, warning, or error TRUE ## 31 (Should yield same result) TRUE ## 32 (Should yield same result) TRUE ## 33 Should error TRUE ## 34 Should trigger message, warning, or error TRUE ## 35 <NA> TRUE ## 36 <NA> TRUE ## 37 <NA> TRUE ## 38 <NA> TRUE ## 39 <NA> TRUE ## 40 <NA> TRUE ## 41 <NA> TRUE ## 42 <NA> TRUE ## 43 <NA> TRUE ## 44 <NA> TRUE ## 45 <NA> TRUE ## 46 <NA> TRUE ## 47 (Should yield same result) TRUE ## 48 (Should yield same result) TRUE ## 49 Should error TRUE ## 50 Should trigger message, warning, or error TRUE ## 51 (Should yield same result) TRUE ## 52 (Should yield same result) TRUE ## 53 Should error TRUE ## 54 Should trigger message, warning, or error TRUE ## 55 <NA> TRUE ## 56 <NA> TRUE ## 57 <NA> TRUE ## 58 <NA> TRUE ## 59 <NA> TRUE ## 60 <NA> TRUE ## 61 <NA> TRUE ## 62 <NA> TRUE ## 63 <NA> TRUE ## 64 <NA> TRUE ## 65 check for error/warning messages TRUE ## 66 check for error/warning messages TRUE ## 67 expect dimensions are same TRUE ## 68 expect dimensions are same TRUE ## 69 expect column names are retained TRUE ## 70 expect column names are retained TRUE ## 71 expect all columns retain identical structure TRUE ## 72 expect all columns retain identical structure TRUE ## 73 (Should yield same result) TRUE ## 74 (Should yield same result) TRUE ## 75 (Should yield same result) TRUE ## 76 (Should yield same result) TRUE ## 77 Should error TRUE ## 78 Should trigger message, warning, or error TRUE ## 79 (Should yield same result) TRUE ## 80 (Should yield same result) TRUE ## 81 Should error TRUE ## 82 Should trigger message, warning, or error TRUE ## 83 <NA> TRUE ## 84 <NA> TRUE ## 85 <NA> TRUE ## 86 <NA> TRUE ## 87 <NA> TRUE ## 88 <NA> TRUE ## 89 <NA> TRUE ## 90 <NA> TRUE ## 91 <NA> TRUE ## 92 <NA> TRUE ## 93 check for error/warning messages TRUE ## 94 check for error/warning messages TRUE ## 95 expect dimensions are same TRUE ## 96 expect dimensions are same TRUE ## 97 expect column names are retained TRUE ## 98 expect column names are retained TRUE ## 99 expect all columns retain identical structure TRUE ## 100 expect all columns retain identical structure TRUE ## 101 (Should yield same result) TRUE ## 102 (Should yield same result) TRUE ## 103 (Should yield same result) TRUE ## 104 (Should yield same result) TRUE ## 105 Should error TRUE ## 106 Should trigger message, warning, or error TRUE ## 107 (Should yield same result) TRUE ## 108 (Should yield same result) TRUE ## 109 Should error TRUE ## 110 Should trigger message, warning, or error TRUE ## 111 <NA> TRUE ## 112 <NA> TRUE ## 113 <NA> TRUE ## 114 <NA> TRUE ## 115 <NA> TRUE ## 116 <NA> TRUE ## 117 <NA> TRUE ## 118 <NA> TRUE ## 119 <NA> TRUE ## 120 <NA> TRUE ## 121 check for error/warning messages TRUE ## 122 check for error/warning messages TRUE ## 123 expect dimensions are same TRUE ## 124 expect dimensions are same TRUE ## 125 expect column names are retained TRUE ## 126 expect column names are retained TRUE ## 127 expect all columns retain identical structure TRUE ## 128 expect all columns retain identical structure TRUE ## 129 (Should yield same result) TRUE ## 130 (Should yield same result) TRUE ## 131 (Should yield same result) TRUE ## 132 (Should yield same result) TRUE ## 133 Should error TRUE ## 134 Should trigger message, warning, or error TRUE ## 135 (Should yield same result) TRUE ## 136 (Should yield same result) TRUE ## 137 Should error TRUE ## 138 Should trigger message, warning, or error TRUE ## 139 <NA> TRUE ## 140 <NA> TRUE ## 141 <NA> TRUE ## 142 <NA> TRUE ## 143 <NA> TRUE ## 144 <NA> TRUE ## 145 <NA> TRUE ## 146 <NA> TRUE ## 147 <NA> TRUE ## 148 <NA> TRUE ## 149 Examples do not demonstrate usage of this parameter TRUE ## 150 Examples do not demonstrate usage of this parameter TRUE ## yaml_hash ## 1 02348a11fab40b2a91b30d7859acec77 ## 2 02348a11fab40b2a91b30d7859acec77 ## 3 02348a11fab40b2a91b30d7859acec77 ## 4 02348a11fab40b2a91b30d7859acec77 ## 5 02348a11fab40b2a91b30d7859acec77 ## 6 02348a11fab40b2a91b30d7859acec77 ## 7 02348a11fab40b2a91b30d7859acec77 ## 8 02348a11fab40b2a91b30d7859acec77 ## 9 02348a11fab40b2a91b30d7859acec77 ## 10 02348a11fab40b2a91b30d7859acec77 ## 11 02348a11fab40b2a91b30d7859acec77 ## 12 02348a11fab40b2a91b30d7859acec77 ## 13 02348a11fab40b2a91b30d7859acec77 ## 14 02348a11fab40b2a91b30d7859acec77 ## 15 02348a11fab40b2a91b30d7859acec77 ## 16 02348a11fab40b2a91b30d7859acec77 ## 17 02348a11fab40b2a91b30d7859acec77 ## 18 02348a11fab40b2a91b30d7859acec77 ## 19 02348a11fab40b2a91b30d7859acec77 ## 20 02348a11fab40b2a91b30d7859acec77 ## 21 02348a11fab40b2a91b30d7859acec77 ## 22 02348a11fab40b2a91b30d7859acec77 ## 23 02348a11fab40b2a91b30d7859acec77 ## 24 02348a11fab40b2a91b30d7859acec77 ## 25 02348a11fab40b2a91b30d7859acec77 ## 26 02348a11fab40b2a91b30d7859acec77 ## 27 02348a11fab40b2a91b30d7859acec77 ## 28 02348a11fab40b2a91b30d7859acec77 ## 29 02348a11fab40b2a91b30d7859acec77 ## 30 02348a11fab40b2a91b30d7859acec77 ## 31 02348a11fab40b2a91b30d7859acec77 ## 32 02348a11fab40b2a91b30d7859acec77 ## 33 02348a11fab40b2a91b30d7859acec77 ## 34 02348a11fab40b2a91b30d7859acec77 ## 35 02348a11fab40b2a91b30d7859acec77 ## 36 02348a11fab40b2a91b30d7859acec77 ## 37 02348a11fab40b2a91b30d7859acec77 ## 38 02348a11fab40b2a91b30d7859acec77 ## 39 02348a11fab40b2a91b30d7859acec77 ## 40 02348a11fab40b2a91b30d7859acec77 ## 41 02348a11fab40b2a91b30d7859acec77 ## 42 02348a11fab40b2a91b30d7859acec77 ## 43 02348a11fab40b2a91b30d7859acec77 ## 44 02348a11fab40b2a91b30d7859acec77 ## 45 02348a11fab40b2a91b30d7859acec77 ## 46 02348a11fab40b2a91b30d7859acec77 ## 47 0d9d96bd53607623a1a051557c409662 ## 48 0d9d96bd53607623a1a051557c409662 ## 49 0d9d96bd53607623a1a051557c409662 ## 50 0d9d96bd53607623a1a051557c409662 ## 51 0d9d96bd53607623a1a051557c409662 ## 52 0d9d96bd53607623a1a051557c409662 ## 53 0d9d96bd53607623a1a051557c409662 ## 54 0d9d96bd53607623a1a051557c409662 ## 55 0d9d96bd53607623a1a051557c409662 ## 56 0d9d96bd53607623a1a051557c409662 ## 57 0d9d96bd53607623a1a051557c409662 ## 58 0d9d96bd53607623a1a051557c409662 ## 59 0d9d96bd53607623a1a051557c409662 ## 60 0d9d96bd53607623a1a051557c409662 ## 61 0d9d96bd53607623a1a051557c409662 ## 62 0d9d96bd53607623a1a051557c409662 ## 63 0d9d96bd53607623a1a051557c409662 ## 64 0d9d96bd53607623a1a051557c409662 ## 65 0d9d96bd53607623a1a051557c409662 ## 66 0d9d96bd53607623a1a051557c409662 ## 67 0d9d96bd53607623a1a051557c409662 ## 68 0d9d96bd53607623a1a051557c409662 ## 69 0d9d96bd53607623a1a051557c409662 ## 70 0d9d96bd53607623a1a051557c409662 ## 71 0d9d96bd53607623a1a051557c409662 ## 72 0d9d96bd53607623a1a051557c409662 ## 73 0d9d96bd53607623a1a051557c409662 ## 74 0d9d96bd53607623a1a051557c409662 ## 75 0d9d96bd53607623a1a051557c409662 ## 76 0d9d96bd53607623a1a051557c409662 ## 77 0d9d96bd53607623a1a051557c409662 ## 78 0d9d96bd53607623a1a051557c409662 ## 79 0d9d96bd53607623a1a051557c409662 ## 80 0d9d96bd53607623a1a051557c409662 ## 81 0d9d96bd53607623a1a051557c409662 ## 82 0d9d96bd53607623a1a051557c409662 ## 83 0d9d96bd53607623a1a051557c409662 ## 84 0d9d96bd53607623a1a051557c409662 ## 85 0d9d96bd53607623a1a051557c409662 ## 86 0d9d96bd53607623a1a051557c409662 ## 87 0d9d96bd53607623a1a051557c409662 ## 88 0d9d96bd53607623a1a051557c409662 ## 89 0d9d96bd53607623a1a051557c409662 ## 90 0d9d96bd53607623a1a051557c409662 ## 91 0d9d96bd53607623a1a051557c409662 ## 92 0d9d96bd53607623a1a051557c409662 ## 93 66ebac03416364077c042d0d7ddfd4ed ## 94 66ebac03416364077c042d0d7ddfd4ed ## 95 66ebac03416364077c042d0d7ddfd4ed ## 96 66ebac03416364077c042d0d7ddfd4ed ## 97 66ebac03416364077c042d0d7ddfd4ed ## 98 66ebac03416364077c042d0d7ddfd4ed ## 99 66ebac03416364077c042d0d7ddfd4ed ## 100 66ebac03416364077c042d0d7ddfd4ed ## 101 66ebac03416364077c042d0d7ddfd4ed ## 102 66ebac03416364077c042d0d7ddfd4ed ## 103 66ebac03416364077c042d0d7ddfd4ed ## 104 66ebac03416364077c042d0d7ddfd4ed ## 105 66ebac03416364077c042d0d7ddfd4ed ## 106 66ebac03416364077c042d0d7ddfd4ed ## 107 66ebac03416364077c042d0d7ddfd4ed ## 108 66ebac03416364077c042d0d7ddfd4ed ## 109 66ebac03416364077c042d0d7ddfd4ed ## 110 66ebac03416364077c042d0d7ddfd4ed ## 111 66ebac03416364077c042d0d7ddfd4ed ## 112 66ebac03416364077c042d0d7ddfd4ed ## 113 66ebac03416364077c042d0d7ddfd4ed ## 114 66ebac03416364077c042d0d7ddfd4ed ## 115 66ebac03416364077c042d0d7ddfd4ed ## 116 66ebac03416364077c042d0d7ddfd4ed ## 117 66ebac03416364077c042d0d7ddfd4ed ## 118 66ebac03416364077c042d0d7ddfd4ed ## 119 66ebac03416364077c042d0d7ddfd4ed ## 120 66ebac03416364077c042d0d7ddfd4ed ## 121 54de9419436dfaed52b8ea2055a394d1 ## 122 54de9419436dfaed52b8ea2055a394d1 ## 123 54de9419436dfaed52b8ea2055a394d1 ## 124 54de9419436dfaed52b8ea2055a394d1 ## 125 54de9419436dfaed52b8ea2055a394d1 ## 126 54de9419436dfaed52b8ea2055a394d1 ## 127 54de9419436dfaed52b8ea2055a394d1 ## 128 54de9419436dfaed52b8ea2055a394d1 ## 129 54de9419436dfaed52b8ea2055a394d1 ## 130 54de9419436dfaed52b8ea2055a394d1 ## 131 54de9419436dfaed52b8ea2055a394d1 ## 132 54de9419436dfaed52b8ea2055a394d1 ## 133 54de9419436dfaed52b8ea2055a394d1 ## 134 54de9419436dfaed52b8ea2055a394d1 ## 135 54de9419436dfaed52b8ea2055a394d1 ## 136 54de9419436dfaed52b8ea2055a394d1 ## 137 54de9419436dfaed52b8ea2055a394d1 ## 138 54de9419436dfaed52b8ea2055a394d1 ## 139 54de9419436dfaed52b8ea2055a394d1 ## 140 54de9419436dfaed52b8ea2055a394d1 ## 141 54de9419436dfaed52b8ea2055a394d1 ## 142 54de9419436dfaed52b8ea2055a394d1 ## 143 54de9419436dfaed52b8ea2055a394d1 ## 144 54de9419436dfaed52b8ea2055a394d1 ## 145 54de9419436dfaed52b8ea2055a394d1 ## 146 54de9419436dfaed52b8ea2055a394d1 ## 147 54de9419436dfaed52b8ea2055a394d1 ## 148 54de9419436dfaed52b8ea2055a394d1 ## 149 <NA> ## 150 <NA> ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package Example: `autotest`ing the `stats` package ``` ## test_name n ## 1 extend_rect_class 3 ## 2 int_as_numeric 4 ## 3 negate_logical 1 ## 4 par_is_demonstrated 2 ## 5 par_is_documented 19 ## 6 par_matches_docs 19 ## 7 random_char_string 10 ## 8 rect_as_other 6 ## 9 rect_compare_col_names 6 ## 10 rect_compare_col_structure 6 ## 11 rect_compare_dims 6 ## 12 replace_rect_class 3 ## 13 return_class_matches_desc 6 ## 14 return_desc_includes_class 6 ## 15 return_successful 6 ## 16 return_val_described 6 ## 17 single_char_case 20 ## 18 single_par_as_length_2 11 ## 19 subst_char_for_logical 1 ## 20 subst_int_for_logical 1 ## 21 vector_custom_class 4 ## 22 vector_to_list_col 4 ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package Example: `autotest`ing the `stats` package ```r a <- autotest_package (package = "stats", functions = "var", test = TRUE) # takes 20-30 seconds ``` ``` ## ## ── autotesting stats ## ✔ [1 / 4]: var ## ✔ [2 / 4]: cor ## ✔ [3 / 4]: cov ## ✔ [4 / 4]: cov ``` ```r dim (a) ``` ``` ## [1] 15 9 ``` 150 tests &#8594; 15 results --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package Example: `autotest`ing the `stats` package ``` ## # A tibble: 15 × 9 ## type test_name fn_name parameter parameter_type operation content test ## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <lgl> ## 1 warning par_is_d… var use <NA> Check th… "Examp… TRUE ## 2 warning par_is_d… cov y <NA> Check th… "Examp… TRUE ## 3 diagnostic single_c… cor use single charac… upper-ca… "is ca… TRUE ## 4 diagnostic single_c… cor method single charac… upper-ca… "is ca… TRUE ## 5 diagnostic single_c… cor use single charac… upper-ca… "is ca… TRUE ## 6 diagnostic single_c… cor method single charac… upper-ca… "is ca… TRUE ## 7 diagnostic single_c… cov use single charac… upper-ca… "is ca… TRUE ## 8 diagnostic single_c… cov method single charac… upper-ca… "is ca… TRUE ## 9 diagnostic single_c… cov use single charac… upper-ca… "is ca… TRUE ## 10 diagnostic single_c… cov method single charac… upper-ca… "is ca… TRUE ## 11 diagnostic single_c… cov use single charac… upper-ca… "is ca… TRUE ## 12 diagnostic single_c… cov method single charac… upper-ca… "is ca… TRUE ## 13 diagnostic vector_t… var x vector Convert … "Funct… TRUE ## 14 diagnostic vector_t… var x vector Convert … "Funct… TRUE ## 15 diagnostic vector_t… var y vector Convert … "Funct… TRUE ## # … with 1 more variable: yaml_hash <chr> ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package Example: `autotest`ing the `stats` package - Use `DT` package to inspect results! ```r a <- autotest_package (package = "stats", functions = "var", test = TRUE) # takes 20-30 seconds DT::datatable (a) ``` --- class: left, top, inverse background-image: url(data:image/png;base64,#img/stats-var-TRUE.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package Example: `autotest`ing the `stats` package ```r a <- autotest_package (package = "stats", functions = "var", test = TRUE) ``` - `stats` package is algorithmially robust - But `autotest` reveals a few gaps in documentation! --- class: left, top, inverse background-image: url(data:image/png;base64,#img/package-flow.png) background-size: contain background-position: 50% 50% --- class: left, top, inverse background-image: url(data:image/png;base64,#img/tsbox-pkgcheck.png) background-size: contain background-position: 50% 50% --- class: left, top, inverse background-image: url(data:image/png;base64,#img/tsbox.png) background-size: contain background-position: 50% 50% --- class: left, middle, inverse # Statistical Software Peer Review - Now accepting submissions - Search "rOpenSci statistical software" - `stats-devguide.ropensci.org` - All tools available for use now at `github.com/ropensci-review-tools`