Skip to content

Commit 0ebcf2e

Browse files
jhudsl-robotjhudsl-robot
committed
Render bookdown
1 parent a7c4114 commit 0ebcf2e

20 files changed

+1072
-820
lines changed

‎docs/01-intro.md‎

Lines changed: 48 additions & 82 deletions
Original file line numberDiff line numberDiff line change
@@ -4,87 +4,53 @@
44
# Introduction
55

66

7+
This chapter will be about the concept of reproducible research and we will cover some basic concepts and ideas that are related to reproducible research.
8+
Before we get to *reproducibility*, we need to cover a little background with respect to how science works (even if you're not a scientist, this is important). The basic idea is that in science, *replication* is the most important element of verifying and validating findings. So if you claim that X causes Y, or that Vitamin C improves disease, or that something causes a problem, what happens is that other scientists that are independent of you will try to investigate that same question and see if they come up with a similar result. If lots of different people come up with the same result and replicate the original finding, then we tend to think that the original finding was probably true and that this is a real relationship or real finding.
9+
10+
The ultimate standard in strengthening scientific evidence is replication. The goal is to have independent people to do independent things with different data, different methods, and different laboratories and see if you get the same result. There's a sense that if a relationship in nature is truly there, then it should be robust to having different people discover it in different ways. Replication is particularly important in areas where findings can have big policy impacts or can influence regulatory types of decisions.
11+
12+
13+
14+
## What's Wrong with Replication?
15+
16+
So what's wrong with replication? There's really nothing wrong with it. This is what science has been doing for a long time, through hundreds of years. And there's nothing wrong with it today. But the problem is that it's becoming more and more challenging to do replication or to replicate other studies. Part of the reason is because studies are getting bigger and bigger.
17+
18+
In order to do big studies you need a lot of money and so, well, there's a lot of money involved! If you want to do ten versions of the same study, you need ten times as much money and there's not as much money around as there used to be. Sometimes it's difficult to replicate a study because if the original study took 20 years to do, it's difficult to wait around another 20 years for replication. Some studies are just plain unique, such as studying the impact of a massive earthquake in a very specific location and time. If you're looking at a unique situation in time or a unique population, you can't readily replicate that situation.
19+
20+
There are a lot of good reasons why you can't replicate a study. If you can't replicate a study, is the alternative just to do nothing, just let that study stand by itself? The idea behind a reproducible research is to create a kind of minimum standard or a middle ground where we won't be replicating a study, but maybe we can do something in between. The basic problem is that you have the gold standard, which is replication, and then you have the worst standard which is doing nothing. What can we do that's in between the gold standard and doing nothing? That is where reproducibility comes in. That's how we can kind of bridge the gap between replication and nothing.
21+
22+
In non-research settings, often full replication isn't even the point. Often the goal is to preserve something to the point where anybody in an organization can repeat what you did (for example, after you leave the organization). In this case, reproducibility is key to maintaining the history of a project and making sure that every step along the way is clear.
23+
24+
25+
26+
## Reproducibility to the Rescue
27+
28+
Why do we need this kind of middle ground? We haven't clearly defined reproducibility yet, but the basic idea is that you need to make the **data** available for the original study and the **computational methods** available so that other people can look at your data and run the kind of analysis that you've run, and come to the same findings that you found.
29+
30+
What reproducible research is about is a *validation of the data analysis*. Because you're not collecting independent data using independent methods, it's a little bit more difficult to validate the scientific question itself. But if you can take someone's data and reproduce their findings, then you can, in some sense, validate the data analysis. This involves having the data and the code because more likely than not, the analysis will have been done on the computer using some sort of programming language, like R. So you can take their code and their data and reproduce the findings that they come up with. Then you can at least have confidence that the analysis was done appropriately and that the correct methods were used.
31+
32+
Recently, there's been a lot of discussion of reproducibility in the media and in the scientific literature. The journal *Science* had a special issue on reproducibility and data replication. Other journals have updated policies on publication to encourage reproducibility. In 2012, a feature on the TV show 60 minutes looked at a major incident at Duke University where many results involving a promising cancer test were found to be not reproducible (more on that later). This led to a number of studies and clinical trials having to be stopped, followed by an investigation which is still ongoing.
33+
34+
## Reproducibility Guidelines
35+
36+
Finally, the Institute of Medicine (now the National Academy of Medicine), in response to a lot of events involving reproducibility of scientific studies, issued a report saying that best practices should be done to promote and encourage reproducibility, particularly in what's called 'omics based research, such as genomics, proteomics, other similar areas involving high-throughput biological measurements. This was a very important report. Of the many recommendations that the IOM made, the key ones were that
37+
38+
* Data and metadata need to be made available;
39+
40+
* Computer code should be fully specified, so that people can examine it to see what was done;
41+
42+
* All the steps of the computational analysis, including any preprocessing of data, should be fully described so that people can study it and reproduce it.
43+
44+
We will expand on these ideas in the chapters that follow.
45+
46+
47+
## Summary
48+
49+
* Replication, whereby scientific questions are examined and verified independently by different scientists, is the gold standard for scientific validity.
50+
51+
* Replication can be difficult and often there are no resources to independently replicate a study.
52+
53+
* Reproducibility, whereby data and code are re-analyzed by independent scientists to obtain the same results of the original investigator, is a reasonable minimum standard when replication is not possible.
54+
755

856

9-
## Motivation
10-
11-
12-
## Target Audience
13-
14-
The course is intended for ...
15-
16-
## Curriculum
17-
18-
The course covers...
19-
20-
21-
22-
```r
23-
devtools::session_info()
24-
```
25-
26-
```
27-
## ─ Session info ───────────────────────────────────────────────────────────────
28-
## setting value
29-
## version R version 4.0.2 (2020-06-22)
30-
## os Ubuntu 20.04.5 LTS
31-
## system x86_64, linux-gnu
32-
## ui X11
33-
## language (EN)
34-
## collate en_US.UTF-8
35-
## ctype en_US.UTF-8
36-
## tz Etc/UTC
37-
## date 2023-04-11
38-
##
39-
## ─ Packages ───────────────────────────────────────────────────────────────────
40-
## package * version date lib source
41-
## assertthat 0.2.1 2019-03-21 [1] RSPM (R 4.0.5)
42-
## bookdown * 0.24 2023-03-28 [1] Github (rstudio/bookdown@88bc4ea)
43-
## cachem 1.0.7 2023-02-24 [1] CRAN (R 4.0.2)
44-
## callr 3.5.0 2020-10-08 [1] RSPM (R 4.0.2)
45-
## cli 3.6.1 2023-03-23 [1] CRAN (R 4.0.2)
46-
## crayon 1.3.4 2017-09-16 [1] RSPM (R 4.0.0)
47-
## desc 1.2.0 2018-05-01 [1] RSPM (R 4.0.3)
48-
## devtools 2.3.2 2020-09-18 [1] RSPM (R 4.0.3)
49-
## digest 0.6.25 2020-02-23 [1] RSPM (R 4.0.0)
50-
## ellipsis 0.3.1 2020-05-15 [1] RSPM (R 4.0.3)
51-
## evaluate 0.20 2023-01-17 [1] CRAN (R 4.0.2)
52-
## fansi 0.4.1 2020-01-08 [1] RSPM (R 4.0.0)
53-
## fastmap 1.1.1 2023-02-24 [1] CRAN (R 4.0.2)
54-
## fs 1.5.0 2020-07-31 [1] RSPM (R 4.0.3)
55-
## glue 1.4.2 2020-08-27 [1] RSPM (R 4.0.5)
56-
## hms 0.5.3 2020-01-08 [1] RSPM (R 4.0.0)
57-
## htmltools 0.5.5 2023-03-23 [1] CRAN (R 4.0.2)
58-
## knitr 1.33 2023-03-28 [1] Github (yihui/knitr@a1052d1)
59-
## lifecycle 1.0.3 2022-10-07 [1] CRAN (R 4.0.2)
60-
## magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.0.2)
61-
## memoise 2.0.1 2021-11-26 [1] CRAN (R 4.0.2)
62-
## ottrpal 1.0.1 2023-03-28 [1] Github (jhudsl/ottrpal@151e412)
63-
## pillar 1.9.0 2023-03-22 [1] CRAN (R 4.0.2)
64-
## pkgbuild 1.1.0 2020-07-13 [1] RSPM (R 4.0.2)
65-
## pkgconfig 2.0.3 2019-09-22 [1] RSPM (R 4.0.3)
66-
## pkgload 1.1.0 2020-05-29 [1] RSPM (R 4.0.3)
67-
## prettyunits 1.1.1 2020-01-24 [1] RSPM (R 4.0.3)
68-
## processx 3.4.4 2020-09-03 [1] RSPM (R 4.0.2)
69-
## ps 1.4.0 2020-10-07 [1] RSPM (R 4.0.2)
70-
## R6 2.4.1 2019-11-12 [1] RSPM (R 4.0.0)
71-
## readr 1.4.0 2020-10-05 [1] RSPM (R 4.0.2)
72-
## remotes 2.2.0 2020-07-21 [1] RSPM (R 4.0.3)
73-
## rlang 1.1.0 2023-03-14 [1] CRAN (R 4.0.2)
74-
## rmarkdown 2.10 2023-03-28 [1] Github (rstudio/rmarkdown@02d3c25)
75-
## rprojroot 2.0.3 2022-04-02 [1] CRAN (R 4.0.2)
76-
## sessioninfo 1.1.1 2018-11-05 [1] RSPM (R 4.0.3)
77-
## stringi 1.5.3 2020-09-09 [1] RSPM (R 4.0.3)
78-
## stringr 1.4.0 2019-02-10 [1] RSPM (R 4.0.3)
79-
## testthat 3.0.1 2023-03-28 [1] Github (R-lib/testthat@e99155a)
80-
## tibble 3.2.1 2023-03-20 [1] CRAN (R 4.0.2)
81-
## usethis 1.6.3 2020-09-17 [1] RSPM (R 4.0.2)
82-
## utf8 1.1.4 2018-05-24 [1] RSPM (R 4.0.3)
83-
## vctrs 0.6.1 2023-03-22 [1] CRAN (R 4.0.2)
84-
## withr 2.3.0 2020-09-22 [1] RSPM (R 4.0.2)
85-
## xfun 0.26 2023-03-28 [1] Github (yihui/xfun@74c2a66)
86-
## yaml 2.2.1 2020-02-01 [1] RSPM (R 4.0.3)
87-
##
88-
## [1] /usr/local/lib/R/site-library
89-
## [2] /usr/local/lib/R/library
90-
```

‎docs/02-intro.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
2+
3+
4+
5+
# From "X" to "Computational X"
6+
7+
What is driving this need for a "reproducibility middle ground" between replication and doing nothing? For starters, there are a lot of new technologies on the scene and in many different fields of study including, biology, chemistry and environmental science. These technologies allow us to collect data at a much higher throughput so we end up with these very complex and very high dimensional data sets. These datasets can be collected almost instantaneously compared to even just ten years ago---the technology has allowed us to create huge data sets at essentially the touch of a button. Furthermore, we the computing power to take existing (already huge) databases and merge them into even bigger and bigger databases. Finally, the massive increase in computing power has allowed us to implement more sophisticated and complex analysis routines.
8+
9+
The analyses themselves, the models that we fit and the algorithms that we run, are much much more complicated than they used to be. Having a basic understanding of these algorithms is difficult, even for a sophisticated person, and it's almost impossible to describe these algorithms with words alone. Understanding what someone did in a data analysis now requires looking at *code* and scrutinizing the computer programs that people used.
10+
11+
The bottom line with all these different trends is that for every field "X", there is now "Computational X". There's computational biology, computational astronomy---whatever it is you want, there is a computational version of it.
12+
13+
14+
15+
## Example: Air Pollution and Health
16+
17+
One example of an area were reproducibility is important comes from the area of air pollution and health. Air pollution and health is a big field and it involves a confluence of features that emphasize the need for reproducibility.
18+
19+
The first feature is that we are estimating very small, but very important, public health effects in the presence of a numerous much stronger signals. You can think about air pollution as something that's perhaps harmful, but even if it were harmful there are likely many other things that are going to be more harmful that you have to worry about. Pollution is going to be at the very top of the list of things that are going to harm you. In other words, there's an inherently weak signal there.
20+
21+
Second, the results of a lot of air pollution research inform substantial policy decisions. Many federal air pollution regulations in the United States are based on scientific research in this area and these regulations can affect a lot of stakeholders in government and industry.
22+
23+
Finally, we use a lot of complex statistical methods to do these studies and these statistical methods are subsequently subjected to intense scrutiny. The combination of an inherently weak signal, substantial policy impacts, and complex statistical methods almost require that the research that we do be reproducible.
24+
25+
26+
27+
## The Data Science Pipeline
28+
29+
30+
The basic issue is when you read a description of a data analysis, such as in an article or a technical report, for the most part, what you get is the report and nothing else. Of course, everyone knows that behind the scenes there's *a lot* that went into this report and that's what I call the **data science pipeline**.
31+
32+
![](resources/images/02-intro_files/figure-docx//1-pVlxwkzTzFH4PHWzm9cuXbKpzlbSd1TvXEy60Sste0_p.png){width=100%}
33+
34+
In this pipeline, there are two "actors": the *author* of the report/article and the *reader*. On the left side, the author is going from left to right along this pipeline. The reader is going from right to left. If you're the reader you read the article and you want to know more about what happened: Where is the data? What was used here? The basic idea behind reproducibility is to focus on the analytic data and the computational results. With reproducibility the goal is to allow the author of a report and the reader of that report to "meet in the middle".
35+
36+
## Elements of Reproducibility
37+
38+
What do we need for reproducibility? There's a variety of ways to talk about this, but one basic definition that we've come up with is that there are four things that are required to make results reproducible:
39+
40+
1. **Analytic data**. The data that were used for the analysis that was presented should be available for others to access. This is different from the *raw data* because very often in a data analysis the raw data are not all used for the analysis, but rather some subset is used. It may be interesting to see the raw data but impractical to actually have it. Analytic data is key to examining the data analysis.
41+
42+
2. **Analytic code**. The analytic code is the code that was applied to the analytic data to produce the key results. This may be preprocessing code, regression modeling code, or really any other code used to produce the results from the analytic data.
43+
44+
3. **Documentation**. Documentation of that code and the data is very important.
45+
46+
4. **Distribution**. Finally, there needs to be some standard means of distribution, so all this data in the code is easily accessible.
47+
48+
49+
## Authors and Readers
50+
51+
It is important to realize that there are multiple players when you talk about reproducibility--there are different types of parties that have different types of interests. There are authors who produce research and they want to make their research reproducible. There are also readers of research and they want to reproduce that work. Everyone needs tools to make their lives easier.
52+
53+
One current challenge is that authors of research have to undergo considerable effort to make their results available to a wide audience. Publishing data and code today is not necessarily a trivial task. Although there are a number of resources available now, that were not available even five years ago, it's still a bit of a challenge to get things out on the web (or at least distributed widely). Resources like [GitHub](https://github.com) and various data repositories have made a big difference, but there is still a ways to go with respect to building up the public reproducibility infrastructure.
54+
55+
Furthermore, even when data and code are available, readers often have to download the data, download the code, and then they have to piece everything together, usually by hand. It's not always an easy task to put the data and code together. Also, readers may not have the same computational resources that the original authors did. If the original authors used an enormous computing cluster, for example, to do their analysis, the readers may not have that same enormous computing cluster at their disposal. It may be difficult for readers to reproduce the same results.
56+
57+
Generally the toolbox for doing reproducible research is small, although it's definitely growing. In practice, authors often just throw things up on the web. There are journals and supplementary materials, but they are famously disorganized. There are only a few central databases that authors can take advantage of to post their data and make it available. So if you're working in a field that has a central database that everyone uses, that's great. If you're not, then you have to assemble your own resources. We will discuss the tools for doing reproducible research in the third course in this series.
58+
59+
60+
## Summary
61+
62+
* The process of conducting and disseminating research can be depicted as a "data science pipeline"
63+
64+
* Readers and consumers of data science research are typically not privy to the details of the data science pipeline
65+
66+
* One view of reproducibility is that it gives research consumers partial access to the raw pipeline elements.
67+

0 commit comments

Comments
 (0)