Becoming a better scientist with open and reproducible research (2)

This is an modified version of the Becoming a better scientist with open and reproducible research talk that I gave at TU Delft in May 2019. This version was modified and prepared for the Virtual Bioinformatics Student Symposium on the 4 December 2020. The slides are available here and the recording is available on youtube.

Introduction

I am always particularly keen to discuss open and reproducible research, which is a topic that is very close to my heart. Interacting with early career researchers makes it even more thrilling because those are the ones that will (hopefully) make a bigger change for the best. Believe it or not, ERCs are unlikely to be able to rely on their senior peers to drive that change.

Disclaimer: I wanted to my personal experiences and opinions. I am not speaking from authority here. Authority generally comes from seniority, and in most cases senior academics aren’t those that have much experience in open and reproducible research.

Another thing I don’t plan doing here is listing technical solutions on how to implement open and reproducible research. I would be very happy to answer your technical questions if you have any, or follow up by email or twitter. You’ll also have the oportunity to discuss with your peers during the social gathering, many of which will be able to help very efficiently. The main reason I won’t focus on technical aspects of open and reproducible research is that I very much doubt that technical aspects are the real barrier. We have many solutions at hand. The real challenges are the (academic) environment we are in, the inertia of academia and the vested interests of many senior actors.

For a start, here are three quick remarks about open and reproducible research in relation with the title of the talk/post, and I will of course define/discuss these terms in more details later - I will just assume for now that you know, at least vaguely, what I am referring to.

Open != good (by default)

• A piece of open research doesn’t automatically make it good, where good is defined as of high academic quality. A piece of closed research doesn’t make it bad, where bad here is defined of low academic quality. So openness doesn’t equate to academic quality. But openness provides some desired quality (i.e. desirable property) independent from academic excellent. Openness leads to trust (more details later).

Reproducible != good (by default)

• A piece of reproducible research doesn’t automatically make it good, where good is defined as of high academic quality. A piece of non reproducible research doesn’t make it bad, where bad here is defined of low academic quality. So reproducible doesn’t equate to academic quality. But reproducibility provides some desired quality (i.e. desirable property) independent from academic excellent. Reproducibility leads (among other things) to trust (see details below).

And

Open != reproducible

Many people seem to think that #OpenScience & the reproducibility crisis in psychology are somehow causally related. They are not. Open science is decades old & did not focus on reproducibility as a single issue — more here: https://t.co/KpJHIEqPj3 & here: https://t.co/KdMeK6PCUT pic.twitter.com/qF5yPTqNqu

— Olivia Guest | Ολίβια Γκεστ (@o_guest) December 1, 2018

• Open research and reproducible research aren’t the same thing, and one doesn’t imply the other. Even though in our modern understanding of these terms and concepts, they are intimately linked, historically, they are very different.

While a lot of the things I will talk about are relatively recent (I consider myself a modern open researcher), note that the principles, implicit or explicity of open research/science aren’t new. I was pleasantly surprised when learning about the Mertonian norms (Robert Merton, 1942).

• Communism: all scientists should have common ownership of scientific goods (intellectual property), to promote collective collaboration; secrecy is the opposite of this norm.

• Universalism: scientific validity is independent of the socio-political status/personal attributes of its participants.

• Disinterestedness: scientific institutions act for the benefit of a common scientific enterprise, rather than for the personal gain of individuals within them

• Organised scepticism: scientific claims should be exposed to critical scrutiny before being accepted: both in methodology and institutional codes of conduct.

Motivation

Inverse problems are hard!

Example and figure borrowed from Stephen Eglen.

• Forward problem: I scored 68, what was my grade?

• Inverse problem: I got a B, what was my score?

Research sharing: the inverse problem

Where is the scholarship?

An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and that complete set of instructions that generated the figures.

[Buckheit and Donoho 1995, after Claerbout]

So what is open research?

Open science has seen a continuous evolution since the 17th century, with the advent of dissemination of research in scientific journals and the societal demand to access scientific knowledge at large. Technology and communication has further accelerated this evolution, and put it in the spot light among researchers and academics (for examples funder mandates) and more widely in the press with the cost of publications (see for example this Guardian long read article Is the staggeringly profitable business of scientific publishing bad for science? or the Paywall movie).

So what is open research? What characterises a modern open researcher?

Let’s start with a definition

Open science/research is the process of transparent dissemination and access to knowledge, that can be applied to various scientific practices (image below from Wikipedia):

and a figure

What I dislike about the previous figure is that it can give the misleading impression that open research is about collecting badges, and that the more badges you possess, the better an open researcher you are. And reciprocally, not having any badge to display excludes one from being an open researcher. And as soon as people start to believe this, we will stop practicing open research and will start doing stamp collection.

Incidently, here’s an ironic tweet that offers tips to do exactly that:

21 Tips on how to sound #openscience: in the last three weeks before x-mas, I will tweet on tip each day on how to sound like an open scientist, without actually doing open science. Enjoy! pic.twitter.com/K7TXb9UmHM

— Egon Willighⓐgen (@egonwillighagen) December 3, 2020

On a side note, I very much prefer the Mertonian norms shown above that address more fundamental principles intimately related to the principles of open research.

Open science/research can mean different things to different people, in particular when declined along its many technical, administrative, legal and philosophical attributes.

An important word above is excludes: as thriving open researchers, we need to understand that it isn’t only the distance towards better/open research that we have travelled that is relevant, but that the starting point matters a lot. The way somebody practices open research, whether that person has the possibility to implement this or that open (and reproducible) research practice, or whether they can be vocal about it mostly depends on their environment and the support or push back they get.

Embrace open and reproducible research to the extend you want and you can. Seek allies and support around you, but do not feel pressured. It isn’t open or close. It is certainly not the same open or close for everybody.

So my very first take-home messages are:

Open isn’t binary, it’s a gradient, it’s multidisciplinary, it’s multidimensional.

How to be an open scientist:

Let’s be open and understanding of different situations and constraints, including our own.

Perverse side-effect: open research as a business

Everything that is labelled as open research isn’t open of good. An easy example is Open Access Open Acce$$, a very lucrative business model set up by commerical publishers… and supported, directly or indirectly by most researchers.

• Green open access: post your pre-print (on arXiv, bioRXiv, …), get a DOI and credit. :-)

• Gold open access: pay a lot of money (APC article processing charge), typically 3000 - 9500 Euros (which many will be ready to pay!) to publish your article in a journal under an open access licence. For-profit publishing (typically huge profits). :-(

• Platinum (diamond) open access: 100% free to pubish and free to read, usually financed by university, funders, research organisations centrally. Non-profit publishing. :-)

• Read and publish ‘transformative’ agreements: combining subscriptions (to read) and APCs into big deals, this (1) obfuscating real costs, (2) lock-in into a contract with specific publishers, and (3) discriminate against institutions/countries that can’t afford them (see this document by Corina Logan and Dieter Lukas for more details). :-(

Open Science in Belgium (openaccess.be and @openaccess_be on twitter) has a complete glossary and general information.

Open research and open research

There is

Open Science as in widely disseminated and openly accessible

and

Open Science as in inclusive and welcoming

It was a damned hard community to break into. Any step I took to be more open, I felt attacked for not doing enough/doing it right.

— Christie Bahlai (@cbahlai) June 4, 2017

As far as I was concerned for a long time (until June 2017 to be accurate - this section is based this Open science and open science post), the former more technical definition was always what I was focusing on, and the second community-level aspect of openness was, somehow, implicit from the former, but that’s clearly not the case.

Even if there are efforts to promote diversity, under-represented minorities (URM) don’t necessarily feel included. When it comes to open science/research URMs can be further discriminated against by greater exposure or, can’t always afford to be vocal.

• Not everybody has the privilege to be open.
• There are different levels in how open one wants, or how open one could afford to be.
• Every voice and support is welcome.

Given the very broad views and opinions about what open research is, or is supposed to be, I think we can agree with Cameron Neylon:

The primary value proposition of #openscience is that diverse contributions allow better critique, refinement, and application 3/n

— CⓐmeronNeylon (@CameronNeylon) August 10, 2017

Ethical publishing

Corina Logan (@LoganCorina on twitter) started to talk already many years ago about ethical publishing (for research dissemination at large). I wasn’t immediately sold on the phrasing, but I now came to the realisation that it’s absolutely spot on.

We should think about the ethical implications of how we perform and disseminate our research.

(Thank you Corina).

Why becoming an open research practitioner

It’s the right thing to do. See the The Mertonian norms… Or is it?

Benefits for your academic career: How open science helps researchers succeed and more examples from the Open as a career boost paragraph:

• Open access articles get more citations.
• Open publications get more media coverage.
• Data availability is associated with citation benefit.
• Openly available software more likely to be used. (I don’t have any reference for this, and there are of course many couter examples).
• Benefit from institutional support of open research practices

Networking opportunities (I gave a similar talk in May 2019 thanks to my open research activities with my former colleague Marta Teperek at the University of Cambridge, UK).

See also Why Open Research

• Increase your visibility: Build a name for yourself. Share your work and make it more visible.

• Reduce publishing costs: Open publishing can cost the same or less than traditional publishing.

• Take back control: Know your rights. Keep your rights. Decide how your work is used

• Publish where you want: Publish in the journal of your choice and archive an open copy. (See The cost of knowledge boycott of Elsevier).

• Get more funding: Meet funder requirements, and qualify for special funds such as the Wellcome Trust Open Research Fund.

• Get that promotion: Open research is increasingly recognised in promotion and tenure. See also Reproducibility and open science are starting to matter in tenure and promotion July 14th, 2017, Brian Nosek) and the EU’s Evaluation of Research Careers fully acknowledging Open Science Practice defines an Open Science Career Assessment Matrix (OS-CAM):

But are there any risks?

Does it take more time to work openly?

Isn’t it worth investing time is managing data in a way that others (including future self) can find and understand it? That’s, IMHO, particularly important from a group leader’s perspective, where I want to build a corpus of data/software/research that other lab members can find, mine and re-use.

Are senior academics always supportive?

No. (not necessarily)

Is there a risk of being scooped?

There certainly is a benefit if releasing one’s research early!

But, importantly, working with open and reproducible research in mind doesn’t mean releasing everything prematurely, it means

• managing research in a way one can find data and results at every stage

• one can reproduce/repeat results, re-run/compare them with new data or different methods/parameters, and

• one can release data (or parts thereof) when/if appropriate.

So, are there any risks?

The Bullied Into Bad Science campaign is an initiative by early career researchers (ECRs) for early career researchers who aim for a fairer, more open and ethical research and publication environment.

Why isn’t it all open?

Or, as Mick Watson (2015) puts it:

Abstract Open science describes the practice of carrying out scientific research in a completely transparent manner, and making the results of that research available to everyone. Isn’t that just ‘science’?

There are aggravating circumstances:

The natural selection of bad science (Smaldino and McElreath, 2016)

Abstract Poor research design and data analysis encourage false-positive findings. Such poor methods persist despite perennial calls for improvement, suggesting that they result from something more than just misunderstanding. The persistence of poor methods results partly from incentives that favour them, leading to the natural selection of bad science. This dynamic requires no conscious strategizing—no deliberate cheating nor loafing—by scientists, only that publication is a principal factor for career advancement. Some normative methods of analysis have almost certainly been selected to further publication instead of discovery. In order to improve the culture of science, a shift must be made away from correcting misunderstandings and towards rewarding understanding. We support this argument with empirical evidence and computational modelling. We first present a 60-year meta-analysis of statistical power in the behavioural sciences and show that power has not improved despite repeated demonstrations of the necessity of increasing power. To demonstrate the logical consequences of structural incentives, we then present a dynamic model of scientific communities in which competing laboratories investigate novel or previously published hypotheses using culturally transmitted research methods. As in the real world, successful labs produce more ‘progeny,’ such that their methods are more often copied and their students are more likely to start labs of their own. Selection for high output leads to poorer methods and increasingly high false discovery rates. We additionally show that replication slows but does not stop the process of methodological deterioration. Improving the quality of research requires change at the institutional level.

Incentives for open career progression aren’t there (yet?), or not fully implemented. On the contrary…

If research is the by-product of researchers getting promoted (a quote by David Barron, Professor of Computer Science, Prof. Leslie Carr, personal communication), then shouldn’t we, early career researchers (ECRs), focus on promotion and being docile academic citizens rather than aiming for the more noble cause of pursuing research to understand the world that surrounds us, and disseminate our findings using modern channels?

But see the Declaration on Research Assessment (DORA) to improve the ways in which the outputs of scholarly research are evaluated.

Their objectives are (taken from their web page):

• Raise awareness: To call attention to new tools and processes in research assessment and the responsible use of metrics that align with core academic values and promote consistency and transparency in decision-making

• Facilitate implementation: To aid development of new policies and practices for hiring, promotion, and funding decisions

• Catalyze change: To spread research assessment reform broadly by working across scholarly disciplines and globally

• Improve equity: To call for broader representation of researchers in the design of research assessment practices that directly address the structural inequalities in academia

Here’s also a recent news about Five better ways to assess science about the Hong Kong Principles seek to replace “publish or perish” culture:

The practice of valuing quantity above quality in research needs to change if trust in science is to be maintained.

• Assess responsible research practices
• Value complete reporting
• Reward the practice of open science (open research)
• Acknowledge a broad range of research activities
• Recognise essential other tasks like peer review and mentoring

which fit with the need for more open, transparent and reproducible (trustworthy) research its dissemination.

In my opinion, barriers are not technological, but rather socio-cultural and political.

• Systemic control and inertia
• Vested interests by people in charge
• Abuse of power dynamics
• Fear of being scooped (an editorial in PLoS Biology on The importance of being second and, and how they prefer to focus on complementary research, recognising its important role in reproducibility of science.)
• Fear of not being credited
• Fear of errors and public humiliation, risk for reputation
• Fear of information overload
• …
• Fear of becoming less competitive in a over-competitive market!

Many if not all of these fears are only perceived risks.

Why reproducibility is important

• For scientific reasons: think reproducibility crisis.

• For political reasons: public trust in science, in data, in experts; without (public) trust in science and research, there won’t be any funding anymore. Lack of (public) trust in science leads to poor public health decisions!

But what do we mean by reproducibility?

From a But what to we mean by reproducibility? blog post.

• Repeat my experiment, i.e. obtain the same tables/graphs/results using the same setup (data, software, …) in the same lab or on the same computer. That’s basically re-running one of my analysis some time after I original developed it.

• Reproduce an experiment (not mine), i.e. obtain the same tables/graphs/results in a different lab or on a different computer, using the same setup (the data would be downloaded from a public repository and the same software, but possibly different version, different OS, is used). I suppose, we should differentiate replication using a fresh install and a virtual machine or docker image that replicates the original setup.

• Replicate an experiment, i.e. obtain the same (similar enough) tables/graphs/results in a different set up. The data could still be downloaded from the public repository, or possibly re-generate/re-simulate it, and the analysis would be re-implemented based on the original description. This requires openness, and one would clearly not be allowed the use a black box approach (VM, docker image) or just re-running a script.

• Finally, re-use the information/knowledge from one experiment to run a different experiment with the aim to confirm results from scratch.

Another view (from a talk by Kirstie Whitaker):

See also this opinion piece by Jeffrey T. Leek and Roger D. Peng, Reproducible research can still be wrong: Adopting a prevention approach.

Why reproducibility is important (as an individual researcher)

From

Gabriel Becker An Imperfect Guide to Imperfect Reproducibility, May Institute for Computational Proteomics, 2019.

(Computational) Reproducibility Is Not The Point

Take home message:

The goal is trust, verification and guarantees:

• Trust in Reporting - result is accurately reported
• Trust in Implementation - analysis code successfully implements chosen methods
• Statistical Trust - data and methods are (still) appropriate
• Scientific Trust - result convincingly supports claim(s) about underlying systems or truths

Reproducibility As A Trust Scale (copyright Genentech Inc)

Take home message:

Reproducibility isn’t binary, it’s a gradient, it’s multidisciplinary, it’s multidimensional.

Another take home message:

Reproducibility isn’t easy.

More reasons to become a reproducible research practitioner

Florian Markowetz, Five selfish reasons to work reproducibly, Genome Biology 2015, 16:274.

And so, my fellow scientists: ask not what you can do for reproducibility; ask what reproducibility can do for you! Here, I present five reasons why working reproducibly pays off in the long run and is in the self-interest of every ambitious, career-oriented scientist.

1. Reproducibility helps to avoid disaster: a project is more than a beautiful result. You need to record in detail how you got there. Starting to work reproducibly early on will save you time later. I had cases where a collaborator told me they preferred the results on the very first plots they received, that I couldn’t recover a couple of month later. But because my work was reproducible and I had tracked it over time (using git and GitHub), I was able, after a little bit of data analysis forensics, to identify why these first, preliminary plots weren’t consistent with later results (and it as a simple, but very relevant bug in the code). Imagine if my collaborators had just used these first plots for publication, or to decide to perform further experiments.
2. Reproducibility makes it easier to write papers: Transparency in your analysis makes writing papers much easier. In dynamic documents (using rmarkdown, juypter notebook and other similar tools), all results are automatically update when the data are changed. You can be confident your numbers, figures and tables are up-to-date.
3. Reproducibility helps reviewers see it your way: a reproducible document will tick many of the boxes enumerated above. You will make me very happy reviewer if I can review a paper that is reproducible.
4. Reproducibility enables continuity of your work: quoting Florian, “In my own group, I don’t even discuss results with students if they are not documented well. No proof of reproducibility, no result!”.
5. Reproducibility helps to build your reputation: publishing reproducible research will build you the reputation of being an honest and careful researcher. In addition, should there ever be a problem with a paper, a reproducible analysis will allow to track the error and show that you reported everything in good faith.

And career perspectives: Faculty promotion must assess reproducibility.

Does it take more time to work reproducibly?

No, it is a matter or relocating time!

From Five things about open and reproducible science that every early career researcher should know.

What can you do to improve trust in (your) research?

1. Be an open research practitioners
2. Be an reproducible research practitioners

Includes (but not limited to)

Preprints are the best!

Read, post, review and cite preprints (see ASAPbio for lots of resources about preprints).

Promoting open research through peer review

This section is based on my The role of peer-reviewers in checking supporting information promoting open science talk.

As an open researcher, I think it is important to apply and promote the importance of data and good data management on a day-to-day basis (see for example Marta Teperek’s 2017 Data Management: Why would I bother? slides), but also to express this ethic in our academic capacity, such as peer review. My responsibility as a reviewer is to

• Accept sound/valid research and provide constructive comments

and hence

• Focus on the validity of the research by inspecting the data, software and method. If the methods and/or data fail, the rest is meaningless.

I don’t see novelty, relevance, news-worthiness as my business as a reviewer. These factors are not the prime qualities of thorough research, but rather characteristics of flashy news.

Here are some aspects that are easy enough to check, and go a long way to verify the availability and validity and of the data

1. Availability: Are the data/software/methods accessible and understandable in a way that would allow an informed researcher in the same or close field to reproduce and/or verify the results underlying the claims? Note that this doesn’t mean that as a reviewer, I will necessarily try to repeat the whole analysis (that would be too time consuming indeed). But, conversely, a submission without data/software will be reviewed (and rejected, or more appropriately send back for completion) in matters of minutes. Are the data available in a public repository that guarantees that it will remain accessible, such as a subject-specific or, if none is available, a generic repository (such as zenodo or figshare, …), an institutional repository, or, but less desirable, supplementary information or a personal webpage¹.

2. Meta-data: It’s of course not enough to provide a wild dump of the data/software/…, but these need to be appropriately documented. Personally, I recommend an README file in every top project directory to summarise the project, the data, …

3. Do numbers match?: The first thing when reproducing someone’s analysis is to match the data files to the experimental design. That is one of the first things I check when reviewing a paper. For example if the experimental design says there are 2 groups, each with 3 replicates, I expect to find 6 (or a multiple thereof) data files or data columns in the data matrix. Along these lines, I also look at the file names (of column names in the data matrix) for a consistent naming convention, that allows to match the files (columns) to the experimental groups and replicates.

4. What data, what format: Is the data readable with standard and open/free software? Are the raw and processed available, and have the authors described how to get from one to the other?

5. License: Is the data shared in a way that allows users to re-use it. Under what conditions? Is the research output shared under a valid license?

Make sure that the data adhere to the FAIR principles:

Findable and Accessible and Interoperable and Reusable

Note that SI are not FAIR, not discoverable, not structured, voluntary, used to bury stuff. A personal web page is likely to disappear in the near future.

As a quick note, my ideal review system would be one where

1. Submit your data to a repository, where it gets checked (by specialists, data scientists, data curators) for quality, annotation, meta-data.

2. Submit your research with a link to the peer reviewed data. First review the intro and methods, then only the results (to avoid positive results bias).

When talking about open research and peer review, one logical extension is open peer review.

While I personally value open peer review and practice it when possible, it can be a difficult issue for ECRs, exposing them unnecessarily when reviewing work from prominent peers. It also can reinforce an already unwelcoming environment for underrepresented minorities. See more about this in the Inclusivity: open science and open science section below.

Publish, then review

eLife has very recently announced that they are implementing a publish, then review model of publishing:

From July 2021 eLife will only review manuscripts already published as preprints, and will focus its editorial process on producing public reviews to be posted alongside the preprints.

Registered reports

Define you data collection and analysis protocol in advance. Get it reviewed and, if accepted, get right to publish once data have been collected and analysed, irrespective of the (positive or negative) result.

• Three challenges: Restrictions on flexibility (no p-hacking of HARKing), The time cost, Incentive structure isn’t in place yet.

• Three benefits: Greater faith in research (no p-hacking of HARKing), New helpful systems (see technical solutions below), Investment in your future.

See https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000246 (2019).

Make allies

This is very important!

• Other ECR
• Librarians
• Data stewards/champions
• Research Software engineers
• On/off-line networking

Open research can lead to collaborative research. The development of the R for Mass Spectrometry initiative and MSnbase is an example I am very proud of.

The credit goes to the outstanding contributions and contributors!

— Laurent Gⓐtt⓪ (@lgatt0) May 6, 2019

Collaborative work and cooperation is certainly one important concept that gravitates around open science/research (see the Mertonian norm of communism), but that isn’t necessary nor sufficient for open science.

Final food for thought

Science as Amateur Software Development, by Richard McElreath.

Science is one of humanity’s greatest inventions. Academia, on the other hand, is not. It is remarkable how successful science has been, given the often chaotic habits of scientists. In contrast to other fields, like say landscaping or software engineering, science as a profession is largely unprofessional—apprentice scientists are taught less about how to work responsibly than about how to earn promotions. This results in ubiquitous and costly errors. Software development has become indispensable to scientific work. I want to playfully ask how it can become even more useful by transferring some aspects of its professionalism, the day-to-day tracking and back-tracking and testing that is especially part of distributed, open-source software development. Science, after all, aspires to be distributed, open-source knowledge development.

Building a brand as an open early career researcher – see Building a brand as a scientist by Stephanie Hicks.

Making discoveries and contributing your ideas and/or work are fundamental components of being a scientist (I am treating of the word “scientist” very broadly here). However, another important component of being a scientist is learning how to build your “brand” as a scientist.

Conclusions

Just do it! (if you are in a position to)

Build openness and reproducibility at the core your research

(according to you possibilities)

Don’t wait

Open and reproducible research doesn’t work if it’s an afterthought.

Standing on the shoulders of giants only really makes sense in the context of open and reproducible research.

• If you are here (or have read this), chances are you are on the path towards open and reproducible research.

• You are the architect of the kind of research and researcher you want to become. I hope these include openness and reproducibility.

• It’s a long path, that constantly evolves, depending on constraints, aspirations, environment, …

• The sky is the limit, be creative: work out the (open and reproducible) research that works for you now …

• … and that you want to work (for you and others) in the future.

Acknowledgements

One of my advice was to make allies. I have been lucky to meet wonderful allies and inspiring friends along the path towards open and reproducible research that works for me. Among these, I would like to highlight Corina Logan, Stephen Eglen, Marta Teperek, Kirstie Whitaker, Chris Hartgenink, Naomie Penfold, Yvonne Nobis.