Should academics offer a money-back guarantee for research results to improve data reproducibility?

Question continued: As suggested in http://stm.sciencemag.org/content/8/336/336ed5.full.pdf+html

Michael Rosenblatt's prescription* that scientists should return money back to private investors if their data's not reproducible reads like a bad solution in search of a problem. Bad solutions emerge when we mis-diagnose problems and examine issues through a distorting lens. Amplifying their role in a systemic problem not entirely of their making, Rosenblatt's prescription assumes academics knowingly generate irreproducible data, and don't change their ways because no one is bringing them to account, ergo private investors need to ride in to the rescue and bring these wayward academics to heel. That's an astoundingly undeserved and incendiary supposition with no evidence whatsoever to back it up. If rank and file academics were really knowingly operating this way, they'd be using an approach lacking any semblance to the Scientific method. Were that the case, reform's futile anyway since they're all bad eggs who need to be summarily dismissed to set the system up from scratch with newcomers. Another weakness of this diagnosis is it assumes academics operate in a vacuum, endowed with absolute potentate-like powers to decide what and how they study. In other words, it compartmentalizes a systemic problem. After all, academia-industry collaboration is a small piece of current biomedical research enterprise, a piece that's likely impossible to influence piecemeal anyway, given how intertwined these various pieces are.

Data Irreproducibility Stems From Undeniably Perverse Incentives In The Academic Enterprise

Perverse incentives start right from a would-be academic's apprenticeship. Perhaps one of the most consequential is the pressure to publish, Publish or perish, because it sets up a positive feedback loop that reinforces what and how an academic studies through the course of their career. Examining what gets published helps understand some of what sustains academia's perverse incentives. After all, to be and stay an academic, one has to publish. Publications determine whether one a) even becomes an academic in the first place, b) gets tenure, c) succeeds in getting grants to fund one's academic work.

However, what gets published is also a consequence of what gets studied. Academic writes a grant proposal about what they'd like to study, a grant committee reviews it and decides to either fund it or not. In the academic culture that developed since WW II, what emerged as a grant winner in terms of what gets studied? Novelty, the thread that runs through the current academic pipeline. From the grant proposal to the peer-reviewed paper, at every node, when a stakeholder with the power decides to okay or not a project, novelty is one of the most important considerations.

Stakeholders are what I call the triumvirate of academic life, employers, grant givers, academic journals. Employers are typically academic institutions and universities, and departments therein. Grant givers are typically government agencies, foundations, trusts and, in the the case of the biomedical research enterprise, the for-profit partners, biopharma. Academic journals, many of them products of large, for-profit publishing houses, are the conduits. Their editorial boards parcel out the manuscripts to academics who peer-review them for free. These three determine the A-to-Z of an academic's career trajectory, and each, in the decades post-WW II, prioritized novelty.

In this ecosystem, reproducibility exists within the extremely narrow and tenuous purview of internal replication, i.e., that the academic themselves repeat their study observations a certain number of times. As this system rooted and fine-tuned itself, its strict mandate truncated scope. Meantime, academic competition intensified as universities continued to churn out more and more PhDs while faculty positions remained stagnant, a supply-demand problem only exacerbated in the US by the abolition of mandatory retirement in 1994. As a result, the pressure to publish within shorter and shorter time frames intensified. No surprise, output evolved towards an oxymoron, risk-averse as well as incrementally novel, the only kind sustainable within such a system. As well, intensified academic competition encourages opacity, discourages sharing.

Nowhere does this system reward or even encourage practitioners to expend effort, resources and time to replicate each other's output. Imagine an National Institutes of Health R01 grant review committee that receives an academic's grant proposal to attempt to reproduce a body of work in a sub-field. What are the chances it would get funded? Sorry, I rolled off my chair and was keeled over, doubled up in laughter. Let me catch my breath first. So steeped is the culture in novelty pursuit and has been for decades that reproducibility is a non-starter in what gets funded. That's a structural problem right there.

Thus, academics are merely responding to perverse incentives in the system they find themselves in, a system they didn't set up though they certainly sustain the status quo by unquestioningly operating to its dictates.

Academia's Systemic Data Irreproducibility Problems Can Only Be Solved Through Systemic Changes

If they’re serious about data reproducibility, each of the three key basic biomedical research stakeholders, employers, grant givers and academic journals, need to reward reproducibility efforts. However, this alone is insufficient. An essential lure of research for many academics, especially in science, is to be the first to uncover novelty. Reproducibility cannot be demanded like water from a tap from rank and file. Instead, rather than relentless focus on novelty, at least stakeholders could initiate change by expanding their purview to reproducible novelty, which would likely engender more serious academic engagement.

• Employers could reward academics who choose to perform reproducibility studies, reward being anything from tenure to extra space and funding for labs, staff and/or research animals and their care facilities.
• Grant givers could offer more than mere lip service in support of reproducibility by funding it.
• In the biomedical research enterprise, likely no one at present comes close to the clout of the US National Institutes of Health. After all, so much of the US output in basic biomedical research is NIH funded.
• Many are likely unaware that NIH also funds its own biomedical research, to the tune of a good 10% or so of its funds. When one considers its overall budget of ~US $30 billion, that's a really serious amount of money, sustaining the careers and labs of some ~1400 Principal Investigators and their staff.
• What was the original mandate of this in-house research? Post-WW II, Vannevar Bush published his hugely influential vision for today's scientific enterprise, Science, The Endless Frontier. This guide informed the process by which NIH became the behemoth it currently is. The concern then was that high-risk, long-term, off-the-wall ideas wouldn't get explored by inherently competitive, high stakes academia, that the government needed to directly fund and nurture such science. That was the original mandate for the NIH Intramural Research Program.
• ~Fifty plus years since it blossomed to full bloom, does its output match its mandate? Not at all. Rather, its output largely adheres to the same narrow risk-averse, incremental novelty that dominates the rest of academia. Clearly a case of costly redundancy.
• Why not divert some of this expenditure and staff to reproducibility instead, when that's clearly the crying need of the hour? And it could even be reproducibility focused on the piece Rosenblatt argues is the most crucial in biomedical science, Translational research.
• Who in the world could be better equipped to study translational research reproducibility than the NIH Intramural Research Program, with its enormous capacity for not just preclinical but also clinical research? After all, it has a truly giddying array of animal facilities that maintain everything from mice and rats to pigs, sheep and non-human primates, not to mention it has the depth and breadth of knowledgeable staff necessary to research them, while Wikipedia claims the National Institutes of Health Clinical Center has '240 inpatient beds, 11 operating rooms, 82 day hospital stations, critical care services and research labs, an ambulatory care research facility and a complex array of imaging services' right in the heart of its enormous campus.
• Other countries should consider similar use of state research institutes in data reproducibility efforts, specifically translational research reproducibility.
• Academic journals. How often do the world's premier multidisciplinary scientific journals, Nature (journal) or Science (journal) publish prominent data reproducibility studies? Rarely. How about discipline-specific staples like Journal of Biological Chemistry or Journal of Immunology, to mention just a couple. Rarely again. And what else could it be when reproducibility is simply not yet a priority for journals? It isn't now and wasn't earlier. After all, what's changed since the File Drawer problem (Publication bias) was first highlighted all the way back in 1979? Negative data continue to remain unpublished. Meantime, how realistic is the expectation, when the status quo dictates that their careers depend on publish or perish, that academics will leap off the springboard into the as-yet unrewarded realm of reproducibility studies, if journals don't even bother publishing them in the first place?

*: An incentive-based approach for improving data reproducibility

Further Reading:

1. Tirumalai Kamala's answer to How can we create truly large bio repositories to aid medical research?

2. Tirumalai Kamala's answer to Does analysing and organising statistics count as research?

3. Tirumalai Kamala's answer to What consequences can bring a recent 2 billion dollars raise of NIH budget?

4. Tirumalai Kamala's answer to What do universities, journals, and government need to do to stimulate breakthrough scientific discovery?

5. Tirumalai Kamala's answer to How does a successful academic research group operate?

6. Tirumalai Kamala's answer to How can we redesign the PhD experience in order to minimize suffering of graduate students?

7. Topol, Eric J. "Money back guarantees for non-reproducible results?." BMJ 353 (2016): i2770.

8. Smaldino, Paul E., and Richard McElreath. "The Natural Selection of Bad Science." arXiv preprint arXiv:1605.09511 (2016). http://arxiv.org/pdf/1605.09511.pdf

https://www.quora.com/Should-academics-offer-a-money-back-guarantee-for-research-results-to-improve-data-reproducibility/answer/Tirumalai-Kamala

What do universities, journals, and government need to do to stimulate breakthrough scientific discovery? Tirumalai Kamala

Short answer: Solutions start from accurately identifying the problem. Fact is biomedical research has plenty of innovation 'on paper'. The feeling it isn't enough arises from a bottleneck because most breakthroughs aren't reproducible and hence fail to translate to the clinic. Once we accept the problem is lack of reproducibility rather than breakthroughs per se, we can more accurately envision remedies.

With top weekly science journals like Nature and Science expanding their specialist journal base at breakneck pace, weekly issues filled with breakthrough scientific discoveries, clearly the problem isn't insufficiency. Rather there's too much done the wrong way. Who sifts gold from dross to determine which breakthrough's reliable? So far no one. That leaves data irreproducibility to unfold slowly years down the road, like a disaster movie in slow motion.

Publications the lifeblood for a successful academic career, how often do journals publish papers with negative data asserting breakthroughs weren't reproducible? Question's rhetorical because the answer is almost never. Why would they when scientists submitting the papers and those reviewing them pro bono, both are fully invested in the novelty model which rewards their careers? Is it reasonable journals all the way from Nature/Science to specialist journals uniformly impose the mandate of novelty? In fact, all three, universities, journals and government operate a novelty-mandating ecosystem that can only change if they give reproducibility some parity.

No doubt such a preposterous idea could invite howls of derision. After all, with generations of scientists trained to go for guts and glory at all costs, who in their right mind would devote their scientific career to re-doing others' work? Seems an intractable problem except it isn't really so. For far too many years at a stretch, universities have been graduating far more biomedical researchers than can be absorbed by academia and industry. Forced into one post-doc after another, a large mass of highly trained scientists is looking for meaningful work beyond the tedium and indignity of being merely trained hands. The worthwhile goal of helping clean-up biomedical research might be just their ticket. With a sprinkling of high-profile reproducibility studies, Nature, Science and their ilk could continue to focus on novelty, providing the fodder for reproducibility studies in specialist journals. This would create a mutually reinforcing virtuous cycle that fuels reproducible breakthroughs that successfully translate to the clinic. Frustrated perception of not enough breakthroughs would start to fade. Biomedical research would also gain back its reputation for probity.

Bonus idea: Perhaps uniquely so, in one stroke the US has the opportunity to make this happen another way as well. The NIH, the largest government funder of US biomedical research, splits its budget two ways, 90% for funding external research, so-called extramural and 10% for funding its own internal research. The so-called intramural is purely play-in-a-sandbox-for-life type of funding for thousands of full-time NIH intramural researchers. Set up to do the kind of esoteric, high-risk science it was thought tenured university researchers just wouldn't undertake, clearly decades later, the system doesn't work the way it was envisaged. Instead, it's more of the same. However, it represents  a net opportunity. Simply switch the NIH intramural mandate from breakthrough to reproducibility. The budget, workforce and infrastructure are all there. Secure lifetime funding attracted precisely those who value stable job security and generous benefits, exactly the workforce needed for diligent reproduction, for sifting gold from dross from among the breakneck-paced breakthroughs.

https://www.quora.com/What-do-universities-journals-and-government-need-to-do-to-stimulate-breakthrough-scientific-discovery/answer/Tirumalai-Kamala

What consequences can bring a recent 2 billion dollars raise of NIH budget?

Pumping more money into the NIH without structural reform of the US biomedical research enterprise is throwing good money after bad, i.e., sheer wastefulness. Why?

• First, decades back, US biomedical research coalesced around a sweatshop structure for staffing labs.
• Second, abolition of mandatory retirement on Jan 1, 1994, means that established PIs (Principal Investigators) who joined in the 1960s and 70s aren't leaving and continue milking the system for what it's worth, at the expense of younger generations.
• Finally, 'the doubling' cemented this already unsustainable structure. 'The doubling' refers to the NIH annual budget increase of 15% for 5 consecutive years from 1998 to 2003, abruptly doubling it from $13 billion to >$27 billion over a short 5 year period. Countrywide, university labs expanded and even increased in number, and more PhDs began entering an already saturated job market. With the Great Recession kicking off in 2007, the economy just couldn't absorb the glut. Some managed to cling on as post-docs, or did multiple post-docs while others left the field altogether.

Sweatshop structure of US biomedical research labs

Biomedical research labs across the US are increasingly staffed by temporary workers, namely, poorly paid graduate students and post-doctoral fellows. In return for hands-on training in the tools of their future trade, such workers perform the nuts and bolts of US biomedical research. Training done, they move on into an already saturated job market hoping the coin toss works in their favor for a faculty position in an ever-shrinking pool, shrinking largely because increasing number of older faculty aren't retiring while US universities can't realistically expand faculty positions to absorb all the newly minted PhDs. Costs aren't in favor of doing that. Upshot is US biomedical research labs operate under conditions of constant labor turnover.

Meantime, sampling a teeming supply of ready labor that applies for biomedical PhDs from all over the world (see figure below in the middle from 1), not just the US, US universities have evolved an assembly-line approach to plug this temporary worker shortage by filling research labs with increasing numbers of PhD students and post-docs. Thus, the US has been graduating a glut of biomedical research PhDs, more than the US job market could possibly absorb.

Faculty positions are mainstays for biomedical PhDs but existing US life sciences faculty positions can't absorb them all so more and more freshly minted PhDs spend many more years in post-doctoral positions. A rarity in the 1950s and 60s, today a post-doc after a PhD is thus the norm in biomedical research (see figure below in the right from 1).

The glut of money that poured into the NIH during 'the doubling' only exacerbated this pre-existing problem, making its way into the university system who graduated ever more life science PhDs (see figure below on the left from 1) even as their ever-aging faculties hung on to their positions. Faculty expansions from this period only contributed to this problem since they were immediately followed by precipitous NIH funding declines from 2004 till date. As research funds evaporated, predictable hyper-competitiveness set in and PIs, especially less secure junior faculty, spend more of their time chasing fewer research dollars, writing and revising more grant proposals. Inevitable gap in training and mentoring slides off onto the hapless shoulders of post-docs while graduate students cover more of the undergraduate tutoring responsibilities. Underpaid, overworked labor thus undergirds the present day US biomedical research enterprise.

Government policy encourages aging of US biomedical research faculty

In 1986, the US congress passed the 1986 Age Discrimination Act. A special exemption in this Act allowed colleges and universities to enforce mandatory faculty retirement at age 70 until 1994. The US Congress allowed this exemption to expire and mandatory retirement for university faculty was abolished on January 1, 1994, just as a big chunk of faculty hired in the early 1960s approached traditional retirement. Thus, tenured US faculty have lifetime employment. Already back in 2001, Orley Ashenfelter and David Card's analysis (2) of 16000 older faculty at 104 colleges and universities across the US found

• Average retirement rates for 70 and 71 year old faculty fell from ~75 and 60%, respectively, to ~30%.
• At age 72, 70 year old faculty who continued working increased from <10% to ~50% once mandatory retirement was abolished.

More proof of aging leadership in US biomedical research enterprise? Most lucrative and consequential for biomedical research faculty, the NIH R01 grants are unmistakably aging (see figure below from 3 with numbers from 4).

• Where in 1983, 18% were awarded to < or =36 years of age, they accounted for only ~3% in 2010.
• OTOH, > or =66 years of age accounted for almost nothing in 1980 but accounted for ~7% in 2010.

Since these trends stayed unchanged during 'the doubling', giving NIH more money isn't going to change this status quo.

The US economy cannot absorb the glut in biomedical research PhDs its universities generate

If not university faculty, then at least jobs in industry should be able to absorb newly minted biomedical PhDs and post-docs, right? No, US pharma employment has stayed flat for at least 20 years (see figure below from 5).

The $2 billion increase is thus meaningless for two reasons,

• If it remains a one-off. Since 2016 is presidential election season, all bets are off on what the future portends.
• Inflationary losses since 2003, when NIH budgets flattened or reduced, means that this increase merely takes funding back to 2003 levels. This is because a dollar's worth of research in 2003 would have cost $1.44 in 2015 (see figure below from 6), according to Federation of American Societies for Experimental Biology (FASEB).

In sheer money terms, 2016 NIH budget needs to be ~$48 billion in 2015 dollars to recoup 2003 research strength (see figure below from 7).

Absent structural reforms, i.e., finding solutions to the politically incendiary issues of the aging of US faculty and the sweatshop construct of US basic biomedical research labs, increase in NIH funding is thus social harm, not social good.

Bibliography

1. 2014 National Science Foundation Science and Engineering Indicators. http://www.nsf.gov/statistics/se...

2. Ashenfelter, Orley, and David Card. Did the elimination of mandatory retirement affect faculty retirement flows?. No. w8378. National bureau of economic research, 2001. http://www.econstor.eu/bitstream...

3. Sally Rockey. Feb 13, 2012. Age Distribution of NIH Principal Investigators and Medical School Faculty. http://nexus.od.nih.gov/all/2012...

4. Alberts, Bruce, et al. "Opinion: Addressing systemic problems in the biomedical research enterprise." Proceedings of the National Academy of Sciences 112.7 (2015): 1912-1913. http://www.pnas.org/content/112/...

5. Leadership In Decline. Assessing US International Competitiveness In Biomedical Research. The Information Technology And Innovation Foundation And United For Medical Research. Robert D. Atkinson, Stephen J. Ezell, L. Val Giddings, Luke A.SStewart, Scott M. Andes. May 2012.  http://www.unitedformedicalresea... 

6. From http://www.faseb.org/Portals/2/P...

7. On the Cusp of the 2016 Election: Why Is Politics Avoiding Science? April 2, 2015. On the Cusp of the 2016 Election: Why Is Politics Avoiding Science?

Further Reading

1. Teitelbaum, Michael S. "Structural disequilibria in biomedical research." Science 321.5889 (2008): 644-645.

2. Stephan, Paula E. "The biomedical workforce in the US: An example of positive feedbacks." International Centre for Economic Research Working Paper 11 (2010). http://sites.gsu.edu/pstephan/fi...

3. How We're Unintentionally Defunding the National Institutes of Health. Pacific Standard Magazine, Michael White, Nov 27, 2013. http://www.psmag.com/health-and-...

4. Chakma, Justin, et al. "Asia's ascent—global trends in biomedical R&D expenditures." New England Journal of Medicine 370.1 (2014): 3-6.

5. Updated: Fountain of youth: A congressman's plan to make NIH grantees younger. Science, Jocelyn Kaiser, Oct 6, 2014. Updated: Fountain of youth: A congressman's plan to make NIH grantees younger

6. Alberts, Bruce, et al. "Rescuing US biomedical research from its systemic flaws." Proceedings of the National Academy of Sciences 111.16 (2014): 5773-5777. http://www.pnas.org/content/111/... 

7. Daniels, Ronald J. "A generation at risk: Young investigators and the future of the biomedical workforce." Proceedings of the National Academy of Sciences 112.2 (2015): 313-318. http://www.pnas.org/content/112/...

8. Pickett, Christopher L., et al. "Toward a sustainable biomedical research enterprise: Finding consensus and implementing recommendations." Proceedings of the National Academy of Sciences 112.35 (2015): 10832-10836. http://www.pnas.org/content/112/...

https://www.quora.com/What-consequences-can-bring-a-recent-2-billion-dollars-raise-of-NIH-budget/answer/Tirumalai-Kamala

In 2010-2011, David Koch was asked to leave the National Cancer Institute at NIH for obfuscating the role of formaldehyde in causing cancer. Which scientists did he rely on most for generating doubt?

The question is slightly inaccurate. David Koch sat on the advisory board of the National Cancer Institute (NCI), the largest (by size and budget) permanent institute at the NIH. The NCI has the mandate for identifying and researching human carcinogens.

In the August 30, 2010 issue of the New Yorker, the award-winning journalist Jane Mayer published an investigative piece on the Koch family where, among other things, she reported, 'Koch Industries has been lobbying to prevent the E.P.A. from classifying formaldehyde, which the company produces in great quantities, as a ‘known carcinogen’ in humans' (Covert Operations - The New Yorker).

As Jane Meyer reports further in her piece, 'Scientists have long known that formaldehyde causes cancer in rats, and several major scientific studies have concluded that formaldehyde causes cancer in human beings—including one published last year by the National Cancer Institute, on whose advisory board Koch sits. The study tracked twenty-five thousand patients for an average of forty years; subjects exposed to higher amounts of formaldehyde had significantly higher rates of leukemia. These results helped lead an expert panel within the National Institutes of Health to conclude that formaldehyde should be categorized as a known carcinogen, and be strictly controlled by the government. Corporations have resisted regulations on formaldehyde for decades, however, and Koch Industries has been a large funder of members of Congress who have stymied the E.P.A., requiring it to defer new regulations until more studies are completed'.

Also, 'James Huff, an associate director at the National Institute for Environmental Health Sciences, a division of the N.I.H., told me that it was “disgusting” for Koch to be serving on the National Cancer Advisory Board: “It’s just not good for public health. Vested interests should not be on the board.” He went on, “Those boards are very important. They’re very influential as to whether N.C.I. goes into formaldehyde or not. Billions of dollars are involved in formaldehyde'.

And, 'Harold Varmus, the director of the National Cancer Institute, knows David Koch from Memorial Sloan-Kettering, which he used to run. He said that, at Sloan-Kettering, “a lot of people who gave to us had large business interests. The one thing we wouldn’t tolerate in our board members is tobacco.” When told of Koch Industries’ stance on formaldehyde, Varmus said that he was “surprised'.

Predictably, this piece created a media and activist firestorm around the clear conflict of interest in Koch, whose Koch Industries is the owner of one of the largest manufacturers of formaldehyde, sitting on an advisory board  of such import on public health policy. It was after this piece appeared in print that David Koch left his advisory board position at the NCI, as the New York Times reported on October 27, 2010  (Koch Leaves Federal Cancer Panel as Groups Urge Ethics Probe).

And it was after this piece appeared in print that the US government announced on June 9, 2011, that it was adding formaldehyde to a list of known human carcinogens (U.S. (finally) Labels Formaldehyde "Known Human Carcinogen").

Read Jane Meyer's New Yorker piece to understand how lobbying relevant government agencies such as the EPA (Environmental Protection Agency) was the approach to prevent or delay classifying formaldehyde as a known carcinogen in humans, not relying on specific scientists to create doubt.

https://www.quora.com/In-2010-2011-David-Koch-was-asked-to-leave-the-National-Cancer-Institute-at-NIH-for-obfuscating-the-role-of-formaldehyde-in-causing-cancer-Which-scientists-did-he-rely-on-most-for-generating-doubt/answer/Tirumalai-Kamala