Sunday, August 28, 2016

Should academics offer a money-back guarantee for research results to improve data reproducibility?


Question continued: As suggested in http://stm.sciencemag.org/content/8/336/336ed5.full.pdf+html

Michael Rosenblatt's prescription* that scientists should return money back to private investors if their data's not reproducible reads like a bad solution in search of a problem. Bad solutions emerge when we mis-diagnose problems and examine issues through a distorting lens. Amplifying their role in a systemic problem not entirely of their making, Rosenblatt's prescription assumes academics knowingly generate irreproducible data, and don't change their ways because no one is bringing them to account, ergo private investors need to ride in to the rescue and bring these wayward academics to heel. That's an astoundingly undeserved and incendiary supposition with no evidence whatsoever to back it up. If rank and file academics were really knowingly operating this way, they'd be using an approach lacking any semblance to the Scientific method. Were that the case, reform's futile anyway since they're all bad eggs who need to be summarily dismissed to set the system up from scratch with newcomers. Another weakness of this diagnosis is it assumes academics operate in a vacuum, endowed with absolute potentate-like powers to decide what and how they study. In other words, it compartmentalizes a systemic problem. After all, academia-industry collaboration is a small piece of current biomedical research enterprise, a piece that's likely impossible to influence piecemeal anyway, given how intertwined these various pieces are.

Data Irreproducibility Stems From Undeniably Perverse Incentives In The Academic Enterprise
Perverse incentives start right from a would-be academic's apprenticeship. Perhaps one of the most consequential is the pressure to publish, Publish or perish, because it sets up a positive feedback loop that reinforces what and how an academic studies through the course of their career. Examining what gets published helps understand some of what sustains academia's perverse incentives. After all, to be and stay an academic, one has to publish. Publications determine whether one a) even becomes an academic in the first place, b) gets tenure, c) succeeds in getting grants to fund one's academic work.
However, what gets published is also a consequence of what gets studied. Academic writes a grant proposal about what they'd like to study, a grant committee reviews it and decides to either fund it or not. In the academic culture that developed since WW II, what emerged as a grant winner in terms of what gets studied? Novelty, the thread that runs through the current academic pipeline. From the grant proposal to the peer-reviewed paper, at every node, when a stakeholder with the power decides to okay or not a project, novelty is one of the most important considerations.

Stakeholders are what I call the triumvirate of academic life, employers, grant givers, academic journals. Employers are typically academic institutions and universities, and departments therein. Grant givers are typically government agencies, foundations, trusts and, in the the case of the biomedical research enterprise, the for-profit partners, biopharma. Academic journals, many of them products of large, for-profit publishing houses, are the conduits. Their editorial boards parcel out the manuscripts to academics who peer-review them for free. These three determine the A-to-Z of an academic's career trajectory, and each, in the decades post-WW II, prioritized novelty.

In this ecosystem, reproducibility exists within the extremely narrow and tenuous purview of internal replication, i.e., that the academic themselves repeat their study observations a certain number of times. As this system rooted and fine-tuned itself, its strict mandate truncated scope. Meantime, academic competition intensified as universities continued to churn out more and more PhDs while faculty positions remained stagnant, a supply-demand problem only exacerbated in the US by the abolition of mandatory retirement in 1994. As a result, the pressure to publish within shorter and shorter time frames intensified. No surprise, output evolved towards an oxymoron, risk-averse as well as incrementally novel, the only kind sustainable within such a system. As well, intensified academic competition encourages opacity, discourages sharing.

Nowhere does this system reward or even encourage practitioners to expend effort, resources and time to replicate each other's output. Imagine an National Institutes of Health R01 grant review committee that receives an academic's grant proposal to attempt to reproduce a body of work in a sub-field. What are the chances it would get funded? Sorry, I rolled off my chair and was keeled over, doubled up in laughter. Let me catch my breath first. So steeped is the culture in novelty pursuit and has been for decades that reproducibility is a non-starter in what gets funded. That's a structural problem right there.

Thus, academics are merely responding to perverse incentives in the system they find themselves in, a system they didn't set up though they certainly sustain the status quo by unquestioningly operating to its dictates.

Academia's Systemic Data Irreproducibility Problems Can Only Be Solved Through Systemic Changes
If they’re serious about data reproducibility, each of the three key basic biomedical research stakeholders, employers, grant givers and academic journals, need to reward reproducibility efforts. However, this alone is insufficient. An essential lure of research for many academics, especially in science, is to be the first to uncover novelty. Reproducibility cannot be demanded like water from a tap from rank and file. Instead, rather than relentless focus on novelty, at least stakeholders could initiate change by expanding their purview to reproducible novelty, which would likely engender more serious academic engagement.
  • Employers could reward academics who choose to perform reproducibility studies, reward being anything from tenure to extra space and funding for labs, staff and/or research animals and their care facilities.
  • Grant givers could offer more than mere lip service in support of reproducibility by funding it.
    • In the biomedical research enterprise, likely no one at present comes close to the clout of the US National Institutes of Health. After all, so much of the US output in basic biomedical research is NIH funded.
    • Many are likely unaware that NIH also funds its own biomedical research, to the tune of a good 10% or so of its funds. When one considers its overall budget of ~US $30 billion, that's a really serious amount of money, sustaining the careers and labs of some ~1400 Principal Investigators and their staff.
    • What was the original mandate of this in-house research? Post-WW II, Vannevar Bush published his hugely influential vision for today's scientific enterprise, Science, The Endless Frontier. This guide informed the process by which NIH became the behemoth it currently is. The concern then was that high-risk, long-term, off-the-wall ideas wouldn't get explored by inherently competitive, high stakes academia, that the government needed to directly fund and nurture such science. That was the original mandate for the NIH Intramural Research Program.
    • ~Fifty plus years since it blossomed to full bloom, does its output match its mandate? Not at all. Rather, its output largely adheres to the same narrow risk-averse, incremental novelty that dominates the rest of academia. Clearly a case of costly redundancy.
    • Why not divert some of this expenditure and staff to reproducibility instead, when that's clearly the crying need of the hour? And it could even be reproducibility focused on the piece Rosenblatt argues is the most crucial in biomedical science, Translational research.
    • Who in the world could be better equipped to study translational research reproducibility than the NIH Intramural Research Program, with its enormous capacity for not just preclinical but also clinical research? After all, it has a truly giddying array of animal facilities that maintain everything from mice and rats to pigs, sheep and non-human primates, not to mention it has the depth and breadth of knowledgeable staff necessary to research them, while Wikipedia claims the National Institutes of Health Clinical Center has '240 inpatient beds, 11 operating rooms, 82 day hospital stations, critical care services and research labs, an ambulatory care research facility and a complex array of imaging services' right in the heart of its enormous campus.
    • Other countries should consider similar use of state research institutes in data reproducibility efforts, specifically translational research reproducibility.
  • Academic journals. How often do the world's premier multidisciplinary scientific journals, Nature (journal) or Science (journal) publish prominent data reproducibility studies? Rarely. How about discipline-specific staples like Journal of Biological Chemistry or Journal of Immunology, to mention just a couple. Rarely again. And what else could it be when reproducibility is simply not yet a priority for journals? It isn't now and wasn't earlier. After all, what's changed since the File Drawer problem (Publication bias) was first highlighted all the way back in 1979? Negative data continue to remain unpublished. Meantime, how realistic is the expectation, when the status quo dictates that their careers depend on publish or perish, that academics will leap off the springboard into the as-yet unrewarded realm of reproducibility studies, if journals don't even bother publishing them in the first place?

Further Reading:
7. Topol, Eric J. "Money back guarantees for non-reproducible results?." BMJ 353 (2016): i2770.
8. Smaldino, Paul E., and Richard McElreath. "The Natural Selection of Bad Science." arXiv preprint arXiv:1605.09511 (2016). http://arxiv.org/pdf/1605.09511.pdf


https://www.quora.com/Should-academics-offer-a-money-back-guarantee-for-research-results-to-improve-data-reproducibility/answer/Tirumalai-Kamala


Sunday, August 21, 2016

What do universities, journals, and government need to do to stimulate breakthrough scientific discovery? Tirumalai Kamala


Short answer: Solutions start from accurately identifying the problem. Fact is biomedical research has plenty of innovation 'on paper'. The feeling it isn't enough arises from a bottleneck because most breakthroughs aren't reproducible and hence fail to translate to the clinic. Once we accept the problem is lack of reproducibility rather than breakthroughs per se, we can more accurately envision remedies.

With top weekly science journals like Nature and Science expanding their specialist journal base at breakneck pace, weekly issues filled with breakthrough scientific discoveries, clearly the problem isn't insufficiency. Rather there's too much done the wrong way. Who sifts gold from dross to determine which breakthrough's reliable? So far no one. That leaves data irreproducibility to unfold slowly years down the road, like a disaster movie in slow motion.

Publications the lifeblood for a successful academic career, how often do journals publish papers with negative data asserting breakthroughs weren't reproducible? Question's rhetorical because the answer is almost never. Why would they when scientists submitting the papers and those reviewing them pro bono, both are fully invested in the novelty model which rewards their careers? Is it reasonable journals all the way from Nature/Science to specialist journals uniformly impose the mandate of novelty? In fact, all three, universities, journals and government operate a novelty-mandating ecosystem that can only change if they give reproducibility some parity.

No doubt such a preposterous idea could invite howls of derision. After all, with generations of scientists trained to go for guts and glory at all costs, who in their right mind would devote their scientific career to re-doing others' work? Seems an intractable problem except it isn't really so. For far too many years at a stretch, universities have been graduating far more biomedical researchers than can be absorbed by academia and industry. Forced into one post-doc after another, a large mass of highly trained scientists is looking for meaningful work beyond the tedium and indignity of being merely trained hands. The worthwhile goal of helping clean-up biomedical research might be just their ticket. With a sprinkling of high-profile reproducibility studies, Nature, Science and their ilk could continue to focus on novelty, providing the fodder for reproducibility studies in specialist journals. This would create a mutually reinforcing virtuous cycle that fuels reproducible breakthroughs that successfully translate to the clinic. Frustrated perception of not enough breakthroughs would start to fade. Biomedical research would also gain back its reputation for probity.

Bonus idea: Perhaps uniquely so, in one stroke the US has the opportunity to make this happen another way as well. The NIH, the largest government funder of US biomedical research, splits its budget two ways, 90% for funding external research, so-called extramural and 10% for funding its own internal research. The so-called intramural is purely play-in-a-sandbox-for-life type of funding for thousands of full-time NIH intramural researchers. Set up to do the kind of esoteric, high-risk science it was thought tenured university researchers just wouldn't undertake, clearly decades later, the system doesn't work the way it was envisaged. Instead, it's more of the same. However, it represents  a net opportunity. Simply switch the NIH intramural mandate from breakthrough to reproducibility. The budget, workforce and infrastructure are all there. Secure lifetime funding attracted precisely those who value stable job security and generous benefits, exactly the workforce needed for diligent reproduction, for sifting gold from dross from among the breakneck-paced breakthroughs.


https://www.quora.com/What-do-universities-journals-and-government-need-to-do-to-stimulate-breakthrough-scientific-discovery/answer/Tirumalai-Kamala


Sunday, August 14, 2016

What should people in other cities do to protect themselves against water being poisoned like it is in Flint, Michigan?


The Safe Drinking Water Act (1), amended in 1996  (2) includes Section 114, i.e., specific consumer protection provisions that water suppliers are required to notify the public of contaminants and other dangers in their drinking water. Mary Tiemann, a specialist in Environmental Policy at the Congressional Research Service explains these provisions in simpler terms (see summary below from 3). 


Simply put, US drinking water customers have the right to know if their tap water's contaminated and it's the duty and responsibility of their water supplier to provide them this information as a matter of course. In particular, they have the right to demand and get these annual right-to-know/Consumer Confidence Reports (CCR). According to the EPA (4),
'A CCR is an annual water quality report delivered by community water systems to their customers. The CCR includes information on source water, the levels of detected contaminants, compliance with drinking water rules, and some educational language.
The reports are due to customers by July 1st of each year'.

Of course, all these safeguards were literally blown out of the water in Flint, Michigan. Looks like a city in receivership is literally beyond the pale of democracy, run by an unelected political appointee who’s wholly unaccountable to the public. So, first order of business would be to flee a city in receivership like a bat out of hell. Of course, this option's not available for the poor, who're screwed six ways till Sunday.

NRDC (Natural Resources Defense Council) Research Suggests Drinking Water Quality Varies Greatly From City To City
In 2003, the NRDC (Natural Resources Defense Council) published a peer-reviewed study of the drinking water systems in 19 US cities (see figure and table below from 5)
This 13 year old study found source water protection ranges from excellent in cities like Seattle to high marks in cities like Boston, San Francisco, Denver to threatened by runoff and industrial or sewage contamination in cities like Atlanta, Chicago, Detroit, Houston, Los Angeles, New Orleans, Newark, Philadelphia, Phoenix, San Diego, Washington, D.C (see below from 5). The NRDC recommends consumers help protect their drinking water by getting involved in community decision making about water resources, attending meetings of their local water supplier, check CCRs, and contact their supplier for details. Bottomline, according to the NRDC, residents need to know how their cities are getting their drinking water, specifically, that
  • Sources are protected from pollution
  • Pipes are sound and well-maintained
  • Modern treatment facilities are a must

In 2013, the American Society for Civil Engineers' Report Card for America's Infrastructure gave the US a D meaning poor in the drinking water category (6). The NRDC's investigation also suggests CCR data cannot be accepted at face value. Flint shows local and state governments can't be trusted. Neither can the EPA. Thus, consumers should probably exercise caution as a way of life. For e.g., use filters on their water taps, specifically filters that reduce major contaminants such as microbial cysts, metals like lead and mercury, industrial chemicals like carbon tetrachloride, herbicides and pesticides, and chlorination by-products such as trihalomethanes (TTHM). A more conservative approach would be to use filtered and boiled water for cooking and drinking, habits second nature for a person like me who grew up in a developing country. Of course, as sociologist Andrew Szasz reports in his book, Shopping Our Way To safety. How We Changed from Protecting the Environment to Protecting Ourselves, through the process of 'inverted quarantine', year on year Americans drink more bottled water anyway because they already believe tap water's 'contaminated with chemicals that can make us ill' (7). 

According to Szasz, from drinking one gallon of bottled water per person per year in 1975, Americans drank 26 gallon per person per year in 2005.

Things will only change for the better if some high-level politicians and bureaucrats resign or are jailed/fined or otherwise severely penalized for what they allowed to happen in Flint, Michigan. This would send a message to others in the water supply business from private business to local government to federal regulators including the EPA that they are going to be held accountable if they fail to provide safe, drinking water to their constituents. If something along these lines doesn't happen, no one in the US can count on their drinking water to be safe. If water suppliers in one place get away with supplying toxic muck, why wouldn’t others follow suit?

Bibliography
3. Safe Drinking Water Act (SDWA): A Summary of the Act and Its Major Requirements. Mary Tiemann, Specialist in Environmental Policy, February 5, 2014. http://fas.org/sgp/crs/misc/RL31...
5. What's on Tap?: Grading Drinking Water in US Cities. 2003. http://www.nrdc.org/water/drinki...
6. American Society for Civil Engineers, 2013. http://www.infrastructurereportc...
7. Szasz, Andrew. Shopping our way to safety: How we changed from protecting the environment to protecting ourselves. U of Minnesota Press, 2007.

Further reading:

https://www.quora.com/What-should-people-in-other-cities-do-to-protect-themselves-against-water-being-poisoned-like-it-is-in-Flint-Michigan/answer/Tirumalai-Kamala


Sunday, August 7, 2016

What are the differences and similarities between India and America when it comes to racism?


Assuming America means the US, in my experience, India and the US are depressingly similar when it comes to racism. Both have a long shameful history of deliberately disenfranchising specific groups of people, who are then socially shunned, economically decimated, and culturally sidelined, of course after the dominant groups appropriate whatever they like or want. As well, this history is much longer in India, the group in question the Dalit, formerly called Untouchables. In the US, it's blacks and Native Americans. Process is also depressingly similar, the marginalized are deliberately segregated geographically, economically, educationally. In both countries, widespread protest movements galvanized change with governments enacting laws attempting to empower and enfranchise, through a variety of Affirmative action programs. However, ensuing changes are often cosmetic and cultures stubbornly hold on to the disenfranchising status quo. After all, easier for the privileged to view their accomplishments as merit-based rather than feasting off the banquet of generational entitlement, easier to dismiss the deliberately, systematically downtrodden groups as deserving of their lowlier fates. Difference is India retains a more visible and persistent feudal structure that exacerbates caste and class divides.

My earliest memory of bigotry is from my own family. With India still stubbornly feudal, everyone worth their salt has servants, now euphemistically called domestic workers. Change in name, not dignity. We had a couple, wife for house work, husband the gardener. I was around 3 or 4 years old. Living with us, my widowed paternal grandmother, conservative, rigid and very casteist. She'd wash her own clothes and hang them out to dry in the back yard. That day, as she used her long bamboo stick to hang it out on the back yard clothes line, Katayan, the gardener, accidentally brushed against her washed saree while weeding a garden patch. I'll never forget the Tamil cry that burst out from her lips, 'Dushta, dushta', i.e. wicked, wicked. Striking him once on the shoulder with her stick, she then agitatedly set to pulling her saree off the clothesline and ran off with it to wash it again, his body's accidental brushing apparently enough to 'pollute' it.

Of course, this was decades back and India has come a long way from such decrepit and utterly repulsive ideas or so the elite would have us believe but nothing could be further from the truth. By many measurable metrics, access to education, literacy rates, per capita income, Dalits remain marginalized and disenfranchised, entirely by design and not by accident. Sure, the Indian media often touts tremendous individual Dalit success stories but the reality is similar to the US. After all, one Obama in the White House doesn't mean blacks are suddenly equal to whites in terms of access to opportunities, wealth and power, and protection under the law, does it? And, lacking even the fig leaf of an Obama, Native Americans are left out of the conversation altogether.

Years later, a conference room at the US NIH fills up as people stream in for a meeting or presentation. Walk in on the tail end of some desultory conversation about a topical news issue, obviously about blacks. Shockingly, a senior white researcher, a famous name in fact, says of course, it's well known that among blacks it's the stupid ones that got caught in the first place. I'd never heard of such a flagrantly odious and self-serving theory to explain black enslavement. To hear it at all was shocking, to hear it stated by someone so highly educated even more so. That it was said at all in that room only emphasizes how spaces deemed to represent power in US society continue to systematically exclude blacks. Of course, that room was filling up with white Americans and immigrants deemed deserving, no blacks.

My own experience of racism in the US was less overt, often expressed as persistent suspicion. For e.g., being asked for my ID when presenting a credit card at the check-out when those ahead of and behind me, all whites, weren't. Once is a quirk. Often? That's a pattern. Then there was that ominous entanglement with a white cop my very first year in the US. Having written about it elsewhere (1), the recent, steady drip-drip of news stories of blacks killed in police encounters underlines in no uncertain terms how lucky I was to not end up with the same ignominious fate, shot and killed during a police encounter.

Recently, India was roiled with the high-profile Suicide of Rohith Vemula, a Dalit scholar at the University of Hyderabad. Didn't make as much of a buzz on Quora. Inevitably, what I read  included commenters unabashedly engaging in character assassination of a young man who's no longer around to defend himself. After all, at a time when internet penetration is <30% in India, no surprise haves would dominate in their access to it. Meantime, protestors in India are rightfully calling it institutional murder. After all, Dalits who demand to be seen as peers are actually perceived as dangerous interlopers and treated as such, creating conditions not so different from those that provoked the recent protests at 2015 University of Missouri protests. Why did Rohith's suicide touch such a nerve? The poignance and eloquence of his suicide note (2). Even birth in penury to a laborer and a seamstress couldn't prevent his rare and marvelous writing talent from blossoming. As James Baldwin famously seared the American public's conscience in his 1963 interview (3) with Dr. Kenneth Clark on the Negro and the American Promise, so Rohith's forlorn words reverberate in India today, in a culture that pays lip-service and doesn't sincerely engage in righting egregious historical wrongs, whose entrenched elite lack the generosity and humanism necessary to restore dignity to the ranks of the generationally oppressed by reaching out, mentoring and nurturing their talent. 

Racism is much more stark and overt for the thousands of African students now streaming into India for higher studies. As a black American PhD Student at the Delhi School of Economics wrote in 2009 (4),
'Discrimination in Delhi surpasses the denial of courtesy. I have been denied visas, apartments, entrance to discos, attentiveness, kindness and the benefit of doubt. Further, the lack of neighbourliness exceeds what locals describe as normal for a capital already known for its coldness.
My partner is white and I am black, facts of which the Indian public reminds us daily. Bank associates have denied me chai, while falling over to please my white friend. Mall shop attendants have denied me attentiveness, while mobbing my partner. Who knows what else is more quietly denied?
"An African has come," a guard announced over the intercom as I showed up. Whites are afforded the luxury of their own names, but this careful attention to my presence was not new. ATM guards stand and salute my white friend, while one guard actually asked me why I had come to the bank machine as if I might have said that I was taking over his shift'.
Be it Caste Hindus in India or whites in the US, bigotry goes underground following laws ostensibly banning it. Enacting laws isn't the same as implementing them. Entrenched power doesn't give its hegemony away. It has to be wrested out of cold dead, usually male, hands. Meantime, institutionalized bigotry cleverly engages in tokenism to gain the cover of plausible deniability, something B. R. Ambedkar, Dalit architect of India's constitution foresaw (5). Obvious symbols of tokenism, when blacks and mixed race sporadically rotated through my NIH lab, systematic lack of structural support and thoughtful mentoring ensured their inevitable failure. Self-fulfilling prophecy in place, such tokenism also gestates the far more dangerous seed of resentment within the easily self-satisfied and less self-aware among the privileged, yes, the same ones quick to myopically complain about oppressive levels of political correctness in society.

A quote often attributed to Gandhi advises, 'First they ignore you, then they laugh at you, then they fight you and then you win'. Yet the process isn't so conveniently linear, tying itself in the pretty bow tie of victory at the end. All too often, initial victories are pyrrhic and the persistently disenfranchised need to cycle back through the process. So, be it India or the US, we're cycling back to the 3rd stage. In both cultures, pressures from such an abhorrent and unsustainable form of institutionalized apartheid are now again coming to the boiling point. Maybe a course correction, maybe some more laws, some more tokenism, ending with marginally more power to the Dalits and blacks. Dare we ever hope for better than this?

Foot-notes
2. Rohith Vemula's suicide letter published by the Times of India, Jan 19, 2016. Full text: Dalit scholar Rohith Vemula's suicide note - Times of India 
3. James Baldwin interviewed by Dr Kenneth Clark, 1963. WGBH American Experience
 . Bonus Video
 
4. Outlook India, Diepiriye Kuku, June 29, 2009. 'India Is Racist, And Happy About It' 


https://www.quora.com/What-are-the-differences-and-similarities-between-India-and-America-when-it-comes-to-racism/answer/Tirumalai-Kamala