Obsessed with surrogates?

Posted on November 1, 2013 by carl

Not really something you should tell your partner, that you are obsessed with surrogates. But it seems to me, we should all be pretty anxious about the rise in the use of surrogate outcomes in clinical trials over the last decade. Indeed, if the pharmaceutical industry could, they would make them mandatory for all primary outcomes, now, and in the future.

It would be the end of hard outcomes; throw in a few patient reported outcomes as well, and there you go, a ready made regulatory approved treatment to go.

Therefore it was with interest that I read the article in the BMJ, by John Yudkin, Kasia Lipska, and Victor Montori, which argued that our obsession with surrogates is damaging patient care.

It was a relief to see there are other folk out there, implying that there is a widespread problem affecting healthcare worldwide – might even need to declare it a pandemic.

This is what they say: ‘overinterpretation of surrogates can lead to misinterpretation of the evidence,’ as well as posing the important question: ‘why have doctors become so invested in surrogate markers?’

If you are wondering what surrogates are, and to get you up to speed, then read Healthy Skepticism’s collection of misleading surrogate end points.

Anyone, though, might think the problems with surrogates are a new issue. Yet, in 2000 Ralph D’Agostino was debating ‘The slippery slope of surrogate outcomes’

At the time this is what he concluded:

‘Surrogate outcomes are frequently used in cardiovascular disease research. A concern is that changes in surrogate markers may not reflect changes in disease outcomes. Two recent clinical trials (Heart and Estrogen/Progestin Replacement Study [HERS], and the Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial [ALLHAT]) underscore this problem since their results contradicted what was expected based on the surrogate outcomes. The current regulatory policy to allow new therapies to be introduced onto the market based solely on surrogate outcomes may need to be reviewed.’

This year, Larry Huston reported, the “FDA is actually seeking to make it easier for manufacturers of one therapeutic category, antihypertensives, to claim cardiovascular benefit for their drugs despite the absence of evidence to support most of these drugs.”

And further to this, the FDA was heavily criticised this year, for its reliance on surrogate markers in the approval of a tuberculosis drug. Pharmalot reported earlier this year that ‘an essay in the Journal of the American Medical Association notes that the pivotal Sirturo trials were small and produced troubling data. Ten of 79 patients given the drug died, compared with two of 81 in the control group. And five of the 10 deaths were due to TB, indicating treatment failure. The trials measured the ability of the drug to convert a patient’s sputum culture from positive to negative.’

I can hear your thoughts: how do you exactly get to approval of Sirturo on sputum culture? Obviously, through over reliance on surrogates.

What has therefore changed in the last decade? Surrogates are used more, they are relied on more for regulatory approval, and less effective drugs are coming to the market.

Therefore let me ask you, who is it then, that is obsessed with surrogates?

Non publication of trial results: commoner than you think

Posted on October 30, 2013 by carl

This is what a BMJ publication, out today,states in its conclusions: ‘among this group of large clinical trials, non-publication of results was common and the availability of results in the ClinicalTrials.gov database was limited.’

I have highlighted the important bits in bold, as if it isn’t clear to anyone yet, and you may need to read it again, this is a huge problem: the non-publication of results is common and the results of clinical trials remain limited.

I have previously published, on this blog, that a high proportion of clinical trial data is never published, and the evidence backing up this point is is growing.

So what did the authors do this time ?

They looked at 585 registered trials, with a sample size larger than 500, of which 29% remained unpublished, after a median of 60months. This is what makes this study slightly different to the others that that have gone before. Smaller trials may have acounted for most of the problem, and by eliminating them we can readily see non-publication is just as bad for those trials with larger samples: those that are most likely to impact on healthcare.

In addtion, industry studies were nearly twice as likely not to be published 32% versus 18% for those rhat received no industry funding. And of the 171 unpublished trials, 78% had no available results in ClinicalTrials.gov.

I am sorry BMJ, but I need to reproduce this slide, as it says it all, and it is too important not to show the reality of FDA regulation; that states, it is mandatory to post summary results in one year (read the paper, it is open access)

At some point, it will become obvious, to those that make all the decisions, that without access to the full evidence-base, we will remain in the dark as to the true effects of any given intervention.

EBM papers to read: The concept of risk

Posted on October 25, 2013 by carl

Every course of action or inaction in medical care can be associated with both risks and benefits which are often constrained by gaps in medical knowledge. The idea of risk generally embodies at least two distinct concepts: first, an unwanted outcome, and second, some uncertainty about the occurrence of that outcome.

The term risk thus corresponds to two terms unwanted outcome and probability.

Bogardus’ work, published in JAMA, is a must read for those interested in talking about medical risk. It highlights the importance of understanding more than just the identity of the risk. Thus the concept of risk has several fundamental dimensions, and explicit consideration of these dimensions may facilitate more appropriate discussion.

Reproduced from JAMA. 1999;281(11):1037-1041. doi:10.1001/jama.281.11.1037.

Bogardus reports that, the fifth and perhaps most important dimension of risk is its value to the patient, its subjective “badness”. Thus the ultimate determination of the relative importance of risk to the patient is a subjective decision. The first four dimensions of risk identity, permanence, timing, and probability; may help patients determine a personal value for the risk.

Take a read, it’s a great EBM paper.

What you ought to know about PubMed Commons

Posted on October 23, 2013 by carl

For the past few weeks I have been testing out, as one of the beta testers, the new PubMed Commons, having been invited by @HildaBastian.

Have to say soon as I heard about PuBMed commons and used it, in fact as soon as I got the email I was intrigued. It was impossible not to think this is a significant moment in post peer review commentary.

You can see at the top right hand corner I now have my own log in for PubMed with my own NCBI account.

It is pretty simple to use, you need to be invited, as this will certainly cut down on the dross: similar to google mail, which effectively cuts down on spam. Therefore I wouldn’t be as worried as some about the lack of inclusiveness, I’d give this idea a thumbs up, as it will definitely make it much more useful if the comments remain relevant.

Then its simple, you see an abstract that interests you and you can hit the ‘add comment’ button, at the bottom of the article (see the picture below) , and away you go.

The cynics out there will say, ‘yeah but you didn’t even read the full article.’ This might be a short-coming for restricted access journals; however, my thoughts are that this will be addressed by the comments that follow. Where you can respond to the authors but also respond to the flow of other comments.

What should happen is an ongoing interactive peer review, that is easy accessible and readable with each abstract. As authors, it will be great to see the comments that build up around articles over time. Journals may want to take note of the comments that surround articles, and as readers it should help frame some of the important issues.

If you are interested the FAQ is here [FAQs at PubMed Commons.]

You can read some more on PubMed commons, by Hilda Bastian here

Over time I think this is going to be massive, and its that impressive I ‘m off to post another comment. If you know me, and like commenting, as well as testing out new initiatives, and you can find me, I will sign you up as part of the testing phase.

To find out more about PubMed Commons, click here. or Follow on Twitter: @PubMedCommons

Health Warning should come with modified intention to treat analysis

Posted on September 27, 2013 by carl

Internal validity of clinical trials is the extent to which you trust the methods that the results are based on. There are some simple methodological issues, such as the presence or absence of randomization, which allows you to understand whether the effect size is dramatically inflated or not due to differences in the baseline characteristics that can occur.

There are other important methods, also, that affect the results, such as blinding, allocation concealment and intention to treat. Now bear with me while I explain one of these that is often a confusing concept, the intention to treat analysis

In an intention to treat analysis the results are based on those initially assigned to the treatment and not just those who took the treatment (often referred to as per protocol analysis). The intention to treat analysis mainly leads to smaller effect sizes, often because the denominator is greater in the ITT group than the per protocol group (as people stop the treatment, sometimes they don’t take it as prescribed, they are lost to follow up and for a host of other reasons).

The key is that the folk who stopped taking the intervention may have gone onto have the adverse effect of interest. If you don’t analyze them, as in a per protocol analysis, you may be missing important adverse effects. This is exactly what happened in the Vioxx studies where per protocol analysis data presented to the FDA underestimated the true rates of heart attacks.

It is with surprize, and with some confusion that I have therefore seen the dramatic rise in the use of the ‘modified’ intention to treat analysis.

A 2010 systematic review in the BMJ analsysed modified intention to treat reporting in randomised controlled trials: at the time 475 RCTs used a modified intention to treat analysis. When the description was examined 40% reported one type of deviation from the intention to treat approach, 55% reported two or more types.

Is this blindingly obvious, or am I missing something: modified intention to treat is just a play on words, and in any other language it is per protocol analysis. Hence-with it should be removed from the published literature on clinical trials, as it sole purpose is to confuse readers in to missing an inherent bias in the trial methods.

In 2004 CONSORT warned using the term modified intention to treat may lead to confusion and inaccurate results. This what they say about its use:

‘The term “modified intention-to-treat” is quite widely used to describe an analysis that excludes participants who did not adequately adhere to the protocol, in particular those who did not receive a defined minimum amount of the intervention.(232) An alternative term is “per protocol.” Though a per protocol analysis may be appropriate in some settings, it should be properly labelled as a non-randomised, observational comparison. Any exclusion of patients from the analysis compromises the randomisation and may lead to bias in the results.

Does it surprize anyone that a 2011 analysis published in Trials, of 367 RCTs reported a strong association between trials classified as modified ITT and for-profit agency sponsorship. The adjusted odds ratio was 7.41 – Oh, and as well, author conflicts of interest were also strongly associated.

Peer reviewers, editors need to sort this out or at the very least modify something soon. Readers look out for the modified intention to treat analysis, it should come with a health warning.