Last week, the nation’s leading heart organizations released a sweeping new set of guidelines for lowering cholesterol, along with an online calculator meant to help doctors assess risks and treatment options. But, in a major embarrassment to the health groups, the calculator appears to greatly overestimate risk, so much so that it could mistakenly suggest that millions more people are candidates for statin drugs.
Risk Calculator for Cholesterol Appears Flawed
By GINA KOLATA
Published: November 17, 2013 553 Comments
Mark Graham for The New York Times
Dr. Nancy Cook and Dr. Paul M. Ridker of Harvard Medical School found that a new online calculator used to assess heart treatment options overestimated the risks that many people face.
Multimedia
Related in Opinion
Room for Debate
When Medical Experts Disagree
How should doctors and patients react to the latest findings that come out about health concerns? How do you know whom to trust?
Readers’ Comments
Share your thoughts.
The apparent problem prompted one leading cardiologist, a past president of the American College of Cardiology, to call on Sunday for a halt to the implementation of the new guidelines.
“It’s stunning,” said the cardiologist, Dr. Steven Nissen, chief of cardiovascular medicine at the Cleveland Clinic. “We need a pause to further evaluate this approach before it is implemented on a widespread basis.”
The controversy set off turmoil at the annual meeting of the American Heart Association, which started this weekend in Dallas. After an emergency session on Saturday night, the two organizations that published the guidelines — the American Heart Association and the American College of Cardiology — said that while the calculator was not perfect, it was a major step forward, and that the guidelines already say patients and doctors should discuss treatment options rather than blindly follow a calculator.
Dr. Sidney Smith, the executive chairman of the guideline committee, said the associations would examine the flaws found in the calculator and determine if changes were needed. “We need to see if the concerns raised are substantive,” he said in a telephone interview on Sunday. “Do there need to be changes?”
The problems were identified by two Harvard Medical School professors whose findings will be published Tuesday in a commentary in The Lancet, a major medical journal. The professors, Dr. Paul M. Ridker and Dr. Nancy Cook, had pointed out the problems a year earlier when the National Institutes of Health’s National Heart, Lung, and Blood Institute, which originally was developing the guidelines, sent a draft to each professor independently to review. Both reported back that the calculator was not working among the populations it was tested on by the guideline makers.
That was unfortunate because the committee thought the researchers had been given the professors’ responses, said Dr. Donald Lloyd-Jones, co-chairman of the guidelines task force and chairman of the department of preventive medicine at Northwestern University.
Drs. Ridker and Cook saw the final guidelines and risk calculator on Tuesday at 4 p.m., when a news embargo was lifted, and saw that the problems remained.
On Saturday night, members of the association and the college of cardiology held a hastily called closed-door meeting with Dr. Ridker, who directs the Center for Cardiovascular Disease Prevention at Brigham and Women’s Hospital in Boston. He showed them his data and pointed out the problem. On Sunday, officials from the organizations struggled with how to respond.
Other experts said there has not been a real appreciation of the difficulties with this and other risk calculators. “I don’t think people have a good idea of what needs to be done,” said Dr. Michael Blaha, director of clinical research at the Ciccarone Center for the Prevention of Heart Disease at Johns Hopkins University, who was not associated with forming the new guidelines.
Dr. Blaha said the problem might have stemmed from the fact that the calculator uses as reference points data collected more than a decade ago, when more people smoked and had strokes and heart attacks earlier in life. For example, the guideline makers used data from studies in the 1990s to determine how various risk factors like cholesterol levels and blood pressure led to actual heart attacks and strokes over a decade of observation.
But people have changed in the past few decades, Dr. Blaha said. Among other things, there is no longer such a big gap between women’s risks and those of men at a given age. And people get heart attacks and strokes at older ages.
“The cohorts were from a different era,” Dr. Blaha said.
This week, after they saw the guidelines and the calculator, Dr. Ridker and Dr. Cook evaluated it using three large studies that involved thousands of people and continued for at least a decade. They knew the subjects’ characteristics at the start — their ages, whether they smoked, their cholesterol levels, their blood pressures. Then they asked how many had heart attacks or strokes in the next 10 years and how many would the risk calculator predict.
The answer was that the calculator overpredicted risk by 75 to 150 percent, depending on the population. A man whose risk was 4 percent, for example, might show up as having an 8 percent risk. With a 4 percent risk, he would not warrant treatment — the guidelines that say treatment is advised for those with at least a 7.5 percent risk and that treatment can be considered for those whose risk is 5 percent.
“Miscalibration to this extent should be reconciled and addressed before these new prediction models are widely implemented,” Dr. Ridker and Dr. Cook wrote in The Lancet. “If real, such systematic overestimation of risk will lead to considerable overprescription.”
In a response on Sunday, Dr. Smith of the guidelines committee said the concerns raised by Dr. Cook and Dr. Ridker “merit attention.”
But, he continued, “a lot of people put a lot of thought into how can we identify people who can benefit from therapy.” Further, said Dr. Smith, who is also a professor of medicine at the University of North Carolina and a past president of the American Heart Association, “What we have come forward with represents the best efforts of people who have been working for five years.”
The chairmen of the guidelines panel said they believed the three populations Dr. Ridker and Dr. Cook examined were unusually healthy and so their heart attack and stroke rates might be lower than expected.
Asked to comment on the situation on Sunday, some doctors said they worried that, with many people already leery of statins, the public would lose its trust in the guidelines or the heart associations.
“We’re surrounded by a real disaster in terms of credibility,” said Dr. Peter Libby, the chairman of the department of cardiovascular medicine at Brigham and Women’s Hospital.
What are patients and doctors to do? On Sunday, there seemed to be no firm answers, except that those at the highest risk, like people who have had a heart attack or have diabetes, should take statins.
The guideline developers said they were not totally surprised by the problems with the calculator.
“We recognize a potential for overestimates, especially at the high end of risk,” said Dr. David Goff, the dean of the University of Colorado School of Public Health and the co-chairman of the guidelines’ risk assessment working group.
Last year, not long after it received the assessments from Dr. Ridker and Dr. Cook, the National Heart, Lung and Blood Institute removed itself from the development of the guidelines, saying that was not its mission. The institute handed responsibility to the American Heart Association and the American College of Cardiology.
Dr. Michael Lauer, the director of the division of cardiovascular sciences at the institute, said on Sunday that it had received many reviews and sent them to the other groups, together with the responses of the guidelines’ authors.
Some doctors who tested the calculator with hypothetical patients wondered if they should trust the results.
Dr. Nissen entered the figures for a 60-year-old African-American man with no risk factors — total cholesterol of 150, HDL (the good cholesterol) of 45, systolic blood pressure of 125 — who was not a diabetic or a smoker. He ended up with a 10-year risk of 7.5 percent, meaning he should be taking cholesterol-lowering statins despite being in a seemingly low-risk group.
Dr. Nissen also calculated the figures for a healthy white man, age 60, and also got a risk factor of 7.5 percent.
“Something is terribly wrong,” Dr. Nissen said. Using the calculator’s results, he said, “your average healthy Joe gets treated, virtually every African-American man over 65 gets treated.”
Get 50% Off The New York Times & Free All Digital Access.
553 Comments
Share your thoughts.
That the calculator has glitches is one thing. That the researchers, once confronted with the evidence it was faulty, struggled with how to handle the issue is quite telling...
What decision other than "we had better announce that we were wrong, we're sorry, and that we need to fix this immediately" could they possibly have ever considered ... for even a moment?
And they were are worrying about their credibility? Too late... it's shot.
This, plus the Obama administration's lack of candor in promising that your health care insurance won't change if you don't want it to, is more of the same credibility-destroying hapless decision making in action.
I can understand the cynicism. But as an academic cardiologist, I can unequivocally say that the guideline writers were motivated by only the best interest of public health.
I think the bigger problem is that although we have gotten very good at things like treating acute heart attacks, we still have a lot of difficulty at predicting risk for individual patients. Part of the challenge for guideline writers is how to make the best recommendations when the science is imperfect (which it almost always is), in ways that can still be useful and easy to apply.
The notion of a risk calculator that assigns arbitrary values to a qualitative event is non-scientific, non-statistical hokum.
Tom Ridge's Terror Alert Status color codes would be a more credible way of telling people to worry about their cholesterol status.
I'm a 68 year old triathlete, 17% body fat, with HDL 100, average cholesterol, 118 systolic blood pressure, resting heart rate of 54 and the calculator tells me my risk is over 10% so I should be taking statins. This is a joke for my age cohort.
Using the calculator age seems to dominate as a predictor. At optimal levels for all other risk factors all males over 63 seem to qualify for treatment (7.5 or over) and it is startling that at age 70 the risk jumps to nearly 13 with no other risk factors changing. At age 75 the risk jumps to 18.6 and by 79 the risk is 24. Is there a treatment for age?? Is it really the implication (and recommendation) that everyone over 63 seriously consider taking statins and/or medication to lower blood pressure even if those levels are already optimal?
Short-term memory loss is a very common side effect of statin drugs, and one that you will often read about or hear about from people who take or have taken statins. The drug companies, however, play this down (maybe they hope you won't remember that you don't remember?). They claim that this and other side effects are "not serious." But the fact is, people who suffer the side effects (or their families) will tell you that it is, in fact, quite serious. This needs to be a part of the discussion.
Retired MD here with coronary artery disease. Statins are important. No question they are a useful drug. BUT, they are only one component in the prevention and treatment of coronary artery disease. Item number one is to have a normal weight. Item two is never smoke. Item three is exercise. Four is to eat an intelligent diet. Five is to remove stress from your life as much as practical. If everybody did these five things (all of which are free), the incidence of coronary artery disease would plummet and many fewer would need statins. If your BP is not controlled by the above, it is important to treat it. Nobody should be allowed to have a statin prescription until all of the above are fullfilled. Otherwise you are basically treating sloth with the statin.
It bothers me that the calculator has no corelation to weight or BMI. Common sense dictates that a high BMI implies higher risk. I believe that higher BMI cannot be treated with statins. So is the purpose of this calculator to predict risk or to recommend statin usage? Makes me question the motives.
The ability to prevent heart attacks and strokes is a crucial life and death health issue, especially to those in their sunset years.
These honest, but conflicting and confusing opinions by some of the best and the most respected cardiologists concerning guidelines for statin therapy draw special attention to this issue.
Perhaps cardiologists, now, will not over rely on statistical evidence in deciding whether a cholesterol-lowering drug is needed. Perhaps their decisions now will be mainly made on the basis of empirical evidence each individual condition prompts.
That will be a positive outcome to this open and enlightening debate.
As a primary-care physician since 1977, I have weathered waves of guidelines from on high and have seen several crash upon the shore of "Oops.... Never mind."
Here we go again.
This recent, puzzling broad-brush recommendation of statins for millions more patients does not yet pass the smell test, despite statistics, now suspect, supposedly showing cardiovascular benefit.
Little mention is made of an additional potential downside, i. e., cognitive costs from statins (about 5% in my experience). I wonder about a possible sub-threshold impact in some patients on statins who may not notice that they are subtly dimmer.
Of course, the underlying triple juggernauts of obesity, sedentary lifestyle, and the warping of dietary choices are clinically threatening and tough to counter. Only AFTER maxing ACHIEVABLE benefit from non-pharmacologic recommendations, such as SIGNIFICANT dietary changes, usually SIGNIFICANT weight loss, and SIGNIFICANT aerobic fitness, do I prescribe statins, because they work and are often a necessary, practical compromise, as the least-bad option, after considering pros and cons.
The patient is certainly entitled to decline the recommendation.
It is tiresome to read another embarrassing bit of evidence seeming to show that doctors somehow can’t wait to throw pills at every problem. Despite common misperceptions, the majority of us are not paid to do so, as we don't own a piece of the cash cow, which now appears to be mooing, “More meds for the masses.”
I don't see any evidence of a conspiracy here -- but I do see blind belief in a pharmaceutical solution and some sloppiness. If the web site is executed this badly, what does that say about the scientific soundness of the recommendations?
Remember the Heart Association and the College of Cardiology have been prime advocates of low-fat, high-sugar, high-carb diet that has led to an epidemic of obesity and diabetes in our society.
Their recommendation continues to be a carb-heavy diet and pills aimed at ameliorating the effects of that diet. If you assume people eat junk food, than maybe pills are a good last resort. But is the diet recommendations that, for many, have caused the problem in the first place.
When this news broke, I went online and found several different versions of the risk calculator. I ran my data through them and got widely different results. The "official" one gave me a 12% risk. One of the others came up 6% -- the difference between "Treat" and "Don't Treat". So what's a guy to do ?
I then entered different data -- but kept my age the same; the results were the same every time. This is called a "sensitivity test" -- if you change the inputs, you SHOULD get different outcomes. That didn't happen.
I also note that none of the calculators I found had any input for exercise and activity level, two factors that are clearly correlated with cardiovascular health.
As far as I'm concerned, this whole thing is bogus. And I strongly suspect that the studies leading to it were funded by the companies that make and sell statins.
I am a retired physician. Age 70, total cholesterol 126, LDL 63, BP controlled 100/70, Type 2 diabetes well controlled.
The formula would not accept my actual cholesterol level. It was "too low". My "risk" was 7.5%. I varied EVERY parameter to the "best" level. The "risk" did not change. Does anyone think I will take statins???
The most disturbing part of all this is that they were informed of the problem at least a year before adopting the guidelines.
Be prepared for more of this with Obamacare. It is was I call "cookbook medicine". Get a diagnosis, look up the list, get the drugs. Soon your pharmacist will be prescribing. There will be NO 'individualized" medicine practiced. The Physician Assistants and Nurse Practitioners will be judged by the new "recipe". How many of your patients follow the checklist? Any deviation requires a series of phone calls to one of the specialists that made up the "guidelines" to get approval. Someone who never will have to deal with the patient will make the decisions.
I think self-care is the way to go. I no longer implicitly trust even my own beloved doctors. The profession has been somewhat corrupted by big money. It's not the fault of most individual doctors; they are dependent on their organizations for up to date information. It would be another full-time job for them to investigate whether or not medication X or Y is truly effective - not to mention diet and other factors impacting health. In terms of the new health care law, in terms of what is covered, knowing the truth is basic - but we don't know it. And the larger question, why we are all so sick, isn't even asked, but it should be!
To all those who conclude that the guidelines were influenced by drug companies in order to sustain their profits, you may have a valid concern, but I respectfully suggest you open your minds a bit more. The issue at hand is the age of the data sets used to establish the calculator's regression coefficients. These can be altered, and perhaps should be after further discussion and testing against more up-to-date data. Another key point, not emphasized here, is the overwhelming evidence that risk calculators of this sort, when systemically applied, assure that more patients receive the best evidence-based care. Physicians are humans, and they make errors, such as failing to educate themselves about, or implement, current best practices. The calculator can and certainly will be improved as data accumulates, but the process (using an expert computer system rather than relying on often ill-informed doctors) is the right one. This innovation should lead to many lives extended and saved.
While we may respect the seriousness and good faith of physicians, too much research shows that people both consciously and unconsciously skew their conclusions based on where their private and institutional interests lie. Too many of the people involved with these new guidelines have close connections, financial and in prestige and status, with large pharmaceutical companies. Even if they errors here were not intentional, it seems very likely that they were skewed in many subtle ways to favor more prescriptions, more medication, more treatment. The net effect of many such minor skews can be very substantial. This isn't to attack those involved personally, or accuse them of bad faith (though bad faith is certainly an additional possibility), but simply to recognize that we tend to favor those who have supported and favor us, often in ways that we're not even aware of.
For example: the Boston experts' critique was sent to the panel drafting the guidelines, but somehow didn't make it to the designers of the calculator. Was this oversight -- a hesitation that turned into a failure to pass it on, perhaps? Conscious or unconscious suppression?
Years of evidence have shown that the medical profession signs on to therapies that are beneficial for large pharmaceutical or device manufacturers, 'in good faith' -- and later we discover that the therapies were in fact useless or even harmful to patients. Skepticism about the panel and its operation is thus entirely warranted.
I noticed the same thing about the lifetime risk calculator, and it did not inspire confidence. But as far as I can tell, no decisions are to be made using the lifetime risk calculator -- it's only supposed to be used to scare people.
The decisions about statins etc are supposed to be made using the 10 year risk calculator, which uses continuous rather than binary variables. However, that doesn't mean that model is any better. Even setting aside the concerns raised by those two physicians, just the information in Appendix 4 of the guideline is enough to make me wary of the models. They are actually based on relatively small sample sizes for making predictions about a binary outcome in that many different populations, and it looks to me like they probably overfit the models.
I used to be a medical writer. Several years ago, I wrote up, as internal reports, about two dozen transcripts recorded at meetings with local doctors that a major drug company held all around the country. The meetings concerned its statin. Two ideas presented at these meetings by the marketing team, and agreed with by the physician attendees, were: 1) the muscle pain reported by patients was almost never caused by the statin but was the result of excessive gardening, golfing, etc; 2) many children should be prescribed a statin and told that they would have to take the drug for life. I quit medical writing altogether after that job, and I grow increasingly dismayed at what a variety of corporations have done/continue to do to health care in this country. I have no idea who was behind the cholesterol guidelines debacle, but the error is inexcusable.
It's exasperatingly clear that we do not need to ensure that poorer people can get care that is too often substandard at exorbitant rates. One thing that is needed, and desperately, is strict regulation of the many types of suppliers that influence the insane cost structure and of those who bill the government for their services and supplies. Apparently, an overhaul must include guidelines producers. No health care system should be such an appalling mess as ours is. Super-aggressive, and often wildly inappropriate, drug marketing in general is a big part of the problem, whether or not the prescriber has been misled.
The cardiac risk calculators attempt to quantify an ever moving target; as cultural, lifestyle and dietary trends change, and as medical therapy evolves, the calculators have less meaning. Different ethnic groups and gender considerations are important, as well. The desire to quantify risk has merit, but the treatment of an individual patient is an "art", and cannot be distilled to a number by the medical quants.
The failure in this case was predictable, and is inexcusable.The CONFIRM registry showed that the Diamond-Forrester classification overestimated risk in men with atypical and typical angina. The implication is that risk calculators serve as a clinical tool with a limited shelf-life, and should not be major factors in determining the appropriateness of therapy in large population groups.
The only group that benefits consistently by these obsessive efforts to distill cardiac risk to a number with a simple calculator are personal injury attorneys--and this group of guideline authors are their best friend.
Would everyone who smells conspiracy please bear in mind that the major statins have now all gone generic and sell for pennies, so Big Pharma has absolutely nothing to gain from the risk calculator. Personally, I believe that any statin benefits are derived from their anti-inflammatory effect, which is independent of their lipid-lowering property. Whether that's worth a prescription or not remains to be seen. As for the risk calculator itself, it probably fails because it does not and cannot take into account certain individual characteristics that affect risk. Or, as some have suggested, the info on which the calculator is based comes from outdated patient cohorts.Or some other miscalculation. So the risk calculator is an unfortunate, massively embarrassing mistake, not an evil plot. The to-do will all be sorted out eventually, and all the conspiracy theorists can go back to arguing whether there was a second gunman on the grassy knoll.
A missing piece in this discussion is genetic variability, which apparently is being ignored. My endocrinologist did not stop pestering me to take statins (after severe side effects) until I had my genome sequenced and presented to her the results indicating high risk of complications from taking them.
She is virtually required by the "standards of practice" to insist on a drug that is harmful to me, despite documented difficulties. Now that she has proof that this is not a good medication for me, she can avoid being criticized for practicing real medicine.
With a prescription strength fish oil supplement and one other non-statin medication, my cholesterol and triglyceride profile have improved.
It's time to start practicing based on readily available information on genetic phenotype and stop the "one size fits all" blindered recommendations for wholesale prescribing.