It has been a little over a year since I took over this column from Dr. Narda Robinson, and I have enjoyed the opportunity to explore many interesting topics through the lens of evidence-based medicine (EBM). One core element of EBM is substantive, critical debate, and discussion about the merits and limitations of published research evidence. Every stakeholder, from clinicians to researchers to those in industry to the owners of veterinary patients, will have a different mix of goals and perspectives. The interplay among different sets of biases is one of the mechanisms for cutting through bias generally and achieving an accurate understanding of nature that informs safer and more effective patient care. One of the strengths of science is that it requires a community process. My columns have drawn responses from readers in a variety of forms, including letters to the editor. While extensive debate is not logistically possible in the letters section of any publication, I appreciate the feedback and the opportunity to explore differences of opinion in a way that illuminates underlying issues and the EBM process. In this column, I am going to reach into my virtual mailbag and pull out a few letters that I believe it would be productive to respond to. General issues Readers typically don’t write letters to the editor to express absolute agreement and unqualified praise for an article. As enjoyable as such letters are for the author, people are more often motivated to respond to statements they disagree with or that conflict with their views. This is useful, as disagreement is more likely to lead to reflection and advancement of ideas. Such letters have been referred to as a form of “post-publication peer review,” and are considered an important part of the scientific process.1-3 The tendency of such letters to criticize the original article is also a variety of publication bias, creating an impression of a generally critical response from the community that may or may not be accurate. This has created some problems in public perception of controversial scientific issues, such as climate change or vaccine safety, in which a small number of letters expressing an extreme position have given a false impression of the popularity and legitimacy of such a position.4 Confirmation bias leads us all to evaluate new information in light of our existing beliefs. We automatically accept the validity of information that supports our views and treat disconfirming evidence much more critically. Similarly, we will often weigh the reliability of an information source in ways favorable to our point of view. Criticizing the methods of a study or the potential bias inherent in the institutional affiliation of an author is fair game, but we all tend to do so more aggressively when we disagree with the author. Conversely, we ignore potential bias and overweight credentials and expertise when we want to agree with someone.5-7 These are universal habits worth keeping an eye on. Preoperative blood testing In my column about diagnostic testing,8 I looked at the limitations of the evidence that pre-anesthetic blood tests reduce morbidity or mortality in surgical procedures. One writer responded: “… the tone of the piece clearly is against this common practice… there are scientific and experiential studies showing that the value of such screening, which should include a urinalysis as well, is to reveal the early prodromal issues facing individuals—especially geriatrics—in order to provide a baseline set of values to be monitored forwards.” - Jean Dodds, DVM, HEMOPET Blood Band & Veterinary Diagnostic Laboratory I am not opposed to pre-anesthetic testing and, in fact, use it in appropriately selected cases all the time. My goal was to point out that there is not strong controlled research evidence validating the common belief that such testing improves outcomes for most patients or that the benefits of the practice outweigh the potential costs and harms. Given the growing evidence in human medicine of harm from overdiagnosis with many screening tests, it is worthwhile for veterinarians to question our assumptions and interrogate them with data. If establishing a baseline of clinical laboratory values for each individual and monitoring over time reduces morbidity and mortality, it should be possible to demonstrate this through controlled research. This would then justify such screening, which is currently based only on plausible reasoning and anecdote, not objective evidence. Pheromones My March 2018 column regarding the use of pheromones in dogs and cats9 generated critical responses that focused on my selection and interpretation of published evidence: “[Dr. McKenzie] claims there is limited published peer-reviewed research regarding pheromones and then chose five studies to support his belief. I am aware of at least 22 other peer-reviewed published studies… [and] 11 other studies demonstrating the efficacy of pheromones have been presented at conferences… Dr. McKenzie mischaracterized or misinterpreted some of the findings in the papers he mentions… Based on the bulk of the current research, as well as almost 20 years of clinical experience, there should be no question that pheromones can play a beneficial role in the prevention, treatment and/or management of many anxiety-based behavior problems in our pets.” - Valerie B. Tynes, DVM, DACVB, President of American College of Veterinary Behaviorists, Veterinary Services Specialist, CEVA Animal Health “Let me start by stating that I am using pheromones in my practice, and authored one of the studies that Dr. McKenzie refers to in this article… The scientific level of the studies, for most, is at its highest. From my own experience, I can state that I have run a pilot study with many subjects prior to the actual study. Based on the pilot study, adjustments were made to ensure the highest quality of scientific guidelines. As a clinician, researcher, author, reviewer, and editor of scientific articles I am surprised to read that the author chose to ignore the robust data which exist on the clinical use of pheromones… The author largely bases his opinion on a study by Frank et al. 2010.10 This review had several major flaws in its design that reduce the power of their finding.” - Gary Landsberg, DVM, DACVB, DECAWBM, North Toronto Veterinary Behavioral Specialty Clinic, Vice President, CanCog These letters raise several important issues. The first, of course, is the question of the role of clinical experience in evaluating medical therapies. Both authors cite clinical experience with pheromones as supportive of their belief these products are effective. A key EBM principle is that clinical experience is unreliable. This idea frustrates clinicians tremendously, especially as the lack of high-quality controlled research evidence in veterinary medicine often forces us to rely on our experience. It’s difficult to accept that our observations aren’t a very good guide in treatment. Fortunately, both Dr. Tynes and Dr. Landsberg go beyond clinical experience to focus on the research evidence concerning pheromones. Dr. Tynes emphasizes that I failed to consider research supportive of pheromone therapy. However, much of the literature she refers to was considered, and rejected as insufficiently reliable, in the systematic review by Frank et al. 2010.10 She also refers to studies that exist only as abstracts from conference reports. While such reports are a great way to glimpse ongoing and cutting-edge research, they do not meet the evidentiary standards of peer-reviewed studies, and so the evidence they provide must be viewed as lower in quality. Dr. Landsberg specifically responds to the use of Frank et al. 2010 in order to point out flaws in the conduct of this review. Systematic reviews, of course, always have such flaws, and they are not perfect evidence—merely the best we have. I agree with some of Dr. Landsberg’s criticisms of this review, though I do not think we should dismiss the conclusions entirely. Inevitably, those who support a given therapy will identify the flaws in a review that do not support their views, as Dr. Landsberg does in reading Frank et al. 2010, and as the author of this systematic review did in appraising one of Dr. Landsberg’s own studies.11 Both critiques have merit, and the final conclusion regarding the weight of the evidence should incorporate such appraisal of all the evidence. Finally, both authors claim that I also misinterpreted evidence. This generally refers to my pointing out negative findings in studies that also had positive findings and in which the authors reached positive conclusions about efficacy. While none of us deny there is both positive and negative research evidence and that all the existing studies have flaws, we differ in emphasis and how we interpret the significance of the existing evidence for our clinical practice. Drs. Tynes and Landsberg consider the evidence “robust” and “highest quality,” and believe there should be “no doubt” about the value of these products. I believe the evidence can be more accurately characterized as mixed with limitations, consistent with the conclusions of Frank et al. 2010. Outcomes in behavior research and therapy are often subjective and subject to greater risk of bias than more objective outcomes, such as mortality, in other areas of medical research. I don’t believe Drs. Tynes and Landsberg give sufficient weight to the role of caregiver placebo effects in the evaluation of these products, or to the limitations of the positive studies they cite. I do agree that the risk of harm from pheromones is negligible. Both Dr. Tynes and Dr. Landsberg make legitimate points, and there is room in the evidence for different interpretations; I encourage clinicials to examine and evaluate the literature for themselves. Hopefully further research will be completed, and my views are, as always, provisional pending additional evidence. Lasers After publication of my column on laser therapy,12 Dr. David Bradley, a proponent of laser therapy and the veterinary medical director at K-Laser, began his letter by agreeing with my positive comments about the evidence concerning lasers. He then dismissed the portion of my article not supportive of lasers by citing flaws in the studies referenced and suggesting I lack the qualifications to interpret the relevant science: “Unfortunately, the article also presents statements based on referencing several flawed studies. Without a thorough knowledge of a subject, a reviewer will be unable to competently critique the scientific articles related to that subject. Therefore, the conclusions drawn may be erroneous. This can serve to create more confusion rather than clarification related to the topic. Most of the ‘equivocal’ articles referenced in this editorial to support the premise that laser results are not consistent or lack the impetus to justify the enthusiasm and rapid growth had significant deficiencies to make them a valid resource. I will not critique each individual article, but will outline in general the discrepancies…” - David Bradley, DVM, FASLMS These “discrepancies” include purportedly inappropriate therapeutic technique (e.g. insufficient dose or inappropriate wavelength) or inappropriate statistical methods (e.g. underpowered studies or not viewing statistically significant differences between them as convincing evidence of a clinical effect). Dr. Bradley also emphasized the safety of laser therapy, claiming no possibility of “untoward effects on healthy tissue.” While illustrating the aforementioned tendency to dismiss research inconsistent with one’s beliefs as fatally flawed without giving similar weight to the limitations of research in agreement with these beliefs, Dr. Bradley also raises the subject of expertise. It is sometimes argued that only clinicians or researchers intimately involved with a therapeutic practice are qualified to evaluate it. This view ignores the potential for bias inherent in specialized expertise. No one becomes an expert in a therapy, devoting years to studying and employing it, unless they are deeply committed to a belief in the value of that therapy. This requires those considered experts to be a self-selected group of committed believers in their area of expertise. All of the authors of these letters have academic credentials, clinical experience, and commercial affiliations of one kind of another relevant to the subject matter. These inform their views just as my own background informs mine, and we all have insights and blind spots stemming from our experiences and perspectives. The level of commitment required to qualify as an expert in a particular therapy also raises the bar of cognitive dissonance extremely high, making it very difficult for such experts to change their mind about their chosen practices regardless of the nature of the evidence. This is one reason why major shifts in medical practice often require generational change to take effect. The legitimate respect for expertise also conflicts with another of the core values of EBM—that all medical professionals should be critical thinkers, and that the critical appraisal of published research evidence should not be the domain only of academics and experts, but of all conscientious clinicians. If those who critique the evidence are only those who produce it or take a particular view of the underlying issue, then such echo chambers will only reinforce and perpetuate these views. The communal process of science requires substantive criticism of all ideas from a variety of perspectives. My own expertise is in epidemiology and the process of evaluating evidence to inform clinical practice. This undoubtedly means I may be ignorant of important facts, and this may introduce error into my interpretations of the evidence. It also means I am not committed to specific hypotheses or beliefs about particular interventions, and I can apply the methods of EBM equally to any subject. Both kinds of expertise have a valuable role to play in the community process of generating reliable knowledge and effective medical therapies. I appreciate the authors of these letters for their insight and criticism. It is laudable they have taken the time to offer thoughtful contributions to discussions about specific interventions, and about the use of evidence and experience in evaluating these interventions we must have as a community of scientists and clinicians to sustain progress in veterinary medicine. References Falavarjani KG. Kashkouli MB. Chams H. Letter to Editor, a scientific forum for discussion. J Curr Ophthalmol. 2016;28(1);1–2. Tierney E. O’Rourke C. Fenton JE. Eur Arch Otorhinolaryngol. 2015;272:2089. Papanas N. Georgiadis GS. Maltezos E. et al. Letters to the editor: definitely not children of a lesser god. Int Angiol. 2009;28(5):418-20. Young N. Working the fringes: The role of letters to the editor in advancing non-standard media narratives about climate change. Public Understanding of Sci. 2013;22(4):443-59. Kida, T. (2006). Don’t Believe Everything You Think: The 6 Basic Mistakes We Make in Thinking. New York: Prometheus Books. Tavris C. Aronson, E. (2008) Mistakes Were Made (But Not by Me):Why we Justify Foolish Beliefs, Bad Decisions, and Hurtful Acts. Boston: Mariner Books. Burton, R. (2008). On Being Certain: Believing You’re Right Even When You’re Not. New York: St. Martin’s Press McKenzie B. Why do we run diagnostic tests? VPN. 2018;20(2):38. McKenzie B. Pheromones therapeutic use in animals. VPN. 2018;30(3):36. Frank D. Beauchamp G. Palestrini C. Systematic review of the use of pheromones for treatment of undesirable behavior in cats and dogs. J.Am.Vet.Med.Assoc., 2010;36(12):1308-1316. Denenberg S. Landsberg GM. Effects of dog-appeasing pheromones on anxiety and fear in puppies during training and on long-term socialization. J Am Vet Med Assoc 2008;233:1874–1882. Brennen McKenzie, MA, MSc, VMD, cVMA discovered evidence-based veterinary medicine after attending the University of Pennsylvania School of Veterinary Medicine and working as a small animal general practice veterinarian. He has served as president of the Evidence-Based Veterinary Medicine Association and reaches out to the public through his SkeptVet blog, the Science-Based Medicine blog, and more. He is certified in medical acupuncture for veterinarians. Columnists’ opinions do not necessarily reflect those of Veterinary Practice News.
