Evidence Based OT Practice, Part IV: Fact or Fad? A Case Study with Bal-A-Vis-X

4

This post is part of a series on evidence-based practice in OT. Read the previous posts in the series here, here, and here.


Many rehab therapists have fallen prey to using fad treatments that have little or no evidence to support them. There continue to be many lively Facebook debates and polarized opinions about various occupational therapy interventions. Those who believe the intervention is effective make a point of highlighting the many clients they’ve treated who have experienced significant gains, and non-believers turn their noses up and ignore on any evidence presented by those on the other side. As I’ve stated in previous posts about evidence-based practice, the issue in these cases is not whether to use new interventions without a ton of research evidence behind them, but the lack of clinical judgment and application of evidence-based practice principles when choosing and using OT interventions.

 

For example, in one setting where I worked I witnessed a department of over 50 experienced and intelligent clinicians buy into the promises of an intervention with only weak evidence to support it due to its extreme popularity in the pediatric professional sphere. No matter that this intervention required costly trainings and materials, was unlikely to be generalizable outside of a highly specific context, and was barely occupation-based – therapists wanted to learn it anyway! I was shocked that the department was spending limited continuing education funds on a popular pseudoscience program rather than a well-researched, evidence-based training that could help clinicians obtain better functional outcomes for young clients. Unfortunately, this is the case in many settings and with many practitioners.

 

The goal of this post is to closely examine one such program and break down how principles of evidence-based practice can be used to help make a determination about the validity, clinical applications, and utility of this program for occupational therapy practitioners. Read on to learn more about whether the widely-used Bal-A-Vis-X program is an evidence-based intervention or a highly-flawed fad!


THE PROGRAM

About Bal-A-Vis-X

One writer states “Bal-A-Vis-X Balance, Auditory, Vision eXercises provides 1,000s of rhythmic midline crossings that enable the mind-body system to experience the state of integration.” (http://in2gr8ed.org/wp-content/uploads/2017/10/apr17.pdf)

 

On the Bal-A-Vis-X website, the program is described as follows:

Bal-A-Vis-X is a series of Balance/Auditory/Vision eXercises, of varied complexity, all of which are deeply rooted in rhythm. Exercises are done with sand-filled bags and racquetballs, sometimes while standing on a balance board. In distilled essence, Bal-A-Vis-X enables the whole mind-body system to experience the natural symmetrical flow of a pendulum. Bal-A-Vis-X requires focused attention, demands cooperation, promotes self-challenge, and fosters peer teaching. It is school friendly and just plain fun.” (http://www.bal-a-vis-x.com/about.htm)

 

The website also includes links for purchasing materials, scheduling trainings, customer testimonials, and research articles. It is fairly simple, and appears to have been recently updated. The site mostly features information from program users and instructors touting its benefits for young clients and links to a few companies that “have created materials especially for Bal-A-Vis-X and are the ONLY authorized dealers for Bal-A-Vis-X in the world.” This last part seems a little strange, especially considering that most of the program materials (sandbags, racquetballs, balance boards), with the exception of the mysterious “VISTar balls,” are fairly inexpensive and easily obtained or replicated by program users.

After a brief review of the program website, the real work of uncovering evidence, identifying inconsistencies, and applying research and clinical reasoning skills begins.

 

THE BREAKDOWN

  • Initial Research: Typing Bal-A-Vis-X into Google Scholar yielded few relevant results. A primary result was a citation from a publication by Bal-A-Vis-X creator Bill Hubert. Additionally, the citation was from “Bal-A-Vis-X Incorporated,” which indicates that the “research” was produced by the company itself, rather than an independent source. Despite having been around since 2014, the source was also only cited by 10 other sources. A quick review of these sources showed that they varied in quality, “reliability” and age. Further, the underlying principles on which the program is based (positive effects of midline-crossing activity, brain laterality, and physical movement on learning) appear to be sound. However, the impact of these principles of Bal-A-Vis-X program – in comparison to innumerable other activities that require similar skills (sports, dance, table games, yoga, etc.) – on participant outcomes is unclear.

 

  • Research Quality: Many of the “studies” linking the use of Bal-A-Vis-X to improved client outcomes violated the foundational principles of research, basically invalidating them from the start. For example, almost none of the “studies” were single- or double-blind, and most researchers openly stated a bias towards the effectiveness of the intervention. Additionally, most papers presented an overly simplistic “cause and effect” or correlative conclusion definitively linking the use of Bal-A-Vis-X to improved academic and behavioral outcomes – neglecting to take into account confounds or alternative explanations (see below).

    I could easily identify even more issues that seriously compromise or outright disprove the validity of each author’s results, such as:

    • Lack of a control group
    • No clear intervention protocol (limiting ability to replicate study)
    • Outdated, low-quality, or minimal references
    • No discussion or acknowledgement of potential confounds or other causes for study outcomes (i.e. increased attention paid to the study participants, time effects, the impact of the physical exercise component versus the rhythmic “mind body integration” component of the exercises, and the effects of typical brain development over time)
    • Non-standardized assessments used as benchmarks

 

At best, a reader of the papers included on the Bal-A-Vis-X website could conclude that students enjoyed the intervention and their typing performance or classroom behavior improved over time. However, it is not reasonable to conclude that the program had any measurable effect other than being subjectively enjoyed by participants and administrators.

  • Levels of Evidence: After the initial Google Scholar search, I used the research pyramid to determine where the Bal-A-Vis-X program fit. The site’s “Research” page listed only three studies, which were all case studies implemented with a variety of methods and by individuals who clearly did not have extensive or high-quality research experience. Thus, the Bal-A-Vis-X program falls in one of the lower tiers as a “Case Study.” While qualitative research is also important for informing OT practice, clinicians should be wary of basing their clinical decision-making solely on anecdotes and individual client reports.
  • CRAAP Test: The CRAAP Test is a list of questions to help you evaluate the information you find about a specific intervention, program, or topic.
    • Currency: The lack of recent research (within the past 4-5 years) for the program is concerning, with most of the sparse studies on the “Research Results” page having been conducted from eight to thirteen years ago. The program also does not appear in recent systematic reviews, critically appraised topics, or other sources from professional associations or other reliable sources.
    • Relevancy: The program has been used with people of all ages and in various settings, so it may be relevant for OT practice. However, I would not feel comfortable citing the “research” listed on the site in a formal presentation or conversation with caregivers or colleagues. The program also appears to be fairly decontextualized, which does not align with the principles of occupational therapy practice.
    • Authority: The author is the owner of the Bal-A-Vis-X company, which may be a conflict of interest (but this not uncommon for program creators). He was formerly an experienced teacher and martial arts instructor.
    • Accuracy: Most of the information on the site comes from the creator’s experience, informal research projects, or customer testimonials. The information is primarily supported by personal anecdotes and case studies, and has not been validated by independent researchers or other reliable entities. When attempting to verify the information via Google search, several results describing the program as “pseudo-science” appear.
    • Purpose: The purpose of the program appears to be helping participants succeed (in various ways) and encouraging readers to participate in Bal-A-Vis-X trainings. Thus, there are altruistic and financial purposes for the program, which is true of many interventions.
  • Sources Cited: The “research studies” listed on the site are actually case studies by individuals who do not have clear research experience. The protocols described in the papers vary widely yet arrive at the same conclusion. Personal biases about the expected (positive) effect of the intervention decrease credibility. Additionally, the program can only be used after individuals pay to participate in a training course and purchase materials, limiting its generalizability and possibly increasing participant bias towards its effectiveness.
  • Professional/Expert Opinions: In the case of Bal-A-Vis-X, there are multiple expert opinions online that agree that the evidence base is extremely weak and the efficacy of the program (in comparison to control groups or other treatments) is unclear. Each person makes a point of saying that although the program doesn’t cause harm to users – although it clearly is at risk of causing significant harm to the perception and practice of OT – it is misleading for instructors to claim that it is evidence-based.

    “The Bal-A-Vis-X website posts its own research page, but you won’t find any gold-standard studies here. As with any intervention program, I’d caution you against using research posted on the company’s website. Obviously, no company would want to post research with refutes their claims. The “research” (and I use the term loosely here) supports the effectiveness of Bal-A-Vis-X. The “research” is little more than a handful of kindergarten students and teachers who wrote anecdotal commentary about how keeping kids active improved their academics. No doubt, activity is good for students. I do have doubt that it was the Bal-A-Vis-X method specifically which produced the results. My hypothesis is that the same amount of time given to playing basketball or other physical activity within the classroom would also increase students’ socialization, self-esteem, and general academic performance. Go ahead, prove me wrong.While writing this post, I came across this similar post from Musikinesis.com, which shares both my doubts about Bal-A-Vis-X and my general belief that it’s okay to use a program as long as it doesn’t hurt kids. It’s just important that we don’t tout the research effectiveness of such programs when the research to support them is substandard.…I don’t necessarily feel that it’s wrong to use a program without research to back up its effectiveness, but I certainly don’t think we should promote those programs as being effective unless the research concurs.” (Jason Wright, EdS, GCG)“There are three research studies linked on the site. They are all designed and conducted by, no doubt, sincere and well-meaning teachers. But you don’t have to be a scientist to recognize that the studies aren’t scientifically valid. As in many such studies, Bal-a-Vis-X process and effects aren’t distinguished from any other movement – jogging, dancing, sports, etc. We don’t know what, if anything, the control group did with their time, so it’s unclear whether any exercise, or other exercises with particular features, would have shown the same result.

All three studies begin with presuppositions about Bal-a-Vis-X. There’s nothing wrong with having biases or expecting particular results, but the studies were not double- blind or even single-blind, opening up huge possibilities for the testers’ own biases, behaviors, and expectations to influence results.

Much of the hypotheses rest on the same debunked theories aligned with BrainGym…None of the studies describe random groupings. One grouped the students by classrooms, introducing different peer dynamics, classroom environments, teachers, etc. Sample groups were small, and got smaller, and whether differences were statistically significant isn’t addressed. It’s also unknown whether pre- and post- tests were administered under the same conditions. Like BrainGym, Bal-a-Vis-X is selling around the world.” (Monica Dale, Master of Music, Piano, Dalcroze)

 

“As an OT who has practiced in the school systems for 10 years in 5 different states, I can confirm that this blatant lack of regard for evidence and rigor in our field is stunningly widespread. …My current contract has me working in a region program that has accepted Bal-A-Vis-X as the corner stone of their intervention plans. Not only are these therapists wrongly applying a pseudoscience based perceptual motor program to a broad range of students (from the student with a mild case of ADHD to a child with severe CP who has limited hand functioning) with a subjective treatment goal (improve attention) but the most frightening thing is they appear to not be able to recognize pseudoscience.

I have seen this inability played out over and over again in each district and state that I work in….

My second point, and I think the most important one is, the responsibility does not end with the leaders in the OT world, as street level therapists we must demand more from our colleagues. It should not be acceptable to receive a file from a therapist with limited documentation, sparsely informative evaluations and exclusive use of unproven interventions. No longer can we accept whining about caseload size and driving times as legitimate reasons to be ineffective therapists. We must expect that our colleagues are adhering to the code of ethics published by the AOTA that expressly states that evidence based interventions be given preference over those with limited or no evidence behind them…” (Gretchen Scheibel, MS, OTR/L, BCBA)

 

 THE VERDICT

Ultimately, my findings indicated that while the Bal-A-Vis-X program is well-known and well-liked among therapists, teachers, and others, there is very little compelling or high-quality evidence that the program leads to lasting results or improvements in occupational engagement or performance for users. This isn’t to say that program administrators and participants do not enjoy participating and/or report positive outcomes (improved attention, behavior, etc.) after implementing the program, as there are many Youtube videos, personal anecdotes, and Pinterest Pins indicating that this is true.

 

However, multiple “red flags” I encountered while conducting my research caused me to categorize this program as anything but a skilled occupational therapy intervention, including:

  • Disparity of approaches and outcome measures used to determine program effectiveness
  • Weak credentials of program founder and individuals writing “research papers”
  • High cost of program training and materials
  • Low quality of evidence on program website
  • Lack of citation within peer-reviewed research articles or other reliable sources
  • Decontextualized, non-occupation-based nature of intervention
  • No long-term follow-up on participants
  • Reputable, qualified professionals describing the program as “pseudoscience” or “not evidence-based”

 

Perhaps Bal-A-Vis-X is a fun preparatory or therapeutic activity to have in your OT “toolkit,” but OT practitioners should consider whether teaching teachers or caregivers to administer this specific, non-evidence-based intervention is more effective than teaching strategies clients can use across settings, time, and populations to improve client participation and occupational performance.

When I consider how to make the most of the treatment time I have with caregivers, teachers, and clients – especially with their limited time and resources – I do not believe that using this intervention would be as effective in providing skilled instruction in environmental modification, adaptive equipment, sensory modulation, universal design [for learning], cognitive strategies, and other evidence-based, client-centered, and occupation-based occupational therapy interventions.

 

This is my interpretation of the information I found, and readers are free to arrive at their own conclusions. However, I believe that the evidence above makes a compelling case for why Bal-A-Vis-X should be viewed as a “brain game,” movement break, preparatory activity, or play/leisure activity rather than an evidence-based intervention that will lead to specific functional outcomes for occupational therapy service users or others with and without special needs.

Advertisements

Tagged: , , , , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: