I invited Dr Aaron Frost to contribute his excellent article to my blog because it's about time we as a profession in psychology focused more on outcomes based evidence rather than evidence-based psychotherapy. Based on the meta analysis of decades of research, we find there is equivalence between various treatment models, also known as the "Dodo Verdict," which is: "all have won therefore all must have prizes" as Hubble, Duncan and Miller describe in their book "The Heart And Soul Of Change."
The active ingredient of what works in therapy is not the treatment model per se but other factors - such as the relationship, expectancy (placebo) effects and client resources. Theoretically, if you have a great therapeutic relationship, a resourceful client and a knowledgeable and experienced practitioner, a tarot card reading would be just as effective as CBT.
It's about time common sense returned to our profession and I encourage my colleagues to take a stance on the value we bring to the public by helping to contribute to Dr Aaron Frost's research and to stop politicians framing what we do in terms of cost and pseudo-science that is evidence-based practice.
I am sick of being lectured by politicians and professional health advocates about how expensive the ‘Better Access’ program has become. I am especially sick of the entire discussion being framed in terms of cost.
This critique is naïve; as any economist will tell you, the true test of value is not costs, but return on investment.
So I would like to frame this article in terms of the value of psychological services. As a profession, we need to be able to demonstrate the value of the services we provide. I have spent the last seven years in private practice working towards exactly this goal through the use of routine outcome evaluation. But first lets take a quick look at the cost side of the ledger.
Economics of Mental Health
Estimates vary, but Australia spends between $7 billion and $13 billion annually on direct treatment of mental illness. On top of this, we also spend another $15 billion annually on the indirect costs. Finally, mental health costs the economy approximately $6 billion annually in lost productivity.
Is spending $600 million on these services a worthwhile investment of public funds?
In order to consider the return on this investment, we need to answer three important questions:
People who are treated are less likely to consume indirect services like welfare and the criminal justice system ($15 Billion). They are also likely to be more productive in their workplace ($5.9 Billion).
Potential Savings = $20.9 Billion
In 2009-2010 46% of those with mental health conditions accessed services. So of the costs associated with mental illness, treatment can at most hope to reduce costs in only 46% of the this population.
Potential Savings = $9.6 Billion
According to a synthesis of large international meta-analyses of psychotherapy efficacy trials, approximately 70% of treated clients will experience clinically significant improvement. However, in a large study of real world treatment outcomes only 16.6% of treated clients improved. This discrepancy presents a real credibility gap for our profession, and massively limits the effectiveness of any advocacy that we can do.
Potential Savings (Upper Limit) = $6.7 Billion
Potential Savings (Lower Limit) = $1.6 Billion
Remember, these are only theoretical upper limits. It would be foolish to suggest that the true economic value is anywhere near these figures.
The point remains though, that finding $600 million worth of value in a potential $6.7 Billion dollars worth of potential savings is a lot easier than finding it within $1.6 Billion worth.
However, in the absence of reliable data from psychologists as to the effectiveness of our services, policy makers will be left to draw conclusions from the two numbers above.
How can we improve these estimates?
In our practice we have chosen to ensure that there is no question of how effective our services are. Informed by the pioneering work of Michael Lambert, and Scott Miller, we routinely track client outcomes using a brief symptom- monitoring tool every session. We are then able to calculate effect size as the difference in symptoms scores between the first and the final session.
Therefore I can confidently say that 67% of clients treated by our services last year experiences clinically significant gains. If our figures are representative of the entire profession, then the case for advocacy is much stronger. This type of improvement would suggest total potential savings of $4.8 Billion. If this is the actual potential pool, then treated clients need only reduce indirect costs and increased productivity by 12.5% in order for the program to break even.
An accurate estimate of psychology services effect sizes is needed. I am in the process of collating data from a number of Australian private practices to present a more comprehensive estimate of psychology effect sizes. Click here to see the results so far. If you collect outcome data, I would love to hear from you to include your overall effects in this estimate.
Benefits to clients of tracking outcomes
Besides being good in terms of accountability, routine evaluation has the added virtue of actually being good practice. There is mounting evidence that simply collecting client outcome data improves therapy effectiveness. In our practice, this has had the following effects;
When we identify a client who is not improving, this allows the clinician to reformulate their thinking. Perhaps the diagnosis is incorrect, perhaps there is something missing in the conceptualization, perhaps the treatment of choice just doesn’t fit for this client.
Collecting client outcome data has improved the overall measured outcomes for our clients as well as reducing dropout.
At the end of last year, we were able to produce the data in Figure 2. This graph shows the effect size of each clinician within our practice. Firstly, it is incredibly reassuring for all of us to see that we are doing ok and, for the most part, we are achieving outcomes consistent with best practice.
With the ever-increasing need for private, public and NGO service delivery agencies to demonstrate return on investment and accountability, these systems also form a major part of our commitment to quality assurance and professional development with our clinicians.
By delving a bit deeper into the data clinicians are able to identify their areas of weakness in order to improve.
To give an example: below is the complete 2014 data for one of our clinicians arranged in order of least successful therapy to most effective (NB: negative scores indicate deterioration, positive scores indicate improvement)
From the point of view of supervision, it is now easy to identify a number of cases that are worth auditing to see what lessons can be learned:
Better Access represents a relatively large piece of government spending, and it should rightly be subject to strict evaluation of its effectiveness. I have been involved in service evaluations since 2001, and feel confident in saying that there is a direct relationship between how seriously a program is committed to outcome evaluation, and its 10-year survival rate. Fashions change, governments change and so do funding priorities. However, services that are well sandbagged by a wall of evidence are more likely to ride out these disruptions.
When we as a profession are better able to demonstrate our effectiveness, we will be less vulnerable to political caprice or expedient cost cutting. In addition to simply being accountable, tracking outcomes actually improves client outcomes as well as delivering direct benefits to both clinicians and supervisors.
About Aaron Frost
Dr Aaron Frost is a Clinical Psychologist in private practice with a long interest in mental health service delivery. He has established a group private practice entirely around the collection of outcome data, both to improve client outcomes, but also to improve accountability. As well as being a senior research fellow at Griffith University, he consults to a range of small and large agencies interested in customizing this approach to their setting.
I've have created this blog to help you clarify and resolve pressing issues you are currently struggling with. I hope that you find the following advice useful but remember to contact your preferred mental health professional for help in implementing some of these strategies.