It’s a scene playing out every day all over the world: A client goes to her lawyer with a legal problem and asks – “Do I have a good claim? Should I litigate or settle or just forget about it?”.
The lawyer sits back, realising that this piece of advice is going to be expensive – there are multiple complex contracts involved and a long history of correspondence between the parties. “Let me take this away. My initial view is you’ve got reasonable prospects.
The lawyer spends days reading and mulling things over, and instructs associates to dig out relevant case law. The client receives the written advice with a hefty invoice, and turns straight to the executive summary to find the conclusions: “on balance, you have a significant likelihood of succeeding on the contractual claim, and good prospects of succeeding on the negligence claim.”
But what did the lawyer mean by this?
And what did her client understand?
Results from a recent survey by Balance Legal Capital have confirmed that probabilistic phrases like “good prospects” and “significant likelihood” carry highly variable meanings amongst both givers and users of legal advice.
We argue that if lawyers really want to innovate the way they deliver their advice to benefit their clients, the answer is to reduce reliance on probabilistic phrases and to apply % numbers to their views on prospects.
Litigation Superforecasting Survey
We issued a survey to solicitors, barristers, corporates and other users of legal advice around the world, asking them to put a percentage number against the following ten probabilistic phrases commonly used by lawyers:
- “near certainty”
- “significant likelihood”
- “serious possibility”
- “strong prospects”
- “good prospects”
- “reasonable prospects”
- “reasonably arguable”
- “more likely than not”
- “fair chance”
- “on balance”
The phrases were displayed in a random order each time respondents visited the online survey to reduce any biases that might be induced by the order in which the phrases were presented.
Balance recruited the assistance of cognitive neuroscientist Dr. Stephen Fleming of University College London, to help analyse the data from the 250 responses as at 1 August 2018.
Take the survey here. 1
A key result from the Litigation Superforecasting Survey was that there is substantial variability in the way that probabilistic phrases are interpreted.
For example, the phrase “significant likelihood” was associated with probabilities ranging from below 25% to near 100%. The phrases “significant likelihood”, “serious possibility” and “reasonably arguable” showed the largest variance between respondents, making them the phrases most likely to lead to confusion in legal advice.
There was however structure in the data: despite variability, on average, respondents rated “strong prospects” as better than “good prospects”, which in turn was better than “reasonably arguable”, for instance.
|More likely than not||55.6||55||6.8||48.8||62.4|
“there is substantial variability in the way that probabilistic phrases are interpreted”
“advice receivers” tend to attach higher probability figures to the verbal labels than “advisors”
Robert Rothkopf, Managing Partner of Balance, argues that clients should insist that their lawyers provide numbers in their legal advice, and that lawyers should be happy to do so in order to reduce confusion:
“If you are seeking precision in your communications between lawyer and client or simply within your own team, there is almost no point in using these phrases without also disclosing the percentage number you had in mind when you wrote those words. Without the number, miscommunication and confusion are guaranteed between the writer and the reader. If lawyers really want to innovate the way they deliver their advice to benefit their clients, it is time to use numbers.”
Simon Twigden, Senior Partner of Enyo Law, a disputes-only practice says:
“Balance’s findings have thrown down the gauntlet to all litigators by challenging the utility of the stock phrases lawyers like to use. Without a percentage number, those stock phrases that often underpin, justify or caveat legal advice are pointless”.
In addition to reducing the chance for miscommunication, studies have shown that using numbers rather than vague language actually increases the accuracy of a forecast. This is because the forecaster is forced to consider carefully how he or she is thinking – a process called “meta cognition”. In so doing, the brain is more likely to identify and control for cognitive biases, to switch from over-relying on intuitions and instinct (or ego), and to engage more rigorously with the evidence.
Dr. Stephen Fleming, director of the Metacognition Group points out:
“Studies into metacognition – the ability to think about thinking. have shown that sharing metacognitive estimates of our confidence with each other can improve how groups and committees work together.”
This is also consistent with research by Dr. Adam Harris at UCL, whose studies suggest that numbers are a better communication format than words when dealing with forecasts and uncertainty.4
- Balance’s Litigation Superforecasting Survey was inspired by the research of Philip Tetlock (the author of Superforecasting) which, based on studies of behavioural economists and forecasting tournament data, endorses a number of techniques found to improve forecasting accuracy. One of these is the use of numbers rather than phrases when making probabilistic assessments.
- SD = Standard Deviation
- Raincloud plots are data visualizations which show raw data, probability density, and key summary statistics such as median, mean, and relevant confidence intervals while preserving the ‘inference at a glance’ nature of barplots. For more, see: Allen M, Poggiali D, Whitaker K, Marshall TR, Kievit R. (2018) Raincloud plots: a multi-platform tool for robust data visualization. PeerJ Preprints 6:e27137v1.
- Sarah C. Jenkins, Adam J. L. Harris & R. Murray Lark (2018) When unlikely outcomes occur: the role of communication format in maintaining communicator credibility, Journal of Risk Research, DOI: 10.1080/13669877.2018.1440415