Button Text
November 23, 2018

Litigation Superforecasting Survey Results Part 1: Miscommunication in Legal Advice

Our survey results show that without % numbers in prospects advice, the stock phrases lawyers use in legal advice are bound to confuse.

Robert Rothkopf
Managing Partner

PART 1: MISCOMMUNICATION IN LEGAL ADVICE

Download this article as a PDF.

It’s a scene playing out every day all over the world: A client goes to her lawyer with a legal problem and asks – “Do I have a good claim?  Should I litigate or settle or just forget about it?”.

The lawyer sits back, realising that this piece of advice is going to be expensive – there are multiple complex contracts involved and a long history of correspondence between the parties.  “Let me take this away. My initial view is you’ve got reasonable prospects."

The lawyer spends days reading and mulling things over, and instructs associates to dig out relevant case law.  The client receives the written advice with a hefty invoice, and turns straight to the executive summary to find the conclusions: “on balance, you have a significant likelihood of succeeding on the contractual claim, and good prospects of succeeding on the negligence claim.

But what did the lawyer mean by this?

And what did her client understand?

Results from a recent survey by Balance Legal Capital have confirmed that probabilistic phrases like “good prospects” and “significant likelihood” carry highly variable meanings amongst both givers and users of legal advice.

We argue that if lawyers really want to innovate the way they deliver their advice to benefit their clients, the answer is to reduce reliance on probabilistic phrases and to apply % numbers to their views on prospects.

LITIGATION SUPERFORECASTING SURVEY

We issued a survey to solicitors, barristers, corporates and other users of legal advice around the world, asking them to put a percentage number against the following ten probabilistic phrases commonly used by lawyers:

  1. “near certainty”
  2. “significant likelihood”
  3. “serious possibility”
  4. “strong prospects”
  5. “good prospects”
  6. “reasonable prospects”
  7. “reasonably arguable”
  8. “more likely than not”
  9. “fair chance”
  10. “on balance”

The phrases were displayed in a random order each time respondents visited the online survey to reduce any biases that might be induced by the order in which the phrases were presented.

Balance recruited the assistance of cognitive neuroscientist Dr. Stephen Fleming of University College London, to help analyse the data from the 250 responses as at 1 August 2018.

Take the survey here. 1

THE SURVEY RESULTS

A key result from the Litigation Superforecasting Survey was that there is substantial variability in the way that probabilistic phrases are interpreted.

For example, the phrase “significant likelihood” was associated with probabilities ranging from below 25% to near 100%. The phrases “significant likelihood”, “serious possibility” and “reasonably arguable” showed the largest variance between respondents, making them the phrases most likely to lead to confusion in legal advice.

There was however structure in the data: despite variability, on average, respondents rated “strong prospects” as better than “good prospects”, which in turn was better than “reasonably arguable”, for instance.

TABLE 1: MEAN SCORES FOR EACH PHRASE AND VARIANCE

CHART 1: “RAINCLOUD” PLOTS SHOWING ALL SCORES AGAINST EACH PHRASE3

Proability Rating

“there is substantial variability in the way that probabilistic phrases are interpreted”

ANALYSIS

The implications of this data are clear: there is little utility in legal advice as commonly expressed because there is no common understanding in the language used.  If lawyers really want to be clear, they need to use percentage numbers in their prospects advice.  How much money has been spent on legal advice by clients who promptly get the wrong end of the stick?  How many litigation decisions were made under a misapprehension as to what the legal adviser meant?

Take the example given at the start of this article.  The conclusions were “on balance, you have a significant likelihood of succeeding on the contractual claim, and good prospects of succeeding on the negligence claim.”  The survey data shows that the lawyer could mean the contractual claim has prospects as low as 53% but the client could interpret it as 80%.  On the negligence claim, the lawyer might intend to convey “good prospects” at 55% but the client could understand it to mean 73%.

In fact, when the data is split between advisors (solicitors and barristers) on one hand and compared with “advice receivers” (everyone else) on the other, the data shows that advice receivers tend to attach higher probability figures to the verbal labels than “advisors”.

“advice receivers” tend to attach higher probability figures to the verbal labels than “advisors”

Robert Rothkopf, Managing Partner of Balance, argues that clients should insist that their lawyers provide numbers in their legal advice, and that lawyers should be happy to do so in order to reduce confusion:

If you are seeking precision in your communications between lawyer and client or simply within your own team, there is almost no point in using these phrases without also disclosing the percentage number you had in mind when you wrote those words.  Without the number, miscommunication and confusion are guaranteed between the writer and the reader. If lawyers really want to innovate the way they deliver their advice to benefit their clients, it is time to use numbers.

Simon Twigden, Senior Partner of Enyo Law, a disputes-only practice says:

Balance’s findings have thrown down the gauntlet to all litigators by challenging the utility of the stock phrases lawyers like to use. Without a percentage number, those stock phrases that often underpin, justify or caveat legal advice are pointless”.

In addition to reducing the chance for miscommunication, studies have shown that using numbers rather than vague language actually increases the accuracy of a forecast. This is because the forecaster is forced to consider carefully how he or she is thinking – a process called “meta cognition”. In so doing, the brain is more likely to identify and control for cognitive biases, to switch from over-relying on intuitions and instinct (or ego), and to engage more rigorously with the evidence.

Dr. Stephen Fleming, director of the Metacognition Group points out:

Studies into metacognition – the ability to think about thinking. have shown that sharing metacognitive estimates of our confidence with each other can improve how groups and committees work together.

This is also consistent with research by Dr. Adam Harris at UCL, whose studies suggest that numbers are a better communication format than words when dealing with forecasts and uncertainty.4

Notes:

  1. Balance’s Litigation Superforecasting Survey was inspired by the research of Philip Tetlock (the author of Superforecasting) which, based on studies of behavioural economists and forecasting tournament data, endorses a number of techniques found to improve forecasting accuracy. One of these is the use of numbers rather than phrases when making probabilistic assessments.
  2. SD = Standard Deviation
  3. Raincloud plots are data visualizations which show raw data, probability density, and key summary statistics such as median, mean, and relevant confidence intervals while preserving the ‘inference at a glance’ nature of barplots. For more, see: Allen M, Poggiali D, Whitaker K, Marshall TR, Kievit R. (2018) Raincloud plots: a multi-platform tool for robust data visualization. PeerJ Preprints 6:e27137v1.
  4. Sarah C. Jenkins, Adam J. L. Harris & R. Murray Lark (2018) When unlikely outcomes occur: the role of communication format in maintaining communicator credibility, Journal of Risk Research, DOI: 10.1080/13669877.2018.1440415

See the Financial Times’ coverage of our research here.

insights

Other posts you may be
interested in

February 27, 2024
Balance renews support for the National Pro Bono Centre

We are proud to share that we are once again a Gold Level sponsor, supporting the National Pro Bono Centre

February 8, 2024
Event: Financial Institutions Litigation | TL4 Disputes

Balance partner Oliver Hayes will be joining a specialist panel in a discussion on ‘Paccar and the Future Funding of Collective Actions’, at the upcoming Thought Leaders 4 Disputes Event on Financial Institutions Litigation.

December 8, 2023
Success in the High Court of Australia for the Ruby Princess Class Action

Another very important win for the Ruby Princess class action, a case supported by Balance.  

November 16, 2023
Podcast: Litigation Finance - A Growing Asset Class Which Offers Genuine Diversification Benefits

Our Managing Partner, Robert Rothkopf, makes a guest appearance on the latest episode of the Money Maze podcast, titled "Litigation Finance - A Growing Asset Class Which Offers Genuine Diversification Benefits."

November 6, 2023
Podcast: What is Legal Finance and How is it Used?

Maya Shallita from our Sydney office is a guest on ALFA’s Legal Finance Insights podcast in their latest episode, “What is Legal Finance, and How is it Used?”.

Subscribe to our mailing list

We'd love to stay in touch with you and share our news and insights.
Subscribe

Put a number on it.

What do you mean by "strong prospects"?
Take our survey