Disruptive Innovation for Neuropsychology:

Dr Chris Gaskell

for:

Northern Care Alliance NHS
Foundation Trust

10/04/2024

Link to slides:
https://innovation-in-neuropsychology.netlify.app

Dr. Chris Gaskell

Senior Clinical Psychologist |
Clinical Academic

Neuropsychology

North Staffordshire Combined Healthcare NHS Trust

• Previously worked for SRFT (NEAD, Acute Neuro Rehab)

• CP in Stoke Neuropsychology (Neurology, Neuro-Oncology)

• Clinical Academic!

• Amateur interest in programming.

• Learning to code for about 5 years, and now I am trying to find ways of bringing it into my day job.

• Interested in many things; tools that can improve workflows for neuropsychology.

Contents

• Introduction

• Innovation

  • Relevance for Neuropsychology 🔥

• Shiny Apps:

  • App 1: Score Converter 📈

  • App 2: Reliable Change ⏱️

• Innovation.

• Why it’s relevant for neuropsychology.

• Why I there there is still a lot to be done.

• And then I am going to demonstrate and share a basic web app that I’ve previously worked on.

• Unlikely to have time to go through both apps today.

• I will briefly mention the reliable change app. But is anyone is interested is statistical approaches to change over time then feel free to message me!

Introductory Exercise:

1. Are we Consumers or Producers of technology/innovation?

2. What innovations have You been involved with?

Tips:

• Scorers and Software.
• Tele-Conferencing and Technology.
• Service Developments.
• Interventions.
• Research.

To start off with, is just an introductory exercise to get us fired up. I would like you to spend just a couple of minutes thinking about these questions, and then a little later I will invite people to say what they are, using the chat.

• As a neuropsychologist, do you see yourself as a consumer of technology, or a producer of technology?
• What innovations, developments, or idea have you been involved with or acted upon?
• What about your colleagues? Who do you look to when you think of a colleague who strives for innovation? Who has helped to turn the needle?
• Purpose of this task is to get you thinking explicitly about you as an individual, a service, a department, organisation, and how you have contributed to local innovation.

Neuropsychology as Developers

Why are neuropsychologists also good developers?

• Analytical minds 🤓

• Scientific method 🧑‍🔬

• Our history 🔍

• Empirical approach to assessment

• Progress Values 💪

• Public sector 💸

• Analytical: Some of the most incredibly analytic minds I have ever known.

• Embrace the Scientific Practitioner model, which is one of the things that has always allowed us to stand out. So our combined abilities for think creatively and analytically is bolstered by our ability to evaluate what we do.

• Our history: We literally founded the empirical investigation between brain and behaviour.

• Assessment: Think of all the empirically supported assessment tools.

• Progress: We are often fine advocates of holding important values; to be accessible, inclusive and generally holding an egalitarian perspective.

• Public sector: We appreciate the economical argument.

We should feel good about ourselves!

Psychology sits between academic science and clinical practice, right in the overlapping segment of a venn. That’s where innovation happens.

I would argue that we should position ourselves as innovators, developers even more! It’s a huge part of what we do in neuropsychology. As a field we are heavily involved in neuroscience, measurement, and advancing treatments.

Innovators!

Neuropsychologists are model innovators wtihin healthcare!

But where is leading the endeavour for Neuropsychology?

• Innovation is intrinsic to the way in which we work in neuropsychology.

• Collectively, we are a good profession for others to emulate.

• But now I want to ask you… Within the world neuropsychology, where does the real leading innovation happen.

  • Where is the home of Neuropsychology innovation currently?

  • We think of Apple for Tech, Tesla for cars, Netflix for entertainment, but who is the equivalent for neuropsychology?

  • It certainly isn’t the Oliver Zangwill centre.

  • You might say a University? Sure, but often the neuropsychology models that come out of Universities are based on cases that appear in routine practice.

• I am going to make a case for SRFT.

  • For years I worked as an AP in and around Manchester - it was always Salford that people would talk about.

    • The leading clinicians.

    • The innovative services (surgical pathways, the NEAD service)

    • A heavy output of peer reviewed research.

  • Always trainees wanting to be on placement here. It’s certainly why I requested to come on placement here.

    • If you count up the number of years of experience devoted to neuropsychology, what do you have. Half a millenium?

• I was super chuffed when Alistair asked me to come and speak for the department. Chuffed because I might have something that you want to learn about; but completely unsurprised that if I did have something to offer, that Salford would be the department to come and ask.

Feedback

What came to everyone’s minds?

1. Are we Consumers or Producers of technology?

2. What innovations have You been involved with?

• So with that. Hopefully now is a nice point to step backwards and look to these questions that I previously put.

• If people can put into the chat what came to their mind and share it - this is a time for you to remind each other of the fantastic work that Salford are doing.

• And I am dead nosey, so it’s a chance for me to hear about all the exciting stuff that Salford have got cooking up.

Can be challenging to toot our own trumpet - but it’s important that we are able to communicate this part of what we do.

However…

Outdated products

• Qinteractive 👎

• Old fashioned resources 👎

• Outdated assessments 👎

• Irrelevant norms 👎

• Manual scoring 👎

• Spent the first part of the presentation trying to build us up, but there is a need for us to remain humble because there’s a lot more to do.

• There are inspirational developments happening in neuropsychology.

  • There are also a lot of areas and processes in neuropsychology, where things are at risk of having become a little stale.

• In a world where

  • Surgical can be conducted remotely

  • Chat bots are delivering psychotherapy

  • And now, Elon Musk is putting Microchips into peoples brains.

• So why is it that I am still spending hours

  • Administering and scoring neuropsychology tests

  • And then struggling to put them into a neatly formatted table on Microsoft Word.

  • And then it takes many months to get the results back to someone.

If no time then don’t discuss each point in turn

• Qinteractive - Is this really the best that we can do in terms of an electronic assessment platform.

• Old fashioned resources - Handouts cheaply formatted that look like they have been produced on Microsoft 95 and printed to the cheapest possible paper.

• Outdated assessments - Often joke in the office about the majority of assessments used in in clinical practice being older than I am.

  • WAIS is the best part of 20 years old.

  • DKEFS = 2001.

• Irrelevant norms

• Manual scoring

However…

Unknown products

• Virtual Reality❓

• NLP❓

• Precision medicine❓

• Automated Reports❓

• VR, NLP and precision medicine

• Automated Reports

• Gap between a new innovation (research) and implementing it into healthcare, but also some of these problems aren’t technological - some of them are more about implementation and access. I would certainly say that a lot of these are solvable problems.

• A lot to be proud of in neuropsychology, however we probably are a critical point where we need to kick on with the field, in a similar way that other fields have been.

• These shouldn’t feel like a far reach, or something closer to science fiction than clinical practice.

Disruptive Technology:
An innovation that creates a new market or enters at the bottom of an existing market and eventually displaces established market-leading firms, products, and alliances

• Recently learnt about disruptive tech.

  • New term for me! it’s where a innovation or company starts a competely new market, or they do something so revolutionary that it turns the existing market on its head. Historically, these don’t come around every day. but in recent years, they are happening all the time.

• I am not sure that neuropsychology have had a truly disruptive innovation yet. But I really hope it’s coming!

• Amazon, Tesla, Uber, and AirBnB.

• I don’t know what a disruptive tech would look like for us, but it might resemble something that doesn’t include months for a patient to get a report that they can’t make sense of.

• It totally would not surprise me if whatever it is, it comes from someone at Salford.

Disruptive Technology Initiative

To encourage development of educational and assessment models that embrace multidisciplinary technological innovations, making neuropsychologists not just consumers of technology, but actual cutting-edge producers of technology.

American Academy of Clinical Neuropsychology

• For anyone who is interested in knowing more about the idea of disruptive technology I would encourage you to check out the Disruptive Technology Initiative which is a branch of the AACN.

• Website has some really helpful resources about encouraging innovation.

Why Is It left To Us?

• We have the skills to be able to do it.

• We are already doing it! but usually at a local level.

• Not a competitive market.

• Proprietary companies don’t always have our best interests.

• There is a pressure I think on us to push for more innovation without our field.

• As I previously explained, we have the skills to do it.

• We are already doing a lot of innovation locally.

• The rate of progress shouldn’t be dictated by proprietary companies.

• Unlikely to invest vast amounts in maintenance of development of products.

• Elon Musk brings out a chip to put in people’s brains (not a great comparison) meanwhile Pearson are still scrambling around trying to produce the WAIS-5 (started collecting data in 2016).

Making Our Tools Scalable

Lots of innovative work is done in silos.

• Incredible work being done in silos.

• The amount of exceptional

  • Home made excel scorers.

  • Self help interventions.

  • Unpublished research.

• A lot of this is used to a fraction of it’s maximal reach because it lives in an unknown region of an NHS shared drive gathering dust.

• Not everything has a home in peer reviewed journals

• Need for neuropsychology based innovation platforms that allow us to share, promote, and collaborate on projects.

  • On-line repositories of resources.

  • Innovation and technology based SIGs.

  • IT support that helps us to make our local tools and resources scalable.

  • Make use of colleagues with a diverse skill mix (design, coding etc.)

My Interests

• Tools to help us Expedite administrative process and develop workflows

  • Scoring Tools 🧰

  • Parametrised/Automated Reports 🤖

  • Visualisation 📈

• Data Science

• HTML (Apps, Books, Slides, Manuscripts)

• Some of my interests

• Neuropsychological assessment.

• We have been doing assessment for a long time. But the tools we use and the processes we follow haven’t really changed all that much.

• A lot of motivation improving ways of doing things day-to-day.

• Convenience. I want to reduce the administrative burden that we all have to manage; because I’m far more interested in spending my time with patients.

Shiny
An R package that makes it easy to build interactive web apps in the R programming environment.

Web Apps

• Scalable.

• Simple.

• Open Source.

• Many different ways in which we can innovate, but one of the areas I am becoming interested in is simple html apps.

• Examples from Others

  • Jon Crawford

  • TOPF

• Web apps can be used for many purposes. Data processing.

How Shiny Works

• Making something like took all of a few minutes.

• Granted this isn’t the most useful application, but it provides a proof of concept of how a an app can provide a scalable solution to make our lives easier at work.

What could they do for us

• Efficiency.

• Functionality.

• Reduce Errors.

• Reduce Paper.

My Apps¹

Score Converter: A tool for converting, tabulating, and visualising cognitive test score data.

Reliable Change: Detection of statistical change using standradised regression based models.

• These are my apps!

• Please do take a look at them and send over any feedback or ideas that you have.

App Development:
Problem Definition

• In order to persuade you that there might even be need for an app, I am going to briefly outline some problems to target.

Standard Workflow

flowchart TD
  A(Adminster Tests) --> B(Score Tests)
  B --> C(Convert Scores)
  C --> D(Tabulate)
  D --> E(Visualise)
  E --> F(Report)

Motivation

Many routine neuropsychology workflows

Take a long time.

• Take a moment to think about the routine workflow for a neuropsychological assessment

Motivation

Many routine neuropsychology workflows

1. Converting test scores.

2. Tabulating test scores.

3. Visualising test scores.

Take a long time.

These are all very useful practices however they are laboursome, clunky, and error prone.

➡️ The app theat I am going to discuss, Score Converter is an attempt to overcome some of these points

• There are all in relation to psychometric assessment.

• Apologies if this comes across as awfully dry!

• Reduce that administrative burden.

Conversions

• There are many types of standardised score².

• There is value in working with a unified metric³.

• Two primary approaches

  • Look-Up Tables

  • Manual Conversions

• Important for reports, communication, and interpretation.

• Descriptors are useful but also problematic.

Conversions

• Look-Up Tables

Standard Score	Rank	Scaled Score	T Score	Z Score	Sten Score
110	75	12	57	0.67	—
109	73	—	—	—	—
108	70	—	55	0.55	—
107	68	—	—	—	—
106	66	—	—	—	—
105	63	11	53	0.33	—
104	61	—	—	—	6
103	58	—	—	—	—
102	55	—	52	0.25	—
101	53	—	—	—	—
100	50	10	50	0.00	—
99	47	—	—	—	—
98	45	—	48	-0.25	—
97	42	—	—	—	—
96	40	—	—	—	5
95	37	9	47	-0.33	—
94	34	—	—	—	—
93	32	—	45	-0.50	—
92	30	—	—	—	—
91	27	—	—	—	—
90	25	8	43	-0.67	—

Problems with Look-Up Tables

For a full version of a traditional look-up table click here.

• \(Z\) score of 1.7 to a standard score.
• Standard scores 126, and 127, to \(T\) scores.
• Percentile rank of 87 to a scaled score.

Looks ups become constrained - they often aren’t granular enough to be able to ensure exact matches. And if they did then the table would be really long.

• Decimal places and rounding can influence precision.
• Increment length can influence precision.
• The best metric to use is likely to vary based on the purpose for use.

Conversions

• Look-Up Tables

Considerations:

• Decimal places and rounding.
• Increment length.
• The ideal metric can vary.

Conversions

• Manual Conversions

An alternative approach is to use a Formula.

\[ X_{new} = \frac{s_{new}}{s_{old}} (X_{old} - \bar{X}_{old}) + \bar{X}_{new} \]

However this can be:

• Timely
• Complicated
• Error prone

Conversions

Problem 1:
Need to be able to efficiently and precisely convert scores to a single metric, while also being able to flexibly change metric depending upon the audience.

Tabulation

• Table of results are really important.
• Don’t always report the required information.
• Often look ugly and cumbersome.

Tabulation

• Table of results are really important.
• Don’t always report the required information.
• Often look ugly and cumbersome.

Important

Commonly Incurred Problems:

• Not clear:

  • which metric.

  • which test, test version or test form.

  • which normative data.

  • which formula (e.g., reliable change).

• No confidence intervals.

• Poor formatting, cumbersome.

Tabulation

• Table of results are really important.
• Don’t always report the required information.
• Often look ugly and cumbersome.

Problem 2:
Need to be able to easily construct tables of results that are clearly formatted, intuitive, and promotes good reporting practices.

• Tables need to be really clear.

• But getting them to look really good is quite tricky (particularly using MS Word).

• Particularly if there is a perception that it is unlikely to be used again / by someone else.

• Important that all the important information is there - particularly is someone comes back to repeat the assessment in the future.

Visualisation

Data Visualisation is a really important tool for interpreting data!

Visualisation

Data Visualisation is a really important tool for interpreting data!

• More than just aesthetics!
  • New insights (e.g., trends).
  • Speeds up assimilation.
  • Communication.
• Rarely used in practice.

Visualisation

Data Visualisation is a really important tool for interpreting data!

• More than just aesthetics!
  • New insights (e.g., trends).
  • Speeds up assimilation.
  • Communication.
• Rarely used in practice.

Problem 3:
Need to be able to easily visualise a range of test scores in a way that is intuitive for a variety of audiences.

Finally - problem 3.

• useful when working with data. We work with a lot of data.

• Once you have converted all your scores to a single metric.

• And you have gone to the effort to put the data into a table and attached it to the EPR, the last thing we want to do is spend more time on admin / processing the data.

• And partly because of that, and also because it is just challenging to do, lots of people skip this part out.

• I think we are really missing a trick here because graphs are really powerful - for interpretation and also communication.

Example Visualisations

• Some feature available in Qinteractive.
• Possible to do with spreadsheets.

Case Study:
Score Converter

Score Connveter
A web app for converting, tabulating, and visualising neuropsychological test results.

Score Connveter
A web app for converting, tabulating, and visualising neuropsychological test results.

Conversions

Tabulation

Visualisation

Conversions

Tabulation

Visualisation

Conversions

• Formula based
• Flexibly change

Tabulation

• Consistent principles
• Conditional formatting and row ordering

Visualisation

• A range of plots (bar, forest, radar, bell curve)

• Score converter is an attempt to bring some of these ideas together.

• Converts scores but allows you to alter which your preferred score is.

• It produces tables that are reactive (alter how rows are arranged, for example grouping together tests on certain domains).

• Highlight all test scores that meet a certain criteria (e.g., show in bright red all scores that are 1 SD below the mean).

• A range of plots you can produce - as explained they are variations on a theme. One axis has the test and the other axis has the score.

Demonstration

• The app walks your through how to upload an excel sheet of test scores (e.g., for a battery of scores) so that you can convert, tabulate, and visualise large amounts of test scores at once.

• There are example datasets that people can easily select. This allows people to get a feel for what it produces, before having to commit to inputting scores into the spreadsheet.

• In terms of input it just required you to enter a single standardised score of any type. Unfortunately the app doesn’t convert from raw score score.

• Print to pdf allows for a simple pdf that you can upload to the EPR.

Forest plot

• Rapidly get a sense of the distribution of test scores.

Radar plot

• Look quite nice - although in my opinion they are quite hard to interpret.

• Perhaps good for communication information to patients

Bell curve

• Kind of useful - perhaps less so for patients because need to understand Z scores.

Bar plot

• Based on Z scores.

• Reactive.

Limitations & Next Steps

Score Converter:

• Batch input is clunky (Excel).

• Doesn’t include confidence intervals.

• Limited prior user testing.

Limitations & Next Steps

Other Projects:

• Reliable Change App.

• SCED App.

• Explore other ideas

  • Seizure tracking.

  • Automated online tests.

Next steps

• Write up / disseminate.

• Develop the app to address some of these limitations.

• Continue working on some of the other apps that I also rarely get time to develop.

Thank you for listening
Questions?

Footnotes

1. Hover over an image and click to be directed to the web app.

2. \(Z\) Score, \(T\) Score, Wechsler Standard Score, Wechsler Scaled Score, Sten Score, Stanine Score, Vineland Scaled Score, ETS Score, etc.

3. Crawford (2010) in Goldestein (2010) Clinical Neuropsychology: A Practical Guide to Assessment and Management for Clinicians, 2nd Edition