Is There a Difference Between Methods and Theory?

How do we measure the effectiveness of Compliance programs? It sounds like a simple question, but it is actually hard to answer. And it's not just a challenge facing Compliance Officers.

If you work in Ethics, Risk or even as a Regulator, you'll recognise the dynamic. You've probably either been asked to demonstrate the effectiveness of your program or had questions asked about your budget or resources. And usually, that's not because they want to give you more.

It's equally possible that you might have wondered how good a job you're doing. I know I did when working as a Regulator and as a Compliance Officer.

Now, whatever the reason for the question, it's not an easy one to answer. Because part of it involves proving a negative, you can easily show that you've helped make good things happen, but it's much harder to demonstrate that you've stopped bad things. This is why we often turn to input metrics like 'number of hours of training delivered' or the number of times we executed a particular control. That's interesting, but it doesn't tell you how effective the outcome is. Or isn't. Just because someone has been on a course, and done a little exam, doesn't mean the risk the training is designed to mitigate has been reduced.

So how can we best do that? That's what my two guests on this episode were interested in. One of them, Professor Benjamin Van Rooij, has been on the show four times — links to his previous appearances are below. The other, Professor Melissa Rorie, is making her debut. Together, they've edited a book called Measuring Compliance: Assessing Corporate Crime and Misconduct Prevention which contains chapters written by several eminent experts. Contributors to the book Todd Haugh, Ricardo Pelafone & Florentin Blanc have also previously been guests on the show. There are links to those episodes below.

If you work in a function whose role is to mitigate Human Risk, feel free to substitute the word Compliance for whatever you're responsible for ——–— because it's still relevant — then what Melissa and Benjamin have to say will be of interest.

Because if you don't know how to measure the success of what you do, then there's even less chance that anyone else — say someone who sets your budget, someone who is trying to assess the quality of your program or someone who determines your compensation — will be able to.

To find out more about the book: https://www.cambridge.org/core/books/measuring-compliance/5C1378AB4F9814D0C41198AEF9A5B6D2

For more on Melissa & her research: https://www.unlv.edu/people/melissa-rorie-phd

For more on Benjamin & his research:
https://www.uva.nl/en/profile/r/o/b.vanrooij/b.vanrooij.html

In our discussion, we talk about:

Hui Chen & her work at the DOJ: https://huichenethics.com/about/
Ricardo Pellafone & his work at Broadcat: https://www.thebroadcat.com/ricardo-pellafone
Florentin Blanc & his work at the OECD: https://oecd-events.org/digital-security-for-prosperity/speaker/f46fd492-79c3-eb11-94b3-000d3a219024
Rita Faria's book (co-authored with Olga Petintseva & Yarin Eski) called Interviewing Elites, Experts and the Powerful in Criminology https://link.springer.com/book/10.1007/978-3-030-33000-2
The research on signing at the top of documents to encourage honesty https://www.wired.com/2012/08/signature-honesty/ & the recent findings that it failed to replicate: https://www.pnas.org/doi/10.1073/pnas.1911695117
The story of Abraham Wald: https://www.rigb.org/explore-science/explore/blog/how-not-be-wrong#:
Professor JS Nelson on Workplace Surveillance https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3504408

Previous episodes featuring contributors to the book

Benjamin Van Rooij on COVID Compliance - https://www.humanriskpodcast.com/professor-benjamin-van-rooij-on-1/

Benjamin on The Behavioural Code - https://www.humanriskpodcast.com/professor-benjamin-van-rooij-on/

Benajmiong & Dany Sokol on Compliance 2.0 - https://www.humanriskpodcast.com/professors-benjamin-van-rooij/

Ricardo Pellafone on Behavioural Design in Compliance - https://www.humanriskpodcast.com/ricardo-pellafone-on-why-compliance/

Ricardo Pellafone on the challenges facing Risk & Compliance under COVID - https://www.humanriskpodcast.com/ricardo-pellafone-on-the-challenges/

Florentin Blanc on Regulatory Responses to COVID 19
https://www.humanriskpodcast.com/florentin-blanc-on-regulatory-responses/

Dr Todd Haugh on Behavioural Aspects of Ethics & Compliance https://www.humanriskpodcast.com/dr-todd-haugh-on-the/