How to assess the effectiveness of risk management

July 17, 2019 26 comments

Internal auditors are expected, according to the IIA Standards and some governance codes, to assess the effectiveness of risk management.

That can be a challenge, especially as:

  • There is no commonly accepted idea of what effective risk management is.
  • While both the COSO ERM framework and the ISO 31000 standard provide principles for effective risk management, neither (in my opinion) is sufficient.
  • Few organizations are seen as having effective risk management, so there is no exemplar against which to measure. (The majority of organizations manage the potential for failure, not the likelihood of success – the gold standard of what is commonly called risk management.)

My good friend, Alex Sidorenko has given this challenge a valiant try in his recent video. (I encourage you to follow him as he challenges traditional thinking – something we should all do.)

3 things to look for when auditing risk management identifies three areas to assess:

  1. Organizational performance compared to prior years, industry benchmarks, and so on
  2. How well the company makes decisions. Is risk information integrated with how decisions are made?
  3. Culture, including risk-related policies and procedures and attitudes towards risk

Taking each in turn, organization performance is a poor indicator of effectiveness. Many succeed simply by being lucky; others fail, despite excellent people and processes, when unfortunate and unforeseeable events occur.

How the company makes decisions is at the heart of effective risk management. But looking at minutes and other records of meetings where decisions are being made is not likely to be revealing. Best is to be present when the decisions are made, failing that follow the example of my friend Grant Purdy.

Grant is now retired, but he was a prominent risk practitioner and thought leader (including chairing the committee that developed the excellent Australia/New Zealand’s risk standard on which ISO 31000 is based). He then turned his hand to consulting. When he was hired to upgrade an organization’s risk management practices, he met with the senior executives. Instead of asking about risk management, he asked:

How do you make decisions?

The lesson here is that the individuals assessing ‘risk management’ should meet with decision-makers and ask that question. From there, they can move to questions like:

  • How do you consider all the things that might happen and affect the results of your decision?
  • When you consider the things that might happen, both positive and negative, how do you assess them? How do you weigh the good and bad together?
  • How do you know the information you are using is complete and reliable? What is the likelihood of it being incomplete, inaccurate, out-of-date, or in some other way deficient?
  • Who is involved in making the decision? Do all potentially affected parties participate?
  • If there is a risk function, how does it help you make decisions? Is it worth the cost of the function? How could it help you more?
  • Are you able to adapt with agility when things change? How will you know when there has been a change such that the decision or actions flowing from the decision need to be reconsidered?
  • …and more

Alex’s third is really, in my mind, a continuation of the second. I would prefer to think about how the decision-makers know what risks the board and top management want them to take.

Let me suggest my own top three:

  1. Do decision-makers believe that there are reliable processes to support decision-making, including the availability of current, reasonably complete, and reliable information about what might happen under each of the options they are considering?
  2. Do decisions involve the weighing, in a disciplined way that allows them to be compared, both the upsides and downsides of each option?
  3. Do they believe the risk function (if there is one) is helping them set and then execute on strategy? Is it all it should be?
  4. Do the organization’s processes and practices provide reasonable assurance that there will be an acceptable likelihood of success (measured by the achievement of objectives)?

OK, there are four. I cannot cut any of them out, they are all so important.

Which set of three (or four) do you like more?

Do you have your own?


The next generation of internal auditing

July 14, 2019 2 comments

I want to congratulate Workiva and Jose Tabuena for Internal Audit’s Guide to Planning, Managing and Addressing Risks. I want to focus on the first piece in that publication, Planning to Do the Right Audits: An Effective Internal Audit Risk Assessment.

Here are some excerpts, with comments by me:

  • While the responsibility for identifying and managing risks belongs to management, a key role of internal audit is to provide assurance that those risks are being appropriately addressed and mitigated. [ndm: sometimes it is appropriate to take risk, even more of it, for business reasons.]
  • Are you confident that your department understands the risks that are critical to the delivery of value and the achievement of corporate objectives? Every organization faces numerous risks that matter individually to managers with whom auditors interact, but are they risks that matter to the organization as a whole? The risks that truly matter are those that need to be addressed in the audit plan. [ndm: this sound like something I would say.]
  • Change does not occur on an annual basis. The move to a continuous and dynamic audit plan is significant for most internal audit departments.
  • It’s usually those who are in the details on a daily basis that have the best perspectives on risks and low-hanging fruit when it comes to increasing operational efficiency. [ndm: in other words, don’t just talk to senior management. Talk to the people who know what is really going on.]

The only disagreement of significance I have with Jose is when he talks about the risk assessment and planning being performed every six months. To the contrary, it should be at the speed of risk and of the business.


Protiviti has also shared their perspective. Next Generation Internal Audit: Catch the Wave is a collection of case studies featuring 16 different internal audit departments.

The overall message is not new: internal auditors need to change to meet business needs. That has been a constant in my professional life (going back decades).

I am not going to share excerpts from the Protiviti publication. I found it generally lacking in new and exciting practices. For example, the various CAEs talk about agile, but they are talking about the agile methodology, not necessarily in being agile. By agile, I mean able to change direction quickly to address what matters today as business conditions and related risk change.

Most still audit what matters to a process or business unit, rather than the enterprise as a whole. There is also a continuing failure to perform continuous audit planning.

Finally, many of the CAEs (with consultants cheering them along) are becoming owners of detective controls as they use RPA and other technologies to identify potential problems with data – rather than providing assurance that management is able to do that.

But those of you in internal audit might find value in reading about what other companies are doing.

If you want to know more about my ideas for ‘next generation’ internal audit, consider Auditing that matters.


I welcome your thoughts.


Elevating internal audit’s role

July 9, 2019 9 comments

For many years, PwC has shared with us their view of the State of the Internal Audit Profession.

This year, the subtitle is Elevating internal audit’s role: The digitally fit function.

They have some useful words, but it is mixed in with an agenda with which I don’t totally agree. I will come to that later. But first, the good stuff:

  • Internal audit needs (1) the dexterity to pivot quickly and to keep up with the digital pace of the business, and (2) the knowledge and skills to provide advice and strategic assurance in this new arena.
  • Internal audit has to have a seat at the table with management. As you build these out, you don’t want internal audit to come in afterwards and identify gaps in controls. They really need to be there right at the beginning. However, it’s one thing physically having a seat at the table but another having the credibility to be listened to.
  • Dynamic internal audit functions are embracing new technologies from multiple dimensions by providing advice and assurance that appropriate controls are in place as their organisations adopt new technologies and by using the technologies within their own departments to streamline the function.
  • Internal audit leaders universally agree that annual plans and annual assessments are antiquated. More frequent and more-fluid cycles are what’s [sic] necessary today, and the vast majority of internal audit functions now revisit risk assessments and audit plans more frequently than they used to.
  • We’re doing preimplementation [sic] work focused on key strategic priorities to address any potential concerns real time.

Where I don’t fully agree with PwC is on the need for internal audit to put what they call “digital fitness” at the top of internal audit priorities. In fact PwC seems to assess internal audit effectiveness based on the function’s digital capabilities (both in understanding the enterprise’s digital systems and initiatives and in using digital technologies themselves).

Before considering digital fitness, an internal audit function has to have a deep understanding of the business: its business model, organization, objectives, and related risks.

Far too many audit the weeds of technologies and identify issues management has missed, but are unable to assess how those issues might affect the business as a whole and the achievement of its objectives. In fact, technical auditors can be misled by the romance of new technologies into spending time on issues that are not critical to enterprise success while leaving more mundane but significant areas on the table.

In addition, we must not forget that internal audit is not there to identify what management has missed. They are there to provide assurance that management has the ability to identify and address risks of significance. It’s better to see whether management has assessed and acted on the more significant technology-related risks than to set up internal audit as having that responsibility. If necessary, help management learn to fish (after talking to them and senior leadership about that as a weakness) rather than be the fisher of risks yourself.

PwC is obsessed with robotic-process automation (RPA). While this can be a very effective tool in monitoring data and processes, its use by internal audit should be questioned. After all, it is essentially a detective control and it’s management that should be employing it.

There has to be a good reason for internal audit to be the control, identifying data or other anomalies, rather than assessing whether management has the appropriate controls in place.

Internal audit should be (enterprise) risk-based in its planning, execution, and reporting.

Identify the risks that should be audited (and update the plan continuously). Only then select the tools to use. That includes making sure you have the people tools (staff) to be effective.

Be digitally fit to address and add value on the more significant risks to enterprise objectives.


I welcome your thoughts.

Insight into effective risk management

July 3, 2019 12 comments

I don’t know Christopher Burt of Halex Consulting, although we are connected on LinkedIn.

But I need to draw your attention to a provocative piece by his firm (presumably by him): The risks of risk management. (My thanks go to Tim Leech for tweeting about it.)

While he doesn’t reference either World-Class Risk Management or this blog, what he says is very much in line with my core message:

  • The periodic review of a list of top risks is not effective risk management. It actually has very little value in leading the organization to success.
  • Organizations need to obtain confidence that there is an acceptable likelihood of achieving enterprise objectives. (Some prefer to talk about certainty in achieving objectives; it’s the same concept but I don’t like talking about certainty or uncertainty – it’s confusing.)
  • Its not about managing risk. It’s about achieving those enterprise objectives. Chris talks about performance management whereas I say this is simple effective management.

You will see how Burt’s language is consistent with mine. For example, he says:

  • In many businesses, there is a tendency towards ‘risk listing’, with the primary focus on documenting, assessing and prioritising lists of risks. Sadly, in most cases this approach adds little value, leading to page-turning discussions around the top 10 or 20 risks whilst diverting attention away from the real value of risk management – helping the business deliver its strategy through achieving its objectives.

In the end, the thing risk listing is most successful at is convincing the board and senior management that they are dealing with risk in the same way as other organisations, since this approach is endemic across UK and international businesses.

  • The purpose of risk management is not to manage risks per se. The purpose of risk management is actually to help the business deliver its strategy through focusing on achievement of its strategic business objectives.
  • Moving the focus away from risks and onto business objectives, or key goals, is also more natural and engaging way to consider risks. In effect, it puts risk in the context of reward and focuses senior management and Board attention on the objectives that the organisation is trying to achieve, and what they need to do to increase the certainty of achieving them. It should also lead to a more forward-looking mind set, increased focus on priorities and greater responsiveness to unexpected events.
  • The third line [of Defense] (Internal Audit) remains responsible for providing independent assurance over all aspects of the organisation’s activities, including looking at the ERM system and the work of the second line. A brave Internal Audit function may even opine on whether management has fairly stated the certainty of it achieving its business objectives.

I welcome your comments.

Making intelligent and informed decisions around cyber

June 28, 2019 10 comments

The experts continue to bombard us with their advice, insight, and guidance for addressing cyber.

One of those experts, KPMG, recently shared What’s next: Key cyber considerations for 2019. Unfortunately, I don’t think it has much to say that is new or valuable – it points out what we should all already know. Frankly, its more a marketing piece than thought leadership.

The FAIR Institute has probably the best methodology for quantifying cyber exposure. Their chairman has penned an interesting document, Understanding Cyber Risk Quantification, a Buyer’s Guide.

He makes a number of points with which I agree, including:

  • The cyber risk landscape is increasingly impactful, complex and dynamic, and organizations have limited resources to apply to the problem.
  • Furthermore, every dollar spent on cyber risk management is a dollar that can’t be spent on other business or mission imperatives.
  • It’s important to recognize however, that measuring risk quantitatively shouldn’t be a goal in itself. What is most important is ensuring well-informed decisions through reliable and meaningful risk measurements (whether qualitative or quantitative).

Unfortunately, the decisions envisaged by the author are what I would call siloed decisions. He talks about funds being allocated for cyber and how the FAIR methodology can be used to decide where to spend those funds.

The FAIR and other methodologies and guidance are not nearly as useful as we need in providing the information that executives need to make strategic and tactical decisions, such as:

  • How do I ‘aggregate’ the various risks to my business and its objectives? How do I see the big picture so I can consider whether the potential rewards from a new venture outweigh all the related (downside) risks? A cyber risk assessment using FAIR or other approach doesn’t give me something I can readily add to other business risks to see that big picture.
  • How much should I invest in cyber when (as pointed out in the FAIR document) “every dollar spent on cyber risk management is a dollar that can’t be spent on other business or mission imperatives”? When is it right to accept cyber risk?
  • How do I compare the value to the business of investing in cyber protection to the value obtained from an investment in new products or a marketing initiative?

I tried to address these and other questions in Making Business Sense of Technology Risk.

Have you seen an approach that works, providing management and the board the information they need to make strategic and tactical business decisions?

A list of risks, or a prioritized list of information assets, is not helpful in deciding whether to launch a new highly-automated product or open an office in Warsaw.

I welcome your thoughts.

If risk management is the answer, what is the question?

June 21, 2019 14 comments

This insightful point was made by Roger Estell on my blog last week.

It merits our thoughtful consideration.


Let’s start with some thoughts about the fundamentals underlying any successful enterprise, whether large or small.

Let’s assume that we are all working together to deliver success for the enterprise.

Then how is success measured?


The executive team, from CEO on down, is usually measured based on whether the organization has achieved targets (or metrics) approved by its owners (of their representatives on the board).

Rather than (as in the case of COSO ERM and ICF) assuming that those are the right metrics to measure success, I suggest considering:

  • Have the best objectives been set? Were all opportunities and potential hazards of significance considered during the objective (and strategy) setting process?
  • Have the right targets been set? Are they too low, so that the executives don’t stretch as much as they should; if they are too easily achieved, there is a temptation to store opportunities for the next period. If they are too high, management may take a level of risk (a potential for harm in this case) that is beyond what the owners consider acceptable.
  • Have performance targets and incentives been established throughout the organization that are consistent with the targets set for the enterprise as a whole? Does everybody understand what is needed from them for the organization to succeed? Are there performance metrics that will lead management (at any level) to act in a way that is inconsistent with enterprise goals?
  • Are objectives, strategies, and related metrics adjusted as necessary when conditions change?
  • In other words, is there a reasonable level of assurance that the right objectives (and strategies) are set to deliver optimal levels of shorter and longer-term success.

In a video, Alexei Sidorenko talks about how he worked with the management team to ensure that the objectives they set had a reasonable likelihood of success. He used scenario planning and other tools to help management understand that the first targets they set were unreasonable, with only a 1% (or less) likelihood of being achieved. The target was revised and the new one, approved by management and the board, had a projected 70% likelihood of being achieved.

Management and the board accepted that there was a 30% chance of failing to achieve their objective. (A far more reasonable and practical approach than the concept of risk appetite, as the latter only considers the downside and not the big picture of upside and downside.)

Alex used the tools and techniques he learned for risk management to help the organization set reasonable and appropriate objectives, targets, and metrics for success and the measurement of executive performance.

The question to be asked first is: how can we assess the likelihood of success (achievement of our objectives) given a reasonable understanding of what might happen.

The answer is not really ‘risk management’, because success is not achieved by managing downside risk. We want to manage for success rather than for avoiding failure.

The answer is the use of the tools and techniques traditionally only used for assessing and evaluating the downside – and you can call that risk management if you like. I don’t.


Once the objectives, strategies, metrics for measuring performance, and so on are set, management has to run the business to achieve them.

Management runs the business by making decisions. We hope they are informed and intelligent decisions: informed about what might happen that would affect their achievement, both for the better and for the worse.

How do they get the information about what might happen, both good and bad, on which they will base their decisions?

How will they determine whether their decision will improve or negatively affect the likelihood of achieving their objectives? In Alex’s case, will each decision they make increase the likelihood of success to above 70% or will that likelihood drop below acceptable levels?

Is the answer to those questions ‘risk management’? Certainly, the tools and techniques used to assess adverse events and situations, and their effect on objectives, can be used to paint the larger picture.

But I don’t think the answer is ‘risk management’.

It’s also not ‘objective management’.

It’s effective and intelligent management. It’s the ability to make informed and intelligent decisions, which is the core of effective management.


We need to stop coming up with new words and phrases when all we need to address is the effectiveness of management. So stop talking about ERM, IRM, or even objective assurance, and start thinking about how to obtain reasonable assurance that the management of the organization, including how it sets objectives and makes related execution decisions, is effective.


I welcome your thoughts.

Scratching the surface on Facebook and its problems

June 14, 2019 1 comment

Richard Chambers, President and CEO of the IIA, has shared a short piece that scratches the surface (IMHO) when it comes to the issues faced by Facebook and similar organizations. I am talking about organizations that want either to use or sell data.

Facebook Data Exposure Offers Critical Lesson for Internal Auditors makes some good points, including:

  • From an internal audit perspective, Facebook’s woes offer a clear and compelling lesson: Data, once viewed solely as an asset to be leveraged, now must be viewed as a potential liability or risk, as well.
  • Mining and analyzing data is a fundamental step in strategic business decisions. It helps businesses and organizations build models based on historical information to predict future behavior. But poor data management and a failure to understand what it tells us is a risk.
  • Internal auditors must cultivate and maintain a keen understanding of how their organizations collect, manage, protect, use, and share data. They also must have a handle on past and current practices on data usage and storage.
  • CAEs should speak candidly to boards and executive management on the value of assurance.

It is tempting to focus exclusively on the down (or dark) side of the story. But as Richard says, the use and even commercialization of data is a huge opportunity as well.

I suggest that organizations and their internal audit teams seek assurance regarding:

  • Compliance with applicable laws and regulations in every location. Initiatives and resources should be allocated based on an understanding of relative risk to the organization and its objectives.
  • Compliance with the expectations of the community, governments, and (especially) customers. Again, prioritization of effort should be risk-based.
  • The safety of information, not only within the organization’s internal systems but also when it is in the “cloud” or with a vendor/customer/partner.
  • Whether optimal benefit is being obtained from the data. Consider the internal use of available data to inform and drive business decisions as well as the opportunity to market information. With respect to the marketing of the information, consider the whole sales cycle and the need for assurance that buyers will comply not only with the terms of the contract but with applicable laws, regulations, and societal expectations.
  • The integrity of the data: completeness, accuracy, currency, and timeliness.
  • The validity of the strategic model for using and leveraging the model.

While the focus right now is on the dark side, many organizations can leverage their data far more than they do today.

Internal audit can point out opportunities as well as potential problems.

I welcome your thoughts.