How to assess the effectiveness of risk management
Internal auditors are expected, according to the IIA Standards and some governance codes, to assess the effectiveness of risk management.
That can be a challenge, especially as:
- There is no commonly accepted idea of what effective risk management is.
- While both the COSO ERM framework and the ISO 31000 standard provide principles for effective risk management, neither (in my opinion) is sufficient.
- Few organizations are seen as having effective risk management, so there is no exemplar against which to measure. (The majority of organizations manage the potential for failure, not the likelihood of success – the gold standard of what is commonly called risk management.)
My good friend, Alex Sidorenko has given this challenge a valiant try in his recent video. (I encourage you to follow him as he challenges traditional thinking – something we should all do.)
3 things to look for when auditing risk management identifies three areas to assess:
- Organizational performance compared to prior years, industry benchmarks, and so on
- How well the company makes decisions. Is risk information integrated with how decisions are made?
- Culture, including risk-related policies and procedures and attitudes towards risk
Taking each in turn, organization performance is a poor indicator of effectiveness. Many succeed simply by being lucky; others fail, despite excellent people and processes, when unfortunate and unforeseeable events occur.
How the company makes decisions is at the heart of effective risk management. But looking at minutes and other records of meetings where decisions are being made is not likely to be revealing. Best is to be present when the decisions are made, failing that follow the example of my friend Grant Purdy.
Grant is now retired, but he was a prominent risk practitioner and thought leader (including chairing the committee that developed the excellent Australia/New Zealand’s risk standard on which ISO 31000 is based). He then turned his hand to consulting. When he was hired to upgrade an organization’s risk management practices, he met with the senior executives. Instead of asking about risk management, he asked:
How do you make decisions?
The lesson here is that the individuals assessing ‘risk management’ should meet with decision-makers and ask that question. From there, they can move to questions like:
- How do you consider all the things that might happen and affect the results of your decision?
- When you consider the things that might happen, both positive and negative, how do you assess them? How do you weigh the good and bad together?
- How do you know the information you are using is complete and reliable? What is the likelihood of it being incomplete, inaccurate, out-of-date, or in some other way deficient?
- Who is involved in making the decision? Do all potentially affected parties participate?
- If there is a risk function, how does it help you make decisions? Is it worth the cost of the function? How could it help you more?
- Are you able to adapt with agility when things change? How will you know when there has been a change such that the decision or actions flowing from the decision need to be reconsidered?
- …and more
Alex’s third is really, in my mind, a continuation of the second. I would prefer to think about how the decision-makers know what risks the board and top management want them to take.
Let me suggest my own top three:
- Do decision-makers believe that there are reliable processes to support decision-making, including the availability of current, reasonably complete, and reliable information about what might happen under each of the options they are considering?
- Do decisions involve the weighing, in a disciplined way that allows them to be compared, both the upsides and downsides of each option?
- Do they believe the risk function (if there is one) is helping them set and then execute on strategy? Is it all it should be?
- Do the organization’s processes and practices provide reasonable assurance that there will be an acceptable likelihood of success (measured by the achievement of objectives)?
OK, there are four. I cannot cut any of them out, they are all so important.
Which set of three (or four) do you like more?
Do you have your own?
Leave a comment
Norman Marks on Governance, Risk Management, and Internal Audit 
I would bet quite a lot that if the senior management of the several large organizations for which I worked were asked those four questions they would all say “yes” and they would believe it. When I try and assess the effectiveness of risk management I ask two questions: does the organization have agreed risk criteria to use for evaluating risks and allocating resources because without that every persons’ viewpoint will be different, and what is the frequency and quality of conversations about risks at the board, executive and managerial levels.
Possibly, John. But could they answer the follow-up question of why they say that?
Highly unlikely and no two answers would be the same. Boards are still in the dark ages re risk management, unfortunately.
There are a few ways to evaluate risk management as required under Standard 2120 and provide a report to the Audit Committee, without conducting a traditional process audit:
1. Participate in ERM and subordinate meetings (e.g., IT risk meeting). Are the right people present? Are the materials conducive to good discussion? Do the leaders encourage a culture where risks are put on the table rather than hidden?
2. Operations: Do the strategic plans and budgets clearly show the linkage between the top objectives, key risks to achieving those objectives, and action plans?
3. Legal: Are the top regulatory criteria identified (e.g., GDPR articles) and linked to controls? Are these controls tested periodically by the 2nd line or IA for key laws?
4. Financial reporting: Most have a SOX program, which covers this area thoroughly.
My last report to the Audit Committee on this subject had two slides: A slide that explained the COSO risk categories, and what the risk management process is in each; and a two-column text slide with the left column about what we were doing well and the right column about what we can do better.
P.S. I suggest everyone read Peter Drucker’s article “The Theory of the Business” which gets at business model risk.
This is certainly another way. But it seems to provide assurance only on downside risks, and not the full range of uncertainty expressed in both COSO and ISO. I don’t find it sufficient myself. I would prefer considering the principles in COSO (which I don’t particularly like) or the 2009 version of ISO 31000
Agree organization performance is a poor indicator of effectiveness
Agree – How the company makes decisions is at the heart of effective risk management. But looking at minutes and other records of meetings where decisions are being made is not likely to be revealing.
My top three:
• Has the Board/Owner(s) provided unambiguous directions to the Chief Risk Executive (CRE) and are the directions and ultimate approval of the program memorialized? Board/Owner(s) have the ultimate say in the direction of the risk management program. It is important not only that the CRE has this approval in their back pocket, but this is where auditors gain their basis of review.
• Are all levels of the organization involved in the risk identification process? The CRE will typically have little to no professional support staff and has to depend on units for basic exposure identification; mid-management for initial input on exposure measurement; and senior executives for stratification of concern. Internal audit should have a seat at each step in this process in order to understand risk management in general and the organization’s program specifically.
• Are reporting intervals reasonable? Each exposure should be reviewed at least annually. Some exposures will need review more frequently (some like commodity prices daily in a range). Does the Risk Dashboard in either the purchased risk management information system or the in-house designed system respond in agreed ways to changes in exposure, and does the Dashboard lend itself to “What if” scenarios for potential organizational changes (adding by growth or acquisition or organizational operations changes).
I realize Alex Sidorenko and other of like minds do not support the above risk identification process, but waiting to do an analysis on a project basis as they support is a reactive process that generally will take more time than is allowed. The above is a proactive process which can respond in the least amount of time.
Greg, is this not still focused only on harms, managing failure instead of success?
Norman and others
Central to all that you say is the word risk. You each use it as if it has an unmistakable and common meaning. But we all know that it doesn’t. Reportedly, in ISO standards alone, the word is used to formally label over 40 different concepts. Then there are statutory definitions, common law meanings, and numerous meanings in the daily vernacular (think the TV weather announcer ‘there is a risk of rain tomorrow’). It is impossible to say that any of these meanings is right or wrong. Axiomatically, its lack of meaning infects and renders equally meaningless, all compound nouns and other expressions of which it acts as either noun or adjective. Starting with ‘risk management’. No wonder there is such disagreement to be found in these columns. As I have previously – and frequently – ask “if risk management is the answer, what was the question”?
While there are various definitions, that doesn’t mean this challenge is impossible. Start with what the organization needs and proceed from there.
Thanks for these perspectives.
Norman
Surely the discussion of any matter requires clarity as to the matter! Even if you want to propose, as you do above, that risk means ‘what the organisation needs’, clearly, others think it means something else which is why the word has no utility. One can’t rationally discuss anything unless everyone is on the same page as to the subject.
But that aside, most human discussion does not take place via labels so why is it that whatever it is that people chose to label risk, can’t be discussed in plain language?
Interestingly, you and others refer to ‘definitions’ for risk which implies that the word risk is the reference point to which it is only necessary to add a definition. By contrast, I see the word ‘risk’ as a label that is attached for convenience to an idea. But as I and others have noted, it is attached to so many ideas that it can have no utility.
I have sometimes used the following analogy: Those of us who recall learning about the periodic table of elements in school chemistry, may also recall that the first element in the table is both very light and highly reactive – the two properties that explain its position in the first column in the first row of the table. It does not get those properties because it is called Hydrogen (we know this if only because in, say German, that is not what it is called…it is called der Wasserstoff). Rather, the buoyancy and reactivity of this element are explained by its atomic structure – one proton, one electron and no neutrons. And as I have illustrated, it really doesn’t matter what label you give it. However, the label hydrogen (and in German, der Wasserstoff) is exclusively used to describe material comprising that atom – there is no confusion and so, unlike ‘risk’ that label is very useful and can be used coherently in discussion and the written word.
Roger, I did not define risk as what the company needs. I said that internal audit should assess what people call risk management by what the organization needs (when it comes to making decisions considering what might happen).
In fact, I started by saying there was no common understanding of what effective risk management is. So you start, as you suggested, by agreeing what it is you are assessing and how you will assess it. Rather than assessing compliance with policy or one of the frameworks/standards, I prefer to see what the company needs to be successful and go from there.
In fact, I wrote a book suggesting that we dispense of the word ‘risk’ and use words and phrases that make business sense. Have you seen Risk Management in Plain English?
Norman,
It seems to me that if you describe what you do as ‘risk management’, whether you mean the noun or the verb, then it cannot be ‘effective’ in any rational use of that term.
It is an oft-made claim that ‘risk management’ improves decision making and that this assertion justifies the considerable overheads and investments of time and energy involved. However, as Roger has pointed out above, how can this be so when the phrase and the word ‘risk itself have no settled meanings and the clumsy constructs and confected, ever-expanding jargon of the many versions out there only confuse rather than inform normal people (and even ‘experts’!).
Furthermore, (and to provide some background to the quote you attribute to me) for over 30 years or more, when organisations have asked me to review the effectiveness of their ‘risk management’ I have rarely found any evidence that past decisions have, in any material way, been improved by the output from any form of ‘risk management’ process.
Notwithstanding the confusion over what is ‘risk’ and what therefore is ‘risk management’, the fog of the complicated jargon and contrived constructs risk management practitioners use mean that any useful information they generate goes largely unused and unapplied by decision makers making real decisions.
This also makes a mockery of the other claim or virtue, that ‘risk management’ is ‘integrated’ in decision making. How can that possibly occur when’ risk management’ is described as an externality which must somehow be integrated into what each organisation does and explicitly requires them to change?
The inconvenient fact is that the normal outputs from the risk management process such as ‘risk registers’ have few practical uses by normal people in the real world, especially when they are compiled on some periodic basis to somehow reflect the general context at some point in time rather than relating to any specific decision, the opportunity (the correct definition of this word) to be exploited and its context including the assumptions being made.
However, never fear because I have discovered one excellent use for risk registers – especially if you keep small furry animals like rabbits or guinea pigs. I’ve found that once shredded, they made excellent bedding materials. Furthermore, the bedding, once soiled, is a wonderful base for a compost heap.
I’ll let you construct your own metaphors!
Jokes aside, it quite clear to me that in the few organisations in the world who actually claim to practice it, ‘risk management’ hinders rather than helps decision makers achieve sufficient certainty about the outcomes of their decisions. I’ve found that even where you can detect it has produced some modest improvement, this is at the cost of a disproportionate amount of organisational resources with many concomitant irritations and reduction in organisational agility.
I’m afraid the game is up – which is most unfortunate for those who make a living peddling this confected and unproven belief system and even more so for those who somehow attempt to ‘audit’ it. But then again, you’ve had a good run!
Well said, Grant. That is why I have focused in my posts and books on achieving success (objectives) and what is needed to make the quality decisions necessary for success. That is why I have suggested that you start by understanding what is necessary for the organization and then assess whether that happens.
It’s a shame the IIA don’t read your books Norman. Or the many other ‘professional bodies’ that peddle this rubbish.
Norman, your blog concentrates on risk management and decision making, what I would call ‘decision risks’. There are also ‘process risks’, that is risks which arise from operating the business to achieve its objectives. For example, if I own a small grocery store which keeps cash on the premises overnight, there is a risk that the money will be stolen and so I manage that risk by putting in a safe. The distinction is important from the internal audit point-of-view because process risks are mitigated by controls which can be ‘ticked’. The effectiveness of risks management can be judged on whether sufficient controls are in place to mitigate process risks down to a level at which objectives are likely to be achieved. I appreciate these risks are mostly managing the potential for failure but many organisations have gone out of business by failing to adequately mitigate them.
The auditing of the risk management of decision risks is more difficult, since there is not much that can be ticked. I disagree with your comment, ‘organization performance is a poor indicator of effectiveness. Many succeed simply by being lucky; others fail, despite excellent people and processes, when unfortunate and unforeseeable events occur’. Surely ‘luck’ is a successful seizing of opportunities and ‘unfortunate and unforeseeable events’ indicate a lack of contingency planning. OK, there’s always the ‘asteroid hits earth’ type of risk but I guess most organizations fail because they haven’t mitigated their downside risks. If I wish to judge the decision making of a credit control department I look at the bad debts. Too many indicates a failure to identify bad payers, too few indicates a need to increase sales by accepting customers with a higher risk profile.
All living creatures employ risk management. Those that the have the most effective survive and evolve; those that don’t get eaten.
(More discussion in my book ‘Internal Auditing – an introduction’ free from http://www.internalaudit.biz).
David, each of your process risks requires a decision. How do you decide how to address the risk of loss of cash?
Each of the decision risks are addressed by controls, including who is making the decision, what information they are using, the process they follow, and so on. I see no difference.
Agreed, but the decision can be assessed by the result, no safe/cash box/strong safe. Internal audit can ‘tick’ the safe and provide an opinion as to whether the control is sufficient to ensure the related objectives. Having bought the safe there is then the decision, ‘when to store the cash’. Internal audit can ask for defined procedures (e.g. at 6:00 pm) but cannot ensure they are always obeyed. That’s what can’t be ticked, all IA can do is check that cash hasn’t been stolen.
David, IA can check to see whether the decisions on how much cash to retain (vs frequent deposits during the day), how much to spend on a safe and security guards and cameras, and so on are consistent with the level of “risk’ desired by the company. They can see whether that level of risk makes good business sense, considering the cost of controls and the benefit they deliver. They can question how those holding the cash are hired and whether background checks are performed. They can see how often cash counts are made and what happens if there are differences. So many controls, so little time.
.
Norman, your title is, ‘How to assess the effectiveness of risk management’. I think your reply answers it. Risk management is internal controls. These can be tangible; how often cash counts are made, evidence of background checks). IA can check the existence of these controls and come to a decision about their effectiveness (as they always have). Or internal controls can be intangible: ‘how much cash to retain (vs frequent deposits during the day), how much to spend on a safe and security guards and cameras’. IA can check ‘whether that level of risk makes good business sense, considering the cost of controls and the benefit they deliver’. Based on their assessment of whether the level of risks are acceptable, IA can then deliver an opinion on the effectiveness of risk management (or better, whether the organisation will achieve its objectives based on the effectiveness of risk management).
Indirect variables can be used to measure the effectiveness of risk management. Well designed and operational controls can serve as proxy variables. But organisational performance seem to have a lower correlation with risk management efforts. I would think that performance is the result that has other factors affecting it including the choice of strategy.
Controls are a direct consequence of risk assessment process and therefore can be used as a useful approximation of effectiveness.
I agree, but how do you assess whether the controls are effective? By seeing whether they provide reasonable assurance that the right risks are taken in decision-making, so that the organization succeeds. Before assessing operating effectiveness, it is necessary to assess the design and whether they meet the needs of the organization.
Norman, agreed that the design of internal controls should be assessed before their operating effectiveness. The assessment of design may involve decisions (as in the case of credit control) or not (as in the case of automatic sprinklers – though a decision was made about where to install them). So I think we are in general agreement about decision making. My main point is that internal audits frequently do not consider the decision-making training and environment but concentrate on the ‘tickable’ controls. Hence,
internal audit will answer your question 4 but do they ask questions one to three?
Your distinction of process risk from decision making compares the same thing at different levels. Decisions are at the heart of business activity. Business must decide on all things and all related decision s must be aligned. Shareholders decide how much value they want to make from an investment. They decide which business activity. They appoint a board to provide oversight on the investment.
The board decide to the best strategy that meets shareholders value expectations. Senior Management role is to make decisions that operationalize the strategy and provide regular comparison of actual results against expected results. Management must decide whether the business is on course.
Tactical management and operational staff similarly make decisions which when integrated upwards must lead to maximization of share holder value.
My contribution is that decisions are at the heart of business.
Sitting tenants in some businesses also get rewarded with high performance when their industry as whole or particular individual circumstances become very profitable by virtue of incumbency more than good decision making. Therefore, Norman position is what is good for the business is the ultimate criteria for effectiveness of risk management.
Risk is the cost incurred in achieving the expected value. Risk therefore reduces expected value to actual value or enhances actual value beyond the expected value. The effectiveness of risk management depends on how many decisions that are made that minimize the cost of risk or maximizes the rewards of risk management.
Process risk or operational risk is at mostly low levels in the organization. But does exist and is sometimes synonymous with model risk at higher levels.
N