Testing data vs. testing controls

In a recent post of his on LinkedIn, Joseph Kassapis wrote:

I was reading a typically excellent blog/post of Norman Marks on Control Testing (in the context of commenting on 2 reports on SOX Controls Testing), and was struck and intrigued by his insistence/emphasis on testing “Data” in the mistaken impression that this amounts to testing the Control(‘s effectiveness). He named this twice in his post as a fallacy/defect in the reports, and it instantly caught my attention, being something I always found extremely interesting and important: to what extent correct output can be taken to mean/evidence correct mechanism.

…

External Audit standards, as I fairly confidently recall/understand, expressly preclude this position, i.e. state that the correctness of the recorded transactions, as regards their aspects controlled by the control, can in no way and under no circumstances be taken as evidence of soundness/effectiveness of the control; and I sort of ‘resented’ this, regretted it, wished it was not there; without actually being able to really/genuinely fault it, logically; rather minding its being inconvenient, making things harder, depriving us of easy tests and forcing us to conceive harder ones, (towards the already very hard task/goal of attaining satisifaction of effective functioning of Control), easier said than done !

…

Nobody else seems to, elaborate either, on this very important principle. Nobody seems to take it up. Except, it seems, Norman Marks. In the sense that at least he does consider it is there, it is important, and it is grossly abused. I was badly hoping he would go on to elaborate, in this blog pot, but he didn’t.

I don’t know if he elaborated elsewhere. He can inform/refer us. Whether or not he did, in the past, I would dare invite/provoke/challenge him to do so now. With another, dedicated post. Enlightening us. As he always does.

OK, Joseph. Here we go.

I start with a premise: our objective is to obtain reasonable assurance that the controls relied upon to manage the risk (whether SOX and ICFR, or some other business risk) are (a) adequately designed and (b) operating effectively as designed.

In other words, we are performing an audit of the system of internal controls for that risk.

The situation is different if we are trying to validate that the data (or information, such as in a report) is complete and accurate.

The value of an opinion on the system of internal control is that it provides continuing assurance, while validating the data provides point in time assurance. Validating the data or the information in a report may confirm that that instance of the report is complete and accurate, but it doesn’t tell you that the next instances will be. For that, you either have to continue testing and validating each instance or rely on the system of internal controls.

The quality of assurance is different. An opinion on the system of internal controls only provides reasonable assurance that each instance is complete and accurate, whilst validating data provides more absolute assurance that the data is correct.

Now, let’s return to the challenge.

I have been leading a SOX Masters class for many years, usually multiple times each year. In that class, I ask participants:

“Has your home been burglarized in the last five years or so?

In all that time, only one person raised their hand. (Good news.)

I then ask:

“Does that prove you always closed and locked your doors and windows every time you left home?”

(I don’t even go so far as to ask whether they set the alarm.)

They smile ruefully, very much aware that they have failed to do so: their controls were not operating effectively, yet they did not have an incident (or data exception, if you like).

Consultants are pushing the notion that you can use analytics and other methods like AI and RPA to test controls.

There are very few opportunities to do so, as these techniques may provide some level of assurance that the data is free of error (if not always omissions). But they rarely provide acceptable evidence that the controls management have in place even exist, let alone are adequately designed and operating effectively.

Taking another example.

The city of San Jose, my hometown, has implemented a number of controls to limit accidents at busy intersections. They include:

• Traffic lights
• Lane and other street markings
• Periodic police visits
• Reliance on controls performed by others, such as DMV’s driver licensing controls

If you ran analytics and found that there were no accidents reported at the intersection of Stevens Creek Boulevard and Winchester Boulevard in 2022, does that prove that any of the controls were working?

No. I can tell you that there were times when the lights did not work but drivers exercised appropriate caution.

While detecting that there were incidents may indicate that controls were not working (more work needs to be done to confirm that), the lack of exceptions does not provide assurance that controls were in place, adequately designed, or operating effectively.

I hope that helps.

By the way, the intersection example illustrates another issue that many don’t understand.

The system of internal control only provides reasonable assurance. It does not provide absolute or perfect assurance.

COSO’s Internal Control Framework provides some examples of the limitations, but there is more.

When you test internal controls, you may find exceptions.

For example, you inspect the traffic lights and find that they were inoperative for a few hours on one day.

If that only happened once over a period of a year, I would call that an “isolated incident”. It is reasonable to accept the occasional breakdown.

But if it happened several times in a month, I would call it a “control breakdown”.

You can have effective internal controls despite isolated incidents, but not when there have been control breakdowns.

That is why when we find exceptions we need to expand the sample size to determine whether we have an isolated incident, which would acceptable, or a control breakdown – when we would assess that the control has failed to operate effectively as designed.

I welcome your comments.

Advertisement

Where do our SOX programs stand today? Two reports

Two firms recently released reports on SOX Compliance trends: Protiviti and Deloitte.

I need to make one important point.

When I was responsible for SOX at my company, I wanted to find out what our internal SOX compliance costs were. To my surprise, more than 50% of the costs were incurred by management: supporting testing by both internal and external audit teams, maintaining the documentation, answering questions, and helping with the scoping.

The surveys on cost performed by firms like these two tend to ignore the management-related costs. Keep that in the back of your mind as we review the two reports.

Protiviti shared the results of their annual SOX surveys in Assessing SOX internal costs, hours, controls and other trends in the results of Protiviti’s 2022 Sarbanes-Oxley Compliance Survey. It has a great deal of information and is worth downloading and reading.

Protiviti’s Executive Summary includes this (with my highlights):

Escalating compliance costs, time and efforts have a silver lining: They are driving more investments in automation and technology tools that generate greater efficiencies — and potentially cost savings as well as effectiveness and coverage benefits — into the SOX compliance process. Our data indicates that technology tools currently support an average of one-fourth of SOX compliance work across all companies, and a majority of programs deploy audit management and/or GRC platforms. These results are promising: Greater use of enabling technologies can, over time, help moderate jumps in internal SOX compliance costs. That said, more progress is needed. Many programs have yet to begin using an audit management platform while most have yet to leverage more advanced technology tools in their SOX programs.

There also are opportunities to pursue procedural and structural changes in SOX compliance programs. Shared services or “centers of excellence” approaches — managed internally or by an external outsourcing partner — offer substantial opportunities for efficiency improvements, especially when it comes to the highly defined and repeatable tasks, such as gathering and organizing evidence, and control testing, that dominate SOX compliance efforts. Many of the forces driving internal SOX compliance costs and hours higher are, for the most part, beyond the control of companies. This is not the case with investments in compliance automation and broader technology enablement as well as alternative delivery models that generate greater efficiency over the long term. Internal audit and finance leaders, together with their C-suite colleagues, should avoid delaying their evaluation and pursuit of opportunities in these areas.

I have highlighted two sections:

1. While technology can provide useful functionalities in managing a SOX compliance program, the ROI for what can be expensive software is not always clear for companies without hundreds of key controls. In addition, my experience with some of the software is that it doesn’t always support the top-down and risk-based approach explained in PCAOB and SEC guidance; it doesn’t identify significant accounts and then the key controls relied upon to prevent or detect potential material errors of omissions in those accounts.

The consulting firms preach that you can use technology for testing. However, the potential is not nearly as great as they indicate. We need to perform testing that provides reasonable assurance of the existence, design, and operation of the key controls we rely on. Most of the software tests the data, not the controls – and just because the data is clear you cannot assume that the controls are in place, adequately designed, and consistently operating as they should.

Protiviti says this later on, which is highly questionable:

Automation platforms and applications bring greater efficiency to SOX compliance activities. The deployment of process mining, advanced analytics, robotic process automation (RPA) and continuous monitoring, along with other advanced technological tools, can significantly reduce the volume of manual compliance tasks as well as retention risks associated with subjecting internal full-time staff to heavy loads of repetitive, task-driven work.

2. These “shared service centers” for SOX testing, if outsourced, are a return to the use of expensive consulting firms for testing – not something I recommend. If they are run in-house, staffed by people who do nothing else, then they may not be in tune with the business. I would think twice (or more) before doing this. There is huge value in a SOX team that suggests better controls and process improvements in addition to testing key controls.

Protiviti tells us in the report that, on average, 41% of SOX internal costs is for outsourced resources.

On the other hand, this is correct:

A combination of internal and external factors creating volatility — technology-driven transformation and innovation, talent shortages, strategic pivots and more — is contributing to rising SOX compliance costs. More companies spend $2 million or more on compliance while fewer spend $500,000 or less. A surge in the number of smaller companies spending $2 million or more in SOX compliance costs likely reflects last year’s significant increase in initial public offerings (IPOs), driven by special purpose acquisition companies (SPACs).

The chart on page 12 of the report is very useful information. It shows the typical time taken for various activities, such as testing for operational effectiveness or adequate design of a key control. Unfortunately, Protiviti did not distinguish between manual and automated controls.

The results in one chart disappointed me: the percentage of controls where the external auditors relied on management testing. The average was just 26% and only 10% of respondents said external auditors’ reliance exceeded 50%.

Protiviti tells us:

In assessing year-over-year trends in external auditor reliance on management controls testing, percentages show a year-over-year decline — i.e., external auditors appear to be relying less on this testing.

Two points:

1. At my company, EY told the audit committee they relied on my team for 80%. At the SOX Masters training I lead, a number of attendees have reported similar levels of reliance.
2. It is important to recognize that the external auditors can rely entirely (with review) on management’s testing of key controls that are not high risk, but they can also reduce their work by placing partial reliance with limited reperformance.

I found it interesting that according to the survey, in the average company 50” of the key controls are automated, up from 33%.

I also found it interesting that the average company has 52 significant applications, and more than half of them are cloud applications. That seems too high.

I wonder whether they have done a good job in using the top-down and risk-based approach to identify significant applications, or whether they have included applications that are involved in financial reporting but don’t contain any automated controls or other IT-dependent controls.

I am also surprised that many companies either test key reports (IPE) on a rotational basis (which should not be allowed) or only once and then not until the report is changed – 21% rotational and 36% just once. That conflicts with my empirical experience with the number of companies who have employed a baselining or benchmarking approach.

As a reminder, except when benchmarking is used for IT-dependent controls, every SOX year has to stand on its own.

Let me make one important statement:

The best path to reducing SOX compliance costs and improving effectiveness is through application (and re-application every year) of the top-down and risk-based approach. Right-size your controls!

The Deloitte report is SOX modernization: Optimizing compliance while extracting value.

They seem to agree with my important statement, above, when they say:

A SOX program that has not been challenged in years may be stale, which could be a drain on resources and impede performance, particularly if this compliance program is treated more like a “check-the-box” activity.

Deloitte also comments, with my highlights:

Management’s responsibilities related to internal control over financial reporting is to obtain reasonable assurance over the reliability of financial reporting, not absolute assurance, and the concept of “reasonableness” is objective with a range of judgments and methodologies that could be considered appropriate. Performing an effective risk assessment can help management identify areas with risks of material misstatement within the company and determine which of those areas it should focus its efforts.

Many factors could contribute to a lagging SOX program. Over time, risks evolve, or new risks are identified, and the response may have been to design new controls without always taking into consideration if any existing controls should be modified or removed. Additionally, once risks are identified, the level of risk may not be considered, such as if it’s a lower risk or a significant risk, which could result in not spending enough time in areas of significant risk or spending too much time in areas of lower risk. Controls could also have been added to manage an issue or deficiency identified without actually addressing the root cause.

Deloitte goes on to provide good advice on the risk assessment process.

But they fail miserably by recommending testing data instead of controls:

Automated testing consists of profiling certain populations and transactions with real-time results, allowing a company to be able to test up to 100 percent of the population and potentially achieve more assurance for less time and cost.

As a reminder: the data can be 100% clean even though nobody is performing the controls. Just think about how many times you left your windows open and/or doors unlocked when you left home, and even though those controls were not operating you were not burglarized.

Deloitte makes one good point, but they don’t go far enough.

They talk about automating a current manual process. That can certainly provide both efficiency and effectiveness.

But why not go further and consider whether the process should be changed – with or without modernization. There’s little point in automating an inefficient process!

If you are responsible for your company’s SOX program, I urge you to consider my SOX Masters class (one is planned for September). You can also purchase the IIA’s Management Guide to Sarbanes-Oxley Section 404.

I welcome your comments and experiences.

If you are involved in SOX compliance, you should know about the IIA’s GAIT Methodology

A fact: most companies have included far too many IT General Controls (ITGC) in their scope for SOX.

Why: because they have taken an approach to scoping ITGC that is disconnected from the top-down and risk-based approach used to identify key controls within business processes. The scoping of ITGC has resulted in including ITGC controls in scope where a failure would not present a reasonable possibility of a material error omission in the financial statements.

“The identification of risks and controls within IT should not be a separate evaluation. Instead, it should be an integral part of management’s top-down, risk-based approach to identifying risks and controls and in determining evidential matter necessary to support the assessment.” – SEC Interpretive Guidance

The IIA recognized that there was a need to help practitioners define the right scope of ITGC for SOX, and a team of experts (including a representative from the PCAOB) developed the GAIT Methodology.

GAIT continues the top-down and risk-based approach recommended for companies by the SEC and mandated for their auditors in the PCAOB’s Auditing Standard 2201 (formerly AS5).

“The auditor should use a top-down approach to the audit of internal control over financial reporting to select the controls to test.” – PCAOB Auditing Standard 2201

“Management should identify those risks of misstatement that could, individually or in combination with others, result in a material misstatement of the financial statements (financial reporting risks).” – SEC Interpretive guidance

“In an audit of internal control, if the auditor selects an IT-dependent control for testing, the auditor should test the IT-dependent controls and the IT controls on which the selected control relies to support a conclusion about whether those controls address the risks of material misstatement.” – PCAOB Staff Alert No. 11

“For purposes of the evaluation of ICFR, management only needs to evaluate those IT general controls that are necessary for the proper and consistent operation of other controls designed to adequately address financial reporting risks.” – SEC Interpretive Guidance

Since its publication in 2007, GAIT has been adopted with great success by hundreds of companies and accepted (even recommended) by their CPA firms.

It has helped those organizations right-size their ITGC scope for SOX. Although it is focused on getting the scope right, rather than on cutting unnecessary ITGC out of their SOX scope, companies have been able to reduce the number of ITGC key controls significantly.

15 years have passed since GAIT was published. During that time, technology has advanced and practitioners have gained far more experience in SOX compliance.

It was time to update GAIT.

That update has now been completed (with the help of an eminent review panel of practitioners and partners from independent audit and consulting firms) and the product is available for free download by visiting a dedicated page on this website.

GAIT has stood the test of time very well! This is not surprising as it continues to be used extensively.

Its principles and methods continue to apply, even as technology and its use have changed.

The updated version of GAIT, developed independently from the IIA but with their full knowledge, simplifies the text, adds real-life examples, and references relevant regulatory guidance. The IIA is focused on an update to their International Professional Practices Framework and was not able to lead or participate in the update, but it is expected they will turn to their own update in 2023.

The dedicated web page includes links to the original GAIT Methodology, as well as to the two GAIT products that followed: for general technology-related business risk (GAIT-R), and for the assessment of ITGC deficiencies for SOX.

Comments and feedback are welcome.

Trends in SOX Material Weaknesses

Last week, I shared a post about the firm of Audit Analytics’ report on 2021 financial restatements.

Today, I am going to cover their report, SOX 404 Disclosures: A Seventeen-Year Review. Admittedly, it analyzes 2020 filings, but I would expect that the results would be similar now.

Their report has some interesting news, notably that the number of adverse assessments of internal control over financial reporting (ICFR) decreased in 2020 despite the pandemic.

The percentage of adverse ICFR management reports and auditor attestations decreased in 2020, despite the impact of the COVID-19 pandemic throughout 2020 that necessitated changes to internal controls. The COVID19 pandemic occurring throughout 2020 had particular effects on public companies and their internal control structure and environment.

Some companies with existing control deficiencies disclosed difficulty remediating those weaknesses due to pandemic circumstances. Furthermore, rapid changes to the control environment were required in order for many companies to continue operating, including the need to reduce personnel to comply with pandemic restrictions or conserve cash. A reduced workforce can result in issues in the control environment related to segregation of duties and maintaining appropriate accounting personnel. Additionally, many companies increased reliance on information technology to accommodate a remote workforce, an area of controls ripe for deficiencies.

They also said that there was no significant change in the areas where material weaknesses were found.

Despite the unprecedented nature of the pandemic, little effect was noted in terms of the most common issues disclosed in adverse SOX 404 assessments. For example, the top two internal control issues cited in adverse ICFR management reports in 2020 – issues related to accounting personnel and segregation of duties – have been the top two issues for the previous five years. This illustrates that issues related to personnel are always common for smaller companies, regardless of circumstances arising from an event, such as the pandemic, that could significantly exacerbate existing deficiencies.

One ‘finding’ in the report astonished me.

The report says that the external auditors cited different material weaknesses than management.

• In adverse ICFR auditor attestations for the fiscal year 2020, the most common internal control issue that led to the conclusion that ICFR was ineffective was the need to make year-end adjustments (51%). The second most common reason expressed by auditors was a need for more highly trained accounting personnel (42%). These internal control issues are common, appearing as the top two issues in each of the last five years.
• In adverse ICFR management reports for the fiscal year 2020, the most common internal control issue that led to the conclusion that ICFR was ineffective was a need for more highly trained accounting personnel (75%). The second most common reason was related to segregation of duty issues associated with the design and use of personnel within an organization (63%). These internal control issues are commonly cited in management reports, appearing as the top two issues in each of the last five years.

This makes no sense to me for several reasons.

First and foremost, I find it hard to believe that they couldn’t agree on material weaknesses. If the audit firm said something was a material weakness, it would be next to impossible for management (and the audit committee) to refuse to identify it as such in their report. Similarly, I can’t see the audit firm passing up the opportunity to report something management said was a material weakness.

I am also surprised that the auditors thought having a lot of year-end adjustments reflecting an ineffective system of ICFR. The only explanation I have is that they related to errors during the year that were material to one or more quarters and corrected at year-end – and that should have been the disclosure. The problem with that is that the system of ICFR at the end of the year would probably have been effective! (Management also identified this area in 21% of their adverse assessments.)

The report lists other areas where material weaknesses were identified, either by the audit firm or by management.

• The audit firms identified issues related to IT in 36% of their adverse opinions.
• Both the audit firms and management identified inadequate disclosure controls (21% of the adverse audit attestations, and 25% of management’s). But disclosure controls (the subject of s302 of the Act) are not subject to s404 opinions. This makes no sense to me.
• Management identified an insufficient audit committee as a material weakness in 21% of their reports. It is hard to see how this can be correct. While it is one of the Principles in the COSO Internal Control Framework, defects in the audit committee are highly unlikely to result in a material error or omission in the financial statements.

The report has some more useful information. Again, they contrasted the reports of the audit firms to those of management.

• In adverse ICFR auditor attestations for the fiscal year 2020, the most common accounting issue that led to the conclusion that ICFR was not effective concerned revenue recognition. The second most common reason expressed by auditors was related to taxes. Taxes were the number one issue in 2016 but were less common between 2017-2019. Accounting issues related to PPE, intangible or fixed assets jumped in rank from eighth in 2019 to 2020. In a bigger jump, accounting issues related to the recording of debt and warrants identified in adverse ICFR auditor attestations went from being far outside the top five issues in the last five years to being the sixth most common issue in 2020.
• In adverse ICFR management reports for the fiscal year 2020, the most common accounting issue that led to the conclusion that ICFR was ineffective concerned the recording of debt/warrants/securities. This issue ranked fourth in 2015, but historically, the recording of debt and warrants was not a prevalent accounting issue cited in management reports with adverse ICFR.

I am drawn to conclude that people are having difficulties in this area. I strongly suspect that some auditors and some management teams are not testing their identification of material weaknesses against the definition of “a deficiency, or a combination of deficiencies, in internal control over financial reporting, such that there is a reasonable possibility that a material misstatement of the company’s annual or interim financial statements will not be prevented or detected on a timely basis.”

I also suspect that the audit committees of these companies are not challenging management and the audit firms to report the same and not different material weaknesses.

Finally, too many seem to be failing to assess and report on the state of ICFR at the end of the year, which is the requirement.

Reports like these are useful information to all involved in SOX. We should pay attention and makes sure we have the right top-down and risk-based scope, and test any deficiencies against the definition of a material weakness.

I welcome your thoughts.

Where do material errors occur in the financial statements?

Every so often, the firm of Audit Analytics shares a report with information of interest.

Their latest is 2021 Financial Restatements: A Twenty-One-Year Review. They introduce it:

In this report, we cover twenty-one years of trends in financial restatements – including a closer look at the effect of SPACs on recent trends in financial restatements. We also cover materiality, impacts, severity measures, size and location, and the top accounting issues.

If you are involved in preparing or auditing financial statements or the system of internal control over financial reporting (e.g., for SOX compliance), or on the audit committee of the board, you should read the report.

The big news is that the great majority of restatements were due to issues around the use and accounting for special purpose acquisition companies (SPACs). As the report says:

On April 12, 2021, the SEC’s Acting Director of the Division of Corporation Finance John Coates and Acting Chief Accountant Paul Munter issued a joint statement urging companies with warrants issued by Special Purpose Acquisition Companies (SPACs) to reconsider the accounting treatment of those warrants.3 In November, hundreds of SPACs reclassified redeemable shares from permanent equity to temporary equity.

The SEC’s guidance on accounting for redeemable shares and warrant liabilities resulted in significant increases observed in both the number of restatements filed and the number of companies that disclosed a restatement during 2021. Additionally, the composition of restating companies was altered from previous years, as these two accounting issues had a broad impact on a narrow population: SPACs and companies acquired by SPACs.

As a direct result, the positive trend in the number of restatements was interrupted in 2021. If you exclude SPAC-related restatements, the numbers continued to decline.

If you exclude SPAC-related restatements, the more significant accounting issues were:

• Debt and equity securities – 19.1%
• Revenue recognition – 12%
• Liabilities and accruals – 11.7%
• Expenses – 10.9%
• Taxes – 8.8%
• Cash flows – 7.3%
• Share-based compensation – 7%
• Acquisitions and divestitures – 7%
• Inventories – 6.7%
• Asset valuations – 6.5%

I find it interesting that a hot button in previous years, Revenue Recognition, has dropped from being the reason behind more than 20% of restatements in 2004, around 16% in 2018, to 12% last year.

What does all of this mean for those of us involved in preparing or auditing financial statements and related controls?

I believe this should be factored into your risk assessment activities.

The audit committee might include a discussion of this report with their external auditors.

I welcome your thoughts.

Where should internal audit report?

This is a touchy subject.

While there is very little debate that the head of internal audit, the chief audit executive or CAE, should report functionally to the board (usually the audit committee of the board), there are some strong opinions on whether it should report for administrative purposes.

This is what the IIA’s Standards have to say (with my emphasis):

1110 – Organizational Independence    

The chief audit executive must report to a level within the organization that allows the internal audit activity to fulfill its responsibilities. The chief audit executive must confirm to the board, at least annually, the organizational independence of the internal audit activity.

Interpretation:

Organizational independence is effectively achieved when the chief audit executive reports functionally to the board. Examples of functional reporting to the board involve the board:

• Approving the internal audit charter.
• Approving the risk-based internal audit plan.
• Approving the internal audit budget and resource plan.
• Receiving communications from the chief audit executive on the internal audit activity’s performance relative to its plan and other matters.
• Approving decisions regarding the appointment and removal of the chief audit executive.  Approving the remuneration of the chief audit executive.
• Making appropriate inquiries of management and the chief audit executive to determine whether there are inappropriate scope or resource limitations.

1110.A1 – The internal audit activity must be free from interference in determining the scope of internal auditing, performing work, and communicating results. The chief audit executive must disclose such interference to the board and discuss the implications.

The Standards do not discuss what is included in administrative reporting. This is what I believe is included:

• Reviewing and approving the expenses of the CAE
• Performing other administrative functions that may be required by organizational policy. These vary from organization to organization but may include the approval of purchase orders that exceed the CAE’s authority level, approval of travel, and so on.

There’s little else that I can think of today.

It is customary for the CAE to be able to attend the executive’s direct reports.

It is also customary, but not always a given, that the executive will be a supporter and champion of internal audit.

The CAE’s cost center may or may not roll up to that of the executive.

X

Somebody has to perform these administrative functions, and it is unrealistic (with rare exceptions) to expect the chair of the audit committee to do them.

The debate is whether the CAE should report administratively to the CEO, the CFO, or another senior executive.

While it is possible for the CAE to report for administrative purposes at a lower level, for example to the Corporate Controller, this will generally create a perception that the CAE is middle management at best – rather than the senior executive he or she really is (or should be).

X

Some years ago, the IIA stated its preference (my guess is that this was influenced by its CEO) that the administrative reporting should be to the CEO.

Richard Chambers repeated his strong preference for that in a recent post, New Surveys Raise Alarm Bells for Internal Audit. He tells us:

One of the most jaw-dropping statistics in the IIA’s recent 2022 North American Pulse of Internal Audit report is that 76% of CAEs at publicly traded companies say they work administratively for the CFO! I have never been shy about sharing my views on this reporting relationship. While many CFOs fully respect the need for internal audit to remain independent, and for internal auditors to be objective, the optics indicate that CFOs who “own” internal audit are more likely to use the function to focus on their own priorities. Even more alarming is that only 4% of respondents are concerned about reporting lines. That is, by and large, a uniquely American problem, and fortunately it isn’t widespread in either the public or not-for-profit sectors. But the number of internal audit functions reporting to the CEO in publicly traded companies appears to be retreating. That is not a good development.

He has strong views on this and so do I.

It could be that his many years as CAE in government service influenced his position. My many years as CAE in US and global corporations led me to a totally different position.

First, administrative reporting does not confer, in any way, “ownership” of internal audit.

Second, I have seen CAEs who report administratively to the CEO forced to work on special projects for the CEO, even to the point of being sent to fire non-performing executives! In other words, the CEO thought he owned internal audit.

Third, the CEO is a busy individual and asking him or her to spend their valuable time on administrative duties like approving expense reports is absurd. In practice, the CEO will delegate those responsibilities to the CFO (at best) or an assistant (at worst, but more likely).

Fourth, you can report to the CFO and have free access to the CEO.

Fifth and extremely important, you are far more likely to be included in the CFO’s executive staff meetings than the CEO’s, even if you report administratively to the CEO. In fact, reporting to the CEO may make it harder to attend the CFO’s meetings. These meetings are very valuable sources of information about the strategies and activities of the organization.

Finally, the fact that 96% of CAEs are content with their administrative reporting should tell us something. These are smart people, and their opinion should be respected as being based on reality. Reporting to the CFO satisfies the intent of Standard 1110: “The chief audit executive must report to a level within the organization that allows the internal audit activity to fulfill its responsibilities.”

X

Should the CAE report administratively to another senior executive?

This will depend on the organization and on the individual executive.

I can see a case being made for reporting to one of these people:

• Chief Administrative Officer
• Chief Operating Officer
• General Counsel

I am not a fan of the CAE reporting to a specialist CRO with whom there may be conflict over the assessment of control deficiencies and the risk they represent.

X

Whoever the CAE reports to administratively must respect the fact that the reporting is purely administrative, they do not own internal audit, and their role is limited.

X

How does the CAE make this happen?

That is covered by Standard 1000: Purpose, Authority, and Responsibility.

The purpose, authority, and responsibility of the internal audit activity must be formally defined in an internal audit charter, consistent with the Mission of Internal Audit and the mandatory elements of the International Professional Practices Framework (the Core Principles for the Professional Practice of Internal Auditing, the Code of Ethics, the Standards, and the Definition of Internal Auditing). The chief audit executive must periodically review the internal audit charter and present it to senior management and the board for approval.

Interpretation:

The internal audit charter is a formal document that defines the internal audit activity’s purpose, authority, and responsibility. The internal audit charter establishes the internal audit activity’s position within the organization, including the nature of the chief audit executive’s functional reporting relationship with the board; authorizes access to records, personnel, and physical properties relevant to the performance of engagements; and defines the scope of internal audit activities. Final approval of the internal audit charter resides with the board.

The value of the Charter is not that the CAE can brandish its authority when management doesn’t allow internal audit necessary access to information, etc.

The value is that it is discussed and reviewed by the board or its audit committee. That activity instructs whoever is administratively supporting the CAE where the boundaries of their role lie.

X

What do you think?

X

By the way, I am not commenting today on the other alarm bells that Richard says are ringing except to say that I disagree on SOX and do not agree with his logic on cyber. (I would point you to an IIA webinar we did together, but the IIA has removed it for some reason. In it, he agreed with my position that IA delivers great value if it is given the necessary resources to fulfil its primary mission as well as test controls for SOX.)

Using technology for SOX compliance

There is good guidance on how technology can help an organization address SOX compliance needs, but there is also poor guidance.

Protiviti has shared both over the years. Their latest, Using Technology to Comply With Sarbanes-Oxley: Examining the Latest Trends, falls more in the latter category.

X

The most important error made by the author is to ignore the difference between (a) designing and operating a system of internal control over financial reporting (ICFR) and (b) evaluating and testing it.

X

Technology can be of great value when it comes to implementing controls that are both efficient and effective in addressing ICFR risks.

In my SOX training programs. I share a story about how I eliminated hundreds of detailed HR and payroll key controls, replacing them with three detective controls that used analytics to support a flux review of payroll expenses.

This is where technology can be best deployed for advantage, through analytics and related tools (like RPA and ML) used in detective controls.

When it comes to SOX, reliance can just as well be placed on detective as on preventive controls. (Other business risks may be better served with preventive controls or a combination of preventive and detective.)

X

But caution must be used in using that same technology (analytics, RPA, and ML) in evaluating and testing controls.

Remember that the purpose of the testing is to confirm the design and operation of the controls. Verifying that the data is sound provides little assurance that controls over the data are in place. At best, analytics that detects errors in the data is evidence that the controls may be deficient.

I love to ask in my training sessions how many participants have had their homes burglarized in the last year or two. (Only one person over the many years has raised their hand.) I then ask whether that proves that they always shut and locked the front door every time they left home.

X

Technology can be of value in certain circumstances, such as:

• Helping to manage the overall SOX compliance program. At my companies, I used software designed for this purpose.
• Mining data such as configuration settings (as discussed in the paper) for validation. However, care has to be taken to ensure that this provides assurance over key controls.

X

One of the other issues I have with the Protiviti paper is the reference to so-called “GRC solutions”.

This is a trap!

Rather than looking for and evaluating “GRC solutions”, identify your business needs and select the software that will help you achieve them.

The best solution for your needs is often not a “GRC solution” that has a broad (and often highly valuable) set of functionalities. It can easily be a specialized technology.

For example, you may want to deploy advanced analytics technology as detective controls, and this is not usually considered a “GRC solution”. The software designed to identify access control problems may or may not be part of a broad “GRC” product.

(Note: purchase of a GRC solution may well be justified based on its ability to satisfy multiple business needs, including assisting in managing risk and compliance programs. But, I would probably not get one just for SOX.)

X

Finally, the author has confused SOX compliance and the auditing of other business risks. Issues like duplicate payments, failure to take discounts, and so on are rarely if ever sufficiently material to be included in scope.

X

The author, like many consultants these days (including the major CPA firms), is in love with technology and pushing organizations, and their internal auditors in particular, to buy the latest hammer. The problem is that these organizations then look everywhere for a nail to hit – when all they can see are screws.

I love technology as well. But define your needs and make sure any purchase is justified on business grounds.

I welcome your thoughts.

SOX and the COSO Principles

One of the requirements for the SOX compliance program is that the assessment is based on a recognized internal control framework. In practice, this is (almost) always the 2013 COSO Internal Control Framework.

COSO says that a system of internal control is effective if it “provides reasonable assurance regarding the achievement of an entity’s objectives. An effective system of internal control reduces, to an acceptable level, the risk of not achieving an entity objective and may relate to one, two, or all three categories of objectives.”

However, it goes on to say that for a system of internal control to be considered effective, all relevant principles must be “present and functioning”.

COSO says that they can be considered “present and functioning” if there are no related “major deficiencies” that would prevent there being reasonable assurance of achieving the objective(s); for SOX, this equates to having no related material weaknesses.

When the 2013 update was released, I said that this meant three things:

1. It is necessary to confirm which of the COSO principles are relevant to the assessment.
2. The way to confirm that they are present and functioning is by indicating which key controls are relied upon for that purpose and confirming that they are adequately designed and operating effectively.
3. If there was a failure in a control relied upon for the presence and functioning of a principle, that failure could not be a material weakness. In other words, a principle can be considered present and functioning even if there are failures of related controls as long as those failures do not mean there is at least a reasonable possibility of a material error or omission in the filed financial statements.

XX

It is nearly eight years since that update when I suggested that one of more of the COSO principles might not be relevant for SOX – meaning that even their total absence would not amount to a material weakness (as defined).

For example, the second principle is:

The board of directors demonstrates independence from management and exercises oversight of the development and performance of internal control. of objectives.

I contend that while it may be relevant for some control objectives, it is not relevant for SOX. A private company that does not have independent directors can still have effective internal control over financial reporting.

XX

I have questions for you that I would appreciate your answering in the comments below for everybody to consider. (In other words, please do not post your answers only on LinkedIn.)

1. Have you considered whether any of the COSO principles are not relevant for your SOX program?
2. Which ones were considered not relevant?
3. Have you discussed this with your external auditor?
4. Did they agree, and if not why not?

XX

Thanks – and I look forward to your thoughts on the post and the answers to my questions.

XX

XX

PS – If you are interested in attending one of my SOX Masters classes, please contact Emily Jones at emilyj@marcusevansch.com.

Trends in SOX compliance programs

The software company Workiva has been surveying practitioners to understand what is happening with SOX programs since 2016. They recently shared a summary of trends over these last five years.

They draw four conclusions.

1. Internal audit is the majority owner of the SOX program.

Comments:

• Technically, management always retains ownership of the SOX program. However, internal audit may perform much of the assessment activity on behalf of management.
• Workiva has not shared how many companies were surveyed or whether they are the same companies each year. As a result, it is somewhat speculative to draw conclusions from the survey results. However, it is not unreasonable to assume that the survey sizes have been significant and at least indicative of the trends asserted by the authors.
• There is a huge difference between performing the testing on behalf of management and planning/managing the entire SOX program (a distinction not drawn in the report). My personal observation supports an assertion that the majority of companies rely on internal audit to perform testing. But saying that they own the program goes perhaps a bit too far.

 

2. Even when internal audit is not the owner of the SOX program, it is involved in several facets of the SOX program.

Comments:

• The paper says “we draw the conclusion that the performance of SOX compliance activities is negatively impacting the capacity of internal audit teams to execute assurance reviews”. However, there is no evidence provided to support that position. Just because internal audit in many cases (31% here of the 77% who perform SOX testing, or 23.87% of the population) are spending more than 50% of their time on SOX does not mean that they lack sufficient resources to address their other responsibilities. That question is neither asked nor answered.
• It is interesting that the percentage of internal audit functions performing SOX testing is down from 85% in 2016 to 77% in 2020. Since this is the greatest consumer of resources (compared to performing walkthroughs, issue tracking, and risk assessment), it is likely that internal audit resource allocation to SOX is actually less in 2020 than in 2016.
• It is also interesting to see that a number of internal audit functions perform testing but not walkthroughs. That sounds like an opportunity that has been missed.

 

3. The cost of SOX compliance is increasing.

Comments:

• I would be shocked if it was not increasing, given inflation and escalating external audit fees!
• Workiva says “As organizations continue to grow and processes become more complex, the number of SOX key controls will increase, and survey results reflect this trend as well: the number of respondents who reported 250+ controls increased 10% between 2016 and 2020”. This is not logical if a proper top-down and risk-based approach is taken. Remember that as a company’s revenue grows, so does its level of materiality. In many cases, a careful scrubbing to remove non-key controls from scope should in many if not most cases reduce the number of key controls! As materiality increases, the ways in which there could be an error or omission in the consolidated financial statements will generally go down, not up.
• I do not see the logic that adopting solutions like Workiva’s reduces cost. If anything, it is likely to increase it.

 

4. Practitioners continue to focus SOX programs on cybersecurity risk.

Comments:

• Hackers that take advantage of cybersecurity weaknesses have never, to my knowledge, targeted the financial statements. They may steal data, ask for ransom, or cause disruption, but the likelihood of a material misstatement as a result of a hack is very low indeed.
• If there is a breach that causes disruption and an inability to file financial statements with the SEC on time, that is not a SOX issue. It may be a violation of other SEC requirements.
• While I often hear of pressure from the external auditors to address cybersecurity risks, a proper top-down and risk based approach (preferably using the IIA’s GAIT Methodology, which I strongly recommend) should help organizations determine whether the risk of a material misstatement is real.
• Workiva justifies their assertion by pointing to survey results: In 2017 (there are no 2016 results) 84% had fewer than 100 ITGC key controls in scope, whereas in 2020 that is 80%. However, in 2019 the number was 77%. The survey results simply don’t support their assertion.

 

 

So, what are the SOX program trends based on my experience (I have been leading a SOX Masters[1] training class for 8 years or so)?

1. There continue to be massive opportunities for most organizations to ‘right-size’ their program. Unless regularly pruned using a top-down and risk-based process, the program will grow out of control. Just because a control was in scope last year does not mean it should continue to be in scope in 2021.
2. Leadership of the SOX program continues to change, necessitating training for new SOX program (and internal audit) leaders. Several companies send every new leader to my SOX Masters program.
3. The external auditors continue to latch onto every new risk of the day. The great majority of their requests for scope changes don’t survive the question of “Where is the risk of a material misstatement? Show me!”
4. While technology can be very helpful and increase the efficiency of the SOX program, care has to be taken when it comes to trying to use it to test controls. Most analytics and other tools test the data, not the controls.
5. Internal audit adds tremendous value when it performs SOX testing on behalf of management, and their understanding of risk and controls aids SOX program management. But they should always work with the board to ensure they have sufficient resources to address the more significant sources of risk (including opportunity) to enterprise objectives.

 

I welcome your thoughts.

 

[1] The next class is scheduled for February, 2021

SOC Compliance and Service Providers

I always read advice and guidance from Protiviti, especially when Jim DeLoach is involved in it. The firm is a prolific source and they often have good advice – but not always.

A couple of weeks ago, they published Preparing for Annual SOX Compliance Amid COVID-19:  Outsourced Processes and Use of Third-Party Providers Remain Relevant to ICFR.

First, let me reset your expectations. Their article and this post have next to nothing to do with COVID-19. They are using that as a hook; the only point they make relative to COVID is that the SOC-1 reports might be delayed.

Protiviti has been pushing this article on social media, so I am going to share my thoughts before people start down the wrong path.

They outline and discuss these steps:

1. Inventory your providers
2. Obtain SOC reports
3. Map controls from the SOC report to management’s processes
4. Evaluate deficiencies identified in the SOC report and assess potential impact to your business
5. Obtain bridge letters
6. Determine impacts from the pandemic
7. Take appropriate actions

Now why is this the wrong path?

It is not top-down and risk-based. It is fundamentally bottom-up.

Here’s a better series of steps:

1. When you perform your SOX scoping, identify where you are relying on key controls performed by a service provider to provide reasonable assurance on an ICFR risk identified in your scoping. Just because you are using a service provider doesn’t mean you don’t have adequate key controls to rely on that are performed by your company’s staff. You may or may not be relying on key controls performed by the service provider. (Adequate means that you can rely on the controls to prevent or detect a material error or omission in the filed financial statements.)
2. Identify the specific controls performed by each service provider on which you need assurance and include them in scope as key controls.
3. Make sure – in advance – that these controls will be included in the scope of the SOC-1 audit of the service provider. Where you can, use prior reports but supplement them with inquiries of the service provider to make sure the controls at the service provider that will be audited match your needs. Be prepared for step 5.
4. Obtain the SOC-1 reports.
5. Review the description of the controls they tested and make sure that the design of the controls meets your needs.
6. Confirm that the SOC-1 report indicates that the controls were operating effectively. Pay attention to the timing of the report and the testing.
7. Review the list of controls that the SOC-1 auditor has indicated they expect the company to perform. Confirm that either they are among your key controls, are unnecessary, or take action to include additional controls.
8. Evaluate any deficiencies in the same way you evaluate deficiencies in controls performed in-house.
9. Discuss with the service provider the actions they are taking to address any deficiencies and when those will be completed and rested.
10. Determine what additional actions should be taken given the deficiencies and the remediation planned by the service provider. This may involve identifying and testing additional compensating or mitigating controls.
11. If necessary, obtain bridge letters or otherwise roll forward the assessment.
12. Discuss with management the performance of the service provider and determine if any actions should be taken.

All of this should be carefully documented and discussed with the external auditor through the process, especially where issues are identified or anticipated.

I welcome your thoughts.

I will be leading (virtual) training on SOX in October. See here for details.

Opportunities to upgrade your skills

This pandemic has shut down, as you might expect, all the in-person conferences and seminars that I had expected to participate in this year.

However, I will be leading some small group online training starting in October. If you are interested, please follow the links below to obtain more information.

Each event will be what we call 3X3: three hours each day for three days.

Sarbanes-Oxley s404 Master Class October 20, 21, 22

GRC – A Corporate Discipline November 3, 4, 5

Risk Management that Helps the Organization Succeed November 17, 18, 19

Auditing that Matters: Building a World-Class Internal Audit Function

Is your SOX program both effective and efficient?

Protiviti’s surveys and reports are always worth reading. One I look forward to is their annual survey on SOX compliance.

Those of you who are responsible for the SOX program or SOX testing at their organization are likely to find the benchmarking info in the 2019 survey, Benchmarking SOX Costs, Hours and Controls of interest.

However, I want to share (again) a note of caution.

Protiviti and others are talking about the use of analytics and other tools, such as RPA, for SOX testing.

But, the purpose of the SOX testing is to:

• Confirm that the design of the controls relied upon to prevent or detect a material error or omission in the financial statements filed with the SEC are sufficient, if they are operated as designed, to address such a possibility. The likelihood of a material error or omission is less than reasonably possible.
• Confirm, with a reasonable level of assurance, that those controls are being performed consistently as designed.

The end product is an assessment as to whether the system of internal control over financial reporting is effective; that means that the controls are sufficient to provide reasonable assurance that a material error or omission would be prevented or detected.

What do these newer technology tools do for us?

For the most part, they provide some level of assurance that the data, and possibly the transactions, are free from error.

But do they provide any assurance that the system of internal control is effective?

While the presence of errors is a strong indicator that the controls are not sufficient, the absence of errors is not a strong indicator that the controls are effective!

The data may be free from error even though the controls are not being performed at all!

In my SOX training classes (the next one is in October), I ask the attendees how many of them have had their homes burglarized in the last year. Only on the rare occasion has anybody raised their hand.

I then ask whether the fact that they have not been burglarized is proof that they locked all the doors and windows before they left the house.

I remember one time in England when, as an IT auditor, I was flowcharting and identifying controls in a very complex integrated system. One of the controls that management had identified was a comparison between data at one point in the system to the data at a much later point (a “run to run” control). When I examined the logic of the program that did the comparison, I found that it was coded incorrectly. At each point, early (file E) and late (file L), a file was created that could be compared. But the comparison program was comparing data in file E to data on file E – instead of file L.

The control was doing nothing. But the data happened to be clean anyway (we checked).

So, when it comes to the use of technology tools, will they provide the evidence you need that the controls relied on are both adequately designed and operated? Do they test the controls or only the data?

My second note of caution is to remain focused on whether the system of internal control over financial reporting provides reasonable assurance that material errors will either be prevented or detected. That refers to the possibility of errors in the consolidated financial statements filed with the SEC.

Too many, typically under pressure from the external auditors, are adding controls without asking whether they are needed to prevent or detect a material error.

                WHERE’S THE RISK?

The scope does not, and typically should not, include controls that would never result in material weaknesses should they fail. It’s not a matter of whether they are important controls, or required to address the risk-du-jour. It’s a matter of whether they are being relied upon to prevent or detect a material error in the filed financials.

One final point: I don’t care how many ‘entity-level’ controls you have. I only care whether you have selected the right controls to include in scope.  By ‘right’ I mean the combination of controls that can be relied on to function consistently and address the risk of a material error, and are efficient to operate and test.

I welcome your thoughts.

Cyber and reputation risk are dominoes

Anthony Fitzsimmons recently sent me a review copy of his new book, Rethinking Reputation Risk. He says that it “Provides a new perspective on the true nature of reputational risk and damage to organizations and traces its root causes in individual and collective human behavior”.

I am not sure that there is much that is new in the book, but if you want to understand how human behavior can be the root cause (in fact, it is very often the root cause) of problems for any organization, you may find it of interest.

The authors (Fitsimmons and Professor Derek Atkins) describe several case studies where human failures led to serious issues.

Humans as a root cause is also a topic I cover in World-Class Risk Management.

As I was reading the book, I realized that I have a problem with organizations placing separate attention to reputation risk and its management. It’s simply an element, which should not be overlooked, in how any organization manages risk – or, I should say, how it considers what might happen in its decision-making activities.

The same thing applies to cyber risk and even compliance risk.

They are all dominoes.

A case study:

• There is a possibility that the manager in HR that recruits IT specialists leaves.
• The position is open for three months before an individual is hired.
• An open position for an IT specialist who is responsible for patching a number of systems is not filled for three months.
• A system vulnerability remains open because there is nobody to apply a vendor’s patch.
• A hacker obtains entry. CYBER RISK
• The hacker steals personal information on thousands of customers.
• The information is posted on the Internet.
• Customers are alarmed. REPUTATION RISK
• Sales drop.
• The company fails to meet analyst expectations for earnings.
• The price for the company’s shares drop 20%.
• The CEO decides to slash budgets and headcounts by 10% across the board.
• Individuals in Quality are laid off.
• Materials are not thoroughly inspected.
• Defective materials are used in production.
• Scrap rates rise, but not all defective products are detected and some are shipped to customers.
• Customers complain, return products and demand compensation. REPUTATION RISK
• Sales drop, earnings targets are missed again, and …….
• At the same time as the Quality staff is downsized, the capital expenditure budget is cut.
• The Information Security Officer’s request for analytics to detect hackers who breach the company’s defenses is turned down.
• Multiple breaches are not detected. CYBER RISK
• Hackers steal the company’s trade secrets.
• Competitors acquire the trade secrets and are able to erode any edge the company may have.
• The company’s REPUTATION for a technology edge disappears. REPUTATION RISK
• Sales drop. Earnings targets are not achieved, and……..

It is true that every domino and the source of risk to its stability (what might happen) needs to be addressed.

But, focusing on one or two dominoes in the chain is unlikely to prevent serious issues.

One decision at a low level in the company can have a domino effect.

Consider this slide deck by ERM Strategies, Inc. about the Deep Water Horizon disaster.

I welcome your comments.

A new front opens in the SOX battle

One of the issues that I address in my SOX Master Classes (the next one is in February) has come of age.

I am talking about the certification signed by the CEO and CFO and included in the quarterly filing with the SEC – the one required by Section 302 of the Sarbanes-Oxley Act.

The issue is this:

• The CEO and CFO are required by law to assess the state of internal control over financial reporting (and disclosure control) every quarter and report whether or not it is effective as of the date of the quarterly filing.
• For their own as well as the company’s protection, they need to have a reasonable basis for that assessment.
• Tests of internal control over financial reporting are typically spread over the year. Some perform tests in every quarter; some during at least a couple of quarters; and few limit their testing to the fourth quarter.
• Deficiencies in the controls are identified during that testing.
• Those deficiencies may be assessed as potential material weaknesses if not corrected and retested prior to the end of the year.
• As a result, potential material weakness frequently not only exist but are known to exist at the time that the CEO and CFO are required to assess and certify internal control over financial reporting.
• But, for whatever reason, these potential material weaknesses either are not reported to the CEO and CFO (which fails one of the Section 302 requirements: they have to certify that they know about control issues) or are ignored.
• The CEO and CFO may certify that the systems of internal control and disclosure controls are adequate when they are not.

This is what I have to say in Management’s Guide to Sarbanes-Oxley Section 404: Maximize Value Within Your Organization:

In the past, most CEOs and chief financial officers (CFOs) have signed their annual and quarterly certifications—which are included in the financial statements filed with the SEC on Form 10-Q and required by Section 302 of Sarbanes-Oxley—without a rigorous examination of internal controls. Ideally, management has integrated the quarterly and annual assessment processes. Although management is not required to test all its key controls every quarter, it should perform some degree of testing each quarter to support the quarterly Section 302 certification. At a minimum, the Section 302 certification process should include a consideration of the status of the Sarbanes-Oxley project, the results of testing, the severity of any identified control deficiencies, and management’s corrective action plans.

When I was writing the book, I talked to the SEC about this issue. They said that they understood it but it was not a priority at that time.

Well “the times, they are a-changing”.

This recently appeared on the CFO magazine web site in an article on SEC Focuses on Internal Control by a former chief accountant of the SEC’s Division of Enforcement. In the middle of the article is this section:

Specific issues that investigators have been addressing include whether a material weakness: (1) existed in a reporting period before a restatement; (2) was adequately described as to scope; (3) existed, even if there was no material error; and (4) existed in connection with controls and procedures for disclosure, or in connection with 302 certification processes.

In the book and in the class, I recommend that management and the SOX PMO consider how the results of testing during earlier quarters are incorporated into the Section 302 certification process.

For example, is the SOX PMO (or equivalent) included in the disclosure review process?

When potential material weaknesses are discovered during SOX or internal audit testing, my suggestion is to review the issue with the legal function. They can advise the CEO and CFO whether this should be disclosed as part of the Section 302 certification.

This new front is clearly starting to open.

Don’t let it pull you under.

I welcome your comments.

Lessons Learned from the Transition to COSO 2013

Protiviti has shared with us a useful Top 10 Lessons Learned from Implementing COSO 2013.

I especially like this section:

It is presumed that everyone understands that a top-down, risk-based approach remains applicable to Section 404 compliance, and the transition to the 2013 updated Framework does not affect this. While we don’t list this as a lesson, we could have, because some companies either forgot or neglected to apply this approach when setting the scope and objectives for using the Framework. As a result, they went overboard with their controls documentation and testing. We can’t stress enough that the COSO 2013 Framework did not change the essence of, and the need for, a top-down, risk-based approach in complying with SOX Section 404.

The report has a number of excellent pieces of advice. However, I wouldn’t be me if I didn’t have points of disagreement.

The first is on mapping. It is NOT necessary to map all your controls to the principles. If we take principle 10, for example, it states “The organization selects and develops control activities that contribute to the mitigation of risks to the achievement of objectives to acceptable levels”. Rather than map all your control activities to this principle (or to principle 11, which is the same – just for IT general controls), the organization needs to identify the control(s) it relies on for its assessment that the principles are present and functioning[1]. For principles 10 and 11, that will be the SOX scoping exercise. For the principle on fraud, the control that should be identified is the fraud risk assessment, not every control relied on to detect or prevent fraud.

Then there is the assertion that indirect controls are the same as entity-level controls. COSO (both 1992 and 2013) tell us, correctly, that activities in each of its components may operate at any level within the organization. For example, let’s say that an account analysis is prepared by Corporate Finance as part of the period-end close. This entity-level control may operate with sufficient precision to be relied upon to detect a material error or omission in that account. But the entity-level control is a direct control, not an indirect control. (A direct control can be relied upon to prevent or detect an error. An indirect control is one that serves to increase or decrease the likelihood that other, direct, controls will function effectively. Hiring, integrity, oversight by the board – these are indirect controls where a defect would increase the likelihood that affected direct controls would fail.)

Another example that helps us understand the difference is the hiring process (related to principle 4, in the Control Environment). The hiring process most often is at a lower level than the entity-level, often as deep as the activity level as that is where most hiring managers reside. Controls in the hiring process in this situation are activity level (or what I call ‘intermediate level’ controls, operating at a location or business unit rather than either the top or the bottom of the organization) and are indirect controls.

I could quibble with one or two more points, but I don’t want to detract from the report. I want, instead, to encourage you to read and discuss it.

What do you think?

What additional lessons have you learned?

[1] Full credit for this wording goes to the E&Y national office, who used it in a conversation I had with them about the firm’s training of its audit staff.

The most important sentence in COSO

In my opinion, one sentence stands out, whether you are looking at the COSO Internal Control – Integrated Framework (2013 version) or the COSO Enterprise Risk Management – Integrated Framework.

That sentence is:

An effective system of internal control reduces, to an acceptable level, the risk of not achieving an objective relating to one, two, or all three categories.

The sentence is important because it emphasizes the fact that the purpose of controls is to address risk, and that you have ‘enough’ control when risk is at desired levels.

To me, this means that:

1. Before you assess the effectiveness of internal control, you need to know your objective(s), because we are talking about risk to objectives – not risk out of context
2. You need to know the risk to those objectives
3. You need to know what is an acceptable level of risk for each objective, and
4. You need to be able to assess whether the controls provide reasonable assurance that risk is at acceptable levels

You may ask “where is that sentence?”, because when consultants (and even COSO and IIA) make presentations on COSO 2013 and effective internal control, all you hear about are the principles and components.

In fact, anybody who reads COSO 2013 should have no difficulty finding this most important sentence. It’s in the section headed “Requirements for Effective Internal Control”.

This is how that section starts:

An effective system of internal control provides reasonable assurance regarding achievement of an entity’s objectives. Because internal control is relevant both to the entity and its subunits, an effective system of internal control may relate to a specific part of the organizational structure. An effective system of internal control reduces, to an acceptable level, the risk of not achieving an objective relating to one, two, or all three categories. It requires that:

• Each of the five components of internal control and relevant principles are present and functioning
• The five components are operating together in an integrated manner

There is no mention of satisfying the requirement that the “components and relevant principles are present and functioning” until after the reference to risk being at acceptable levels.

In fact, I believe – and I know of at least one prominent COSO leader agrees – that assessing the presence and functioning of the components and principles is secondary, provided to help with the assessment.

Let’s have a look at the very next paragraph in the section:

When a major deficiency exists with respect to the presence and functioning of a component or relevant principle or in terms of the components operating together, the organization cannot conclude that it has met the requirements for an effective system of internal control.

When you look at this with the (COSO) risk lens, this translates to the ability to assess internal control as effective, and the principles and components as present and functioning, as long as there is no deficiency in internal control that is rated as “major”.

How does COSO determine whether a deficiency is “major”? That can be found in the section, “Deficiencies in Internal Control”.

An internal control deficiency or combination of deficiencies that is severe enough to adversely affect the likelihood that the entity can achieve its objectives is referred to as a “major deficiency”.

Let’s translate this as well:

1. If the likelihood of achieving objective(s) is “severe”, then the risk is outside acceptable levels.
2. If the risk is outside acceptable levels, not only should the related component(s) or principle(s) not be assessed as present and functioning, but internal control is not considered effective.
3. When it comes to SOX compliance, a “major deficiency” translates to a “material weakness”. The objective for SOX is to file financial statements with the SEC that are free of material error or omission. The acceptable level of risk is where the likelihood of a material error or omission is less than reasonably possible.
4. That means that if the deficiency is less than “major” (or “material” for SOX purposes), then the related component(s) or principle(s) can be assessed as present and functioning – and internal control can be assessed as effective.

So, the only way to assess whether the principles and components are present and functioning is to determine whether the risk to objectives (after considering any related control deficiency) is at acceptable levels.

Do you see what I mean?

Risk is at the core. Assessing the presence and functioning of components or principles without first understanding what is an acceptable level of risk to objectives is misunderstanding COSO!

Why are so many blind to this most important sentence?

I have a theory: the presentations were all prepared based on the Exposure Draft. That document failed to reference the requirement that internal control be designed to bring risk within acceptable levels. (The defect was fixed after comments were received on the issue.)

Do you have a better theory?

Can you explain the blindness of so many to the most important sentence in the entire Framework?

Why Internal Audit Fails at Many Organizations

When recent studies by KPMG and PwC indicate that about half of internal audit’s key stakeholders (board members and top executives) do not believe that internal audit is neither delivering the value it should nor addressing the risks that matter, we have to recognize that internal auditing is failing at many organizations.

With that in mind, a recent PwC publication in its Audit Committee Excellence series, Achieving Excellence: Overseeing internal audit, merits our attention.

My opinion is that while the audit committee members may be assessing internal audit performance as ‘needs improvement’, they should be looking in the mirror. Internal audit reports to them; if it is not performing to their satisfaction, they are either failing to communicate expectations clearly, not demanding the necessary improvements, not providing the critical support they need when management is pulling them in a different direction, not taking actions (such as replacing the CAE) to effect change, or all of the above.

Audit committee members need guidance and while the IIA does provide some excellent insights from time to time, the audit firms’ publications are often one of the first that are read.

The PwC publication makes some very good points but unfortunately demonstrates a limited understanding of internal audit best practices. This could be because it was written by their governance team rather than by their internal audit services leaders. (PwC’s internal audit services arm has produced not only good guidance from time to time (including their State of the Internal Audit Profession series), but some excellent thoughts leaders (including the IIA CEO, Richard Chambers).)

Let’s look at what they did well:

“A priority for the audit committee should be empowering the internal audit organization by providing visible support.”

This is an excellent point and PwC describes it well. The audit committee should actively engage internal audit and by showing its respect for the CAE and his team promote respect by management.

“Sometimes internal audit crafts an annual plan that leverages its group’s capabilities rather than addressing the company’s key risks. Audit committees will want to be on the lookout for this.”

Another fine point. The audit committee should take responsibility for ensuring that internal audit addresses the risks that matter to the organization.

“Understand whether resource constraints (e.g., restrictions on travel budgets or the ability to source technical skills) have an impact on the scope of what internal audit plans to do. If the impact of any restrictions concerns the audit committee, take steps to help internal audit get the resources it needs.”

The audit committee should ensure that internal audit has an appropriate level of resources, sufficient to provide quality insight and foresight on the risks that matter now and will matter in the near future.

“Audit committees should determine if they are accepting a sub-excellent level of performance and competence in a CAE (and internal audit function) that it wouldn’t be willing to accept for a CFO (or other key role).”

If the CAE is not considered as critical to the success of the audit committee, something is wrong and the audit committee should take action – even if, perhaps especially if, management holds the CAE in high regard while he delivers little of value to the audit committee.

Periodically discuss whether the amount and type of information internal audit reports to the committee is appropriate.

While this is an essential activity, PwC doesn’t get the issue right. The audit committee should ensure it receives the information it needs to perform its responsibilities for governance and oversight of management. That is not a simple matter, as PwC implies, of being succinct in how the CAE presents audit findings.

What did they miss?

1. The audit committee should ensure that all the risks that matter now and will matter in the near future are getting the appropriate level of attention from internal audit.
2. The audit committee should challenge any audit activity that is not designed to address a risk that matters.
3. The audit committee should take a very strong stance that internal audit reports to them and serves their needs first, not those of management. The PwC paper identifies two reporting lines but is wish-washy on the subject, only saying that “Directors and management should reach consensus on which areas should be internal audit priorities.”
4. The audit committee should challenge internal audit on how they work with the risk management activity. Where it exists, are they assessing its effectiveness? Are they working effectively with risk management? Do they leverage management’s assessment of risk appropriately?
5. The audit committee should be concerned about the CAE’s objectivity and independence from undue management influence. Does he have one eye on internal audit and the other eye on his next position within the company?
6. The audit committee should also ensure that it has an appropriate role in the hiring, performance assessment, compensation, and (where necessary) firing of the CAE.
7. Finally, but in many ways most importantly, the audit committee should require that the CAE provide them with a formal assessment of the company’s management of risks and the effectiveness of related internal controls.

The publication makes some technical mistakes because the authors are not internal audit practitioners. Can you spot them?

That’s my challenge to you – in addition to welcoming your comments.

The effective audit committee

A short article in CGMA Magazine, Ingredients of an effective audit committee, caught my eye. I recommend reading it.

I think there are some key ingredients to an effective audit committee that are often overlooked. They include:

1. The members have to read all the material for the audit committee meeting before the meeting. It’s amazing how often they don’t, which reduces the meeting to absorbing the material rather than a constructive discussion of its implications.
2. The members have to be ready, willing, and able to constructively challenge all the other participants, including the external and internal auditors as well as financial, operating, and executive management. Too often, they are deferent to the external auditor (for reasons that escape me) and too anxious to be collegial to challenge senior management.
3. They need a sufficient understanding of the business, its external context (including competitors and the regulatory environment), its strategies and objectives, risks to the achievement of its objectives, and the fundamentals of risk management and financial reporting, to ask the right questions. They don’t need to have a deep understanding if they are willing to use their common sense.
4. They need to be willing to ask a silly question.
5. They need to persevere until they get a common sense response.
6. No board or committee of the board can be effective if they don’t receive the information they need when they need it. I am frustrated when I read surveys that say they don’t receive the information they need – they should be demanding it and accepting no excuses when management is slow to respond.
7. Audit committee members will not be effective if they are only present and functioning at quarterly meetings. They need to be monitoring and asking questions far more often, as they see or suspect changes that might affect the organization and their oversight responsibilities.

What do you think?

I welcome your comments.

A Rant about the GRC Pundit’s Rant

Michael Rasmussen, a.k.a. the GRC Pundit, is a friend whose intellect, integrity, and insights I respect. He and I, together with another friend, Brian Barnier, were the first three to be honored as OCEG Fellows for our thought leadership around GRC.

Michael and I have had many a debate on the topic of GRC. Michael brings the perspective of an analyst that works with many companies, helping them select and implement software solutions. That is his business: he refers to himself (GRC 20/20 Research, LLC) as a “buyer advocate; solution strategist; and market evangelist”. His latest blog, GRC Analyst Rant: Throwing Down the GRC Analyst Gauntlet, inspired me to write this one.

My background is very different, having been a practitioner and executive responsible for many of the business activities he supports – in other words, I might have been one of his customers. My focus is on helping business run better – and that frequently but not always involves the judicious use of technology.

Michael and I agree on a number of points, disagree on others. For example, I believe he and I agree that:

• The term ‘GRC’ is one that is interpreted in many ways.
  • When I ask practitioners within a company what they mean when they use the term, most say it stands for ‘governance, risk, and compliance’ but cannot explain why anybody would use that term to describe the totality implied by the expression; they may wave their hands in the air and say “what does GRC mean? You know…. it means GRC”. They cannot explain why they don’t refer to governance, or governance and risk management, or risk management and compliance. Sometimes they talk as if GRC is something in the air, something related to the culture of the organization as much as anything else.
  • When I ask people at the IIA, they say it stands for ‘governance, risk, and controls’; in other words, the totality of what internal auditors work on. I don’t personally see anything new in this, nor any value in using the term. In fact, using it with ‘controls’ instead of the more usage of ‘compliance’ is only going to confuse.
  • When I talk to software vendors, they either describe their software solutions (as if GRC is technology) or describe the business solutions that their technology supports.
  • When I read papers from consultants, I find that if I substitute the phrase ‘risk management’ every time they say ‘GRC’, the piece makes more sense. In other words, they are usually talking about risk management but for some reason (some would say to hype the discussion) they use the term GRC instead.
  • When I talk to the people at OCEG and those who follow OCEG and its definition of GRC, they use a definition that makes more sense. That definition adds value by emphasizing the needs for all parts of the organization to work together.

• GRC is not about technology. It is about (as I said last year) “how we can optimize outcomes and performance, addressing uncertainty (risk management) and acting with integrity (regulatory compliance and organizational values)”.
• The key to optimizing outcomes is to for management (with board approval) to set the appropriate strategies, objectives, and goals, and then everything flows from there: managing risks to strategies, managing performance against strategies, and acting with integrity (which includes compliance with applicable laws and regulations) at all times.
• No technology vendor (not even SAP and Oracle, who have the greatest breadth and depth of solutions IMHO) has a complete solution that addresses all GRC needs. The last time I said that, in a September post, several vendors wrote to tell me they had everything. But, they simply didn’t. They have everything that they chose to call GRC, but none included strategy management, support for governance activities like board packages and whistleblower lines, risk management including automated and integrated key risk indicators, compliance training and monitoring, performance management, legal case management, and so on.
• The analysts like Gartner and Forrester have a business model where they need to define technology using buckets. But those buckets do not reflect what individual companies actually need, so their analyses and ratings may be interesting but may well steer organizations to acquire solutions (such as a so-called ‘EGRC platform’) that are not the best use of scarce resources. I would not advise any organization to base their purchase decision on an analyst rating of ‘GRC’, ‘EGRC’ or other made-up bucket of fish.

Where I believe we differ is that I do not advocate the use of the term ‘GRC’.

As I inferred, if not explicitly stated in my post last November, I believe that if the term ‘GRC’ is not dead (and apparently it lingers on), then it should be put to death.

I do not see the value in business people talking about GRC. I have said before and will say again, managers should look to fixing the processes they know need work.

For example, few organizations have effective processes for developing strategies and objectives at the corporate level, cascading them down throughout the organization so every individual knows what they need to do if the organization is to succeed, and minimizing individual objectives that are not clearly necessary to corporate achievement –then rewarding individuals, at least in part, for performance against those cascaded objectives. I have worked at several organizations where we were told what the corporate objectives were and asked to link our personal objectives to them. That is not the same thing. That is tying our personal objectives onto a branch of the corporate objectives, rather than making sure that all the roots of that corporate objective tree are healthy – even when we should be responsible for the health of a root or two.

Another example is the effectiveness of risk management. Most organizations practice enterprise list management at best (i.e., they manage a limited number of risks on a periodic basis), when mature risk management that is dynamic, iterative, and responsive to change, integrated into decision-making at all levels of the organization and into every aspect of daily operations, is essential to success.

Does using the term ‘GRC’ mean anything useful for internal auditors? No. They should continue to “up their game” from a focus on controls and risks that matter to operating management, to providing assurance and insight on organizational governance and risk management.

Effective GRC for OCEG means the integration, among other things, of strategy and risk management. But how many organizations do that well? How many executives receive and manage their area using an integrated report or dashboard that shows for each of their strategies both the current level of performance and the current state of related risks? How many executives see that not only have they accelerated up to the desired level of 100kph but are less than 100m from hitting a brick wall?

So here’s my recommendation to all: stop talking about GRC and start talking the language of the business. Let’s talk about how we can increase value to stakeholders, address potential obstacles and seize opportunities to excel, act with integrity and remain in compliance with current and anticipated regulations, and manage the organization to success.

Don’t try to fix GRC. Fix those parts of the business, those business processes, that are broken.

Good Riddance grC.

I welcome your comments.

What is effective risk management?

Some say that risk management is effective when it has all the components described in their favorite standard (ISO 31000:2009) or framework (COSO ERM). (COSO ERM specifically states this as the requirement).

Some say that risk management is effective when all the principles in their favorite guidance are present and functioning. (ISO talks about its “set of principles that organisations must follow to achieve effective risk management.”) The principles are (from a consultant’s site that provides a high-level view of the standard):

• Creates and protects value;
• Is an integral part of all of the organisation’s processes;
• Forms part of decision making;
• Explicitly expresses uncertainty;
• Is systematic, structured and timely;
• Is based on the best available information;
• Is tailored to the organisation;
• Takes human and cultural factors into account;
• Is transparent and inclusive;
• Is dynamic, iterative and responsive to change; and
• Facilitates continual improvement of the organisation.

Some say that risk management is effective when activities are compliant with the organization’s related policies and standards. But are those policies and standards adequate?

Some will say that risk management is effective when the board, operating and executive management believe it adds value and are satisfied that it provides the information they require. I believe that has merit but they may be satisfied with less than mature risk management (that seems to be the case with many current organizations who are satisfied with enterprise list management, until they are caught short).

Some will say that risk management is effective when an independent assessment/audit/examination is performed and the report says so. The trouble is that the people who do such audits generally rely on one of the above criteria (components present, principles in operation, etc.)

I would like to suggest a different approach.

Let’s start by considering why organizations should have risk management. It’s NOT because laws and regulations mandate it in many cases. It’s NOT because people say you need it. It’s because effective risk management provides a level of assurance that an organization will not only achieve its objectives (or exceed them) but will set the best objectives.

Quoting from COSO ERM:

“Enterprise risk management helps an entity get to where it wants to go and avoid pitfalls and surprises along the way.”

COSO explains that effective risk management enables:

• “A greater likelihood of achieving business objectives”
• “More informed risk-taking and decision-making”

Irish guidance on the ISO 31000:2009 risk management standard says:

“The purpose of managing risk is to increase the likelihood of an organization achieving its objectives by being in a position to manage threats and adverse situations and being ready to take advantage of opportunities that may arise.”

The Australian mining company, BHP Billiton, has a risk management policy signed by its CEO. It includes:

“Risk is inherent in our business. The identification and management of risk is central to delivering on the Corporate Objective.

• By understanding and managing risk we provide greater certainty and confidence for our shareholders, employees, customers and suppliers, and for the communities in which we operate.
• Successful risk management can be a source of competitive advantage.
• Risk Management will be embedded into our critical business activities, functions and processes. Risk understanding and our tolerance for risk will be key considerations in our decision making.

“The effective management of risk is vital to the continued growth and success of our Group.”

I like what E&Y has to say:

“An effective [ERM] capability provides value by giving organizations the confidence to take on risk, rather than avoid it.

“By effectively managing the right risks, management has more timely, comprehensive and a deeper understanding of risk which, in turn, facilitates better decision-making and confidence to take on new ventures or even to accept higher levels of risk.”

So we can see that, as the BHP CEO said, effective risk management is not only essential to the success of an organization but “can be a source of competitive advantage”.

For the last year or two, I have been saying that you assess the effectiveness of risk management by asking decision-makers at all levels whether the risk information is enabling them to make better decisions and be more successful.

In other words, assess risk management not by its structure but by its effect.

I still think that is a key test, but I am going to add a new dimension to my thinking.

Let’s consider a company that has significant foreign currency exposure. It does business globally so it has bank accounts in a number of countries and has both payables and receivables in different currencies.

There are a number of strategies for reducing foreign exchange risk, but to manage the risk effectively you need to know what is happening with rates as well as how your bank account balances, payables, and receivables are changing.

If this company only has the ability to understand its foreign exchange risk once a month, in other words its monitoring of this risk is only monthly because that is the only time it is able to obtain all the necessary information and calculate its exposure, the risk is much higher than if it has the processes, people, and systems to monitor its exposure daily or better.

However, the investment necessary to upgrade the risk monitoring from monthly to daily may be significant. The company has to decide whether the reduction in exchange risk that can be improved by upgrading risk monitoring justifies the additional expense.

Until it upgrades risk monitoring, there is a risk that the information provided by risk management is insufficient. Management needs to decide whether that is an acceptable level of risk.

If management decides that the level of risk is too high, then I would say that the risk management program is less than effective. It is not providing the information necessary for management to take the right risks. But if management decides that the level of risk is acceptable, then that would not prevent me from assessing risk management as effective.

Let’s take another situation. An organization is concerned about its reputation risk. It has engaged a company to monitor reputation risk indicators (using social media analytics) and report once each quarter. However, it is in an industry where customer satisfaction can move quickly and significantly.

Quarterly risk monitoring creates a risk that the risk management program is not providing the information necessary to manage risks to the enterprise objectives. As in the prior example, management will need to decide whether an investment in more frequent reputation risk monitoring is justified by the potential reduction in reputation risk (because it would increase the ability to respond to customer complaints, etc.)

If management decides that quarterly risk monitoring represents a risk outside acceptable ranges, I would say that the risk management program is less than effective. It is not providing the information necessary for management to take the right risks, and management has determined that this is a risk (the risk of a bad decision) is unacceptable.

One final example. The company has an excellent risk management framework, formal policies and procedures, processes, and enabling systems. However, in the last year the level of staff turnover among the champions of risk management in the executive ranks and among the risk officers themselves means that the experience of the individuals relied upon to monitor, understand, assess, evaluate, and respond to risks has diminished.

There is an increased likelihood than in prior years that risks will not be managed as desired, the wrong risks taken, and that risk information that flows to top management and the board may not be reliable.

This is a deficiency in the operation of risk management and may represent a risk to the achievement of objectives because it results in less than reliable risk information on which decisions are based. If the risk is unacceptable, then until it is treated and brought back to within acceptable ranges I would say that the risk management program is less than effective.

So, where am I going?

If we revisit the objective of risk management, we see that we rely on it to provide management and the board with the information they need to run the business, make better decisions, and take the right risks.

But risk management is not and never will be perfect.

It is impossible to monitor every risk, including new risks, in real time and provide useful information – also in real time – to the people who need to act on it.

There will always be risk champions who are new to the company and because they don’t understand the business and their risk-related responsibilities, will fail in that respect.

There will be times when the people required to provide expert insight when assessing and evaluating risks are on vacation, sick, or otherwise unable to participate.

There will always be a risk that the risk management program fails to provide the information necessary for decision-making.

The key is whether that risk is known and is considered acceptable.

If the risk is acceptable, then I would consider the risk management program as effective.

That is not to say that all the principles described in ISO 31000 are not necessary, or that the components discussed in COSO ERM are not required. But, that is the structure of the program and that doesn’t mean it is effective and produces the results necessary for the organization to succeed.

Bottom line: CROs and executive management should assess their risk management program (auditors can help) and determine whether the level of risk that it will provide insufficient information to run the business, make informed decisions, and take the right risks is acceptable.

OK, I understand that this is a little complicated and a very different way of thinking about effective risk management. Does it make sense?

I welcome your views.