Home > Risk > Is your SOX program both effective and efficient?

Is your SOX program both effective and efficient?

Protiviti’s surveys and reports are always worth reading. One I look forward to is their annual survey on SOX compliance.

Those of you who are responsible for the SOX program or SOX testing at their organization are likely to find the benchmarking info in the 2019 survey, Benchmarking SOX Costs, Hours and Controls of interest.

However, I want to share (again) a note of caution.

Protiviti and others are talking about the use of analytics and other tools, such as RPA, for SOX testing.

But, the purpose of the SOX testing is to:

  • Confirm that the design of the controls relied upon to prevent or detect a material error or omission in the financial statements filed with the SEC are sufficient, if they are operated as designed, to address such a possibility. The likelihood of a material error or omission is less than reasonably possible.
  • Confirm, with a reasonable level of assurance, that those controls are being performed consistently as designed.

The end product is an assessment as to whether the system of internal control over financial reporting is effective; that means that the controls are sufficient to provide reasonable assurance that a material error or omission would be prevented or detected.

What do these newer technology tools do for us?

For the most part, they provide some level of assurance that the data, and possibly the transactions, are free from error.

But do they provide any assurance that the system of internal control is effective?

While the presence of errors is a strong indicator that the controls are not sufficient, the absence of errors is not a strong indicator that the controls are effective!

The data may be free from error even though the controls are not being performed at all!

In my SOX training classes (the next one is in October), I ask the attendees how many of them have had their homes burglarized in the last year. Only on the rare occasion has anybody raised their hand.

I then ask whether the fact that they have not been burglarized is proof that they locked all the doors and windows before they left the house.

I remember one time in England when, as an IT auditor, I was flowcharting and identifying controls in a very complex integrated system. One of the controls that management had identified was a comparison between data at one point in the system to the data at a much later point (a “run to run” control). When I examined the logic of the program that did the comparison, I found that it was coded incorrectly. At each point, early (file E) and late (file L), a file was created that could be compared. But the comparison program was comparing data in file E to data on file E – instead of file L.

The control was doing nothing. But the data happened to be clean anyway (we checked).

So, when it comes to the use of technology tools, will they provide the evidence you need that the controls relied on are both adequately designed and operated? Do they test the controls or only the data?

My second note of caution is to remain focused on whether the system of internal control over financial reporting provides reasonable assurance that material errors will either be prevented or detected. That refers to the possibility of errors in the consolidated financial statements filed with the SEC.

Too many, typically under pressure from the external auditors, are adding controls without asking whether they are needed to prevent or detect a material error.

                WHERE’S THE RISK?

The scope does not, and typically should not, include controls that would never result in material weaknesses should they fail. It’s not a matter of whether they are important controls, or required to address the risk-du-jour. It’s a matter of whether they are being relied upon to prevent or detect a material error in the filed financials.

One final point: I don’t care how many ‘entity-level’ controls you have. I only care whether you have selected the right controls to include in scope.  By ‘right’ I mean the combination of controls that can be relied on to function consistently and address the risk of a material error, and are efficient to operate and test.

I welcome your thoughts.

  1. Daniel Kalwiji
    July 21, 2019 at 5:39 PM

    What I am getting is that the audit should align the audit procedures to the overal audit objectives in order to secure the level of assurance needed.
    Audit work performed beyond the scope must also be identified and reported as extra client service.
    The objective of assessing controls should be clearly understood and stated by auditors.
    Assessing controls is process that includes obtaining management assertions controls and obtaining the required assurance.

  2. July 21, 2019 at 11:22 PM

    In the Netherlands, tools for proces mining are being used (still on a small scale). They provide assurance wheter controls perform as designed. E.g. on if 3-way matching between order, receipt and invoice is consistantly applied.

    • Norman Marks
      July 22, 2019 at 6:13 AM

      If they simply confirm that the invoice matches the purchase order and receiving document, they are testing the data but not the control. They can match without any control operating at all.

      • July 23, 2019 at 7:05 AM

        Maybe I oversimplified the example. The tooling I mean will show who was actually responsible for performing the different taks and if violations to SOD occurr, if tolerances are set and correcttly being applied or approved by the right manager, etc.. Imo this is not testing the data but testing the controls.

        • Norman Marks
          July 23, 2019 at 7:09 AM

          I have seen situations where the system writes a record each time a control is performed. For example, a bank reconciliation in SAP. If that is what the analytics are doing, then I concur. It is testing the control. Have to be careful that is testing more than that the control was performed; it has to test whether the control operated as designed and addresses the risk.

  3. John Parsons
    July 22, 2019 at 10:57 AM

    I appreciate your comments, and calling attention to the Protiviti analysis of SOX trends this year. I agree that if you only test that the calculations and summarization were correct, then you have not tested the control (the example of 3-way payable match). I do not agree that all technological analysis tools, such as RPA or advanced analytics are limited to only comparison of data. Real-time monitoring of controls can help fulfill COSO Principle 16, for example in validating the operation of a preventative control such as only authorized users may perform certain transactions or system changes. This type of ‘meta’ control would detect unauthorized changes to the functioning of other controls.

    • Norman Marks
      July 22, 2019 at 2:03 PM

      I said that the generally only test data. If they are able to test the existence and operation of controls, then I concur. But those situations are unusual.

  4. John Verver
    August 9, 2019 at 10:07 AM

    A few thoughts come to mind when considering the question as to whether data analytics “provide the evidence you need that the controls relied on are both adequately designed and operated?”
    One of the issues is that controls are never going to be perfectly effective – they are always prone to some form of failure and workaround (e.g. someone in AP deliberately miss-keys a supplier invoice number to circumvent a duplicate payment control setting in an ERP, a manager splits a P.O. into two or more parts to circumvent their approval limit control). The chances are that if someone tried to actually design and implement a perfect “bullerproof” control it would render a process far too unwieldy to be workable. Running a range of analytics against transactional data can directly provide an increased level of assurance that the risks of bad stuff happening are being managed effectively – as well as that controls at least appear to be doing their job, despite their potential weaknesses.
    Something else analytics can do is help determine whether controls have been designed and implemented to address risks that have never previously been considered. Analytics, for example, can reveal the existence of new situations and new types of transactions for which there are no specific controls – allowing for these previously unknown risks to be addressed through appropriate controls.
    And, of course, at some point analytics can also arguably become a control to the extent that, in some cases, they can be used to prevent bad stuff from happening in the first place, or at least detect when something bad has happened and bring it to light before it reoccurs and grows in magnitude.
    Protiviti is one of the few services firms that have done a very good job in speaking on the potential benefits of increased use of data analytics across risk management, compliance and audit functions – though in practice very few organizations are as yet beyond scratching the surface in their use of analytics in these areas.

    • Norman Marks
      August 9, 2019 at 10:16 AM

      John, thanks for the comment.Trust you are well.

      I agree that analytics can be very useful indeed. I love them as detective controls, but they can also be helpful in understanding what is going on – and thereby understanding changes in risk and the need for specific controls.

      My point is that they are rarely useful in demonstrating that controls are operating effectively as designed. I can only think of a few situations, where the data includes a record that a control was performed, by whom, and when. The control still needs to be reperformed as the analytics in that situation only confirms the control was performed.

  1. July 22, 2019 at 4:33 AM

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: