Testing data vs. testing controls
In a recent post of his on LinkedIn, Joseph Kassapis wrote:
I was reading a typically excellent blog/post of Norman Marks on Control Testing (in the context of commenting on 2 reports on SOX Controls Testing), and was struck and intrigued by his insistence/emphasis on testing “Data” in the mistaken impression that this amounts to testing the Control(‘s effectiveness). He named this twice in his post as a fallacy/defect in the reports, and it instantly caught my attention, being something I always found extremely interesting and important: to what extent correct output can be taken to mean/evidence correct mechanism.
…
External Audit standards, as I fairly confidently recall/understand, expressly preclude this position, i.e. state that the correctness of the recorded transactions, as regards their aspects controlled by the control, can in no way and under no circumstances be taken as evidence of soundness/effectiveness of the control; and I sort of ‘resented’ this, regretted it, wished it was not there; without actually being able to really/genuinely fault it, logically; rather minding its being inconvenient, making things harder, depriving us of easy tests and forcing us to conceive harder ones, (towards the already very hard task/goal of attaining satisifaction of effective functioning of Control), easier said than done !
…
Nobody else seems to, elaborate either, on this very important principle. Nobody seems to take it up. Except, it seems, Norman Marks. In the sense that at least he does consider it is there, it is important, and it is grossly abused. I was badly hoping he would go on to elaborate, in this blog pot, but he didn’t.
I don’t know if he elaborated elsewhere. He can inform/refer us. Whether or not he did, in the past, I would dare invite/provoke/challenge him to do so now. With another, dedicated post. Enlightening us. As he always does.
OK, Joseph. Here we go.
I start with a premise: our objective is to obtain reasonable assurance that the controls relied upon to manage the risk (whether SOX and ICFR, or some other business risk) are (a) adequately designed and (b) operating effectively as designed.
In other words, we are performing an audit of the system of internal controls for that risk.
The situation is different if we are trying to validate that the data (or information, such as in a report) is complete and accurate.
The value of an opinion on the system of internal control is that it provides continuing assurance, while validating the data provides point in time assurance. Validating the data or the information in a report may confirm that that instance of the report is complete and accurate, but it doesn’t tell you that the next instances will be. For that, you either have to continue testing and validating each instance or rely on the system of internal controls.
The quality of assurance is different. An opinion on the system of internal controls only provides reasonable assurance that each instance is complete and accurate, whilst validating data provides more absolute assurance that the data is correct.
Now, let’s return to the challenge.
I have been leading a SOX Masters class for many years, usually multiple times each year. In that class, I ask participants:
“Has your home been burglarized in the last five years or so?
In all that time, only one person raised their hand. (Good news.)
I then ask:
“Does that prove you always closed and locked your doors and windows every time you left home?”
(I don’t even go so far as to ask whether they set the alarm.)
They smile ruefully, very much aware that they have failed to do so: their controls were not operating effectively, yet they did not have an incident (or data exception, if you like).
Consultants are pushing the notion that you can use analytics and other methods like AI and RPA to test controls.
There are very few opportunities to do so, as these techniques may provide some level of assurance that the data is free of error (if not always omissions). But they rarely provide acceptable evidence that the controls management have in place even exist, let alone are adequately designed and operating effectively.
Taking another example.
The city of San Jose, my hometown, has implemented a number of controls to limit accidents at busy intersections. They include:
- Traffic lights
- Lane and other street markings
- Periodic police visits
- Reliance on controls performed by others, such as DMV’s driver licensing controls
If you ran analytics and found that there were no accidents reported at the intersection of Stevens Creek Boulevard and Winchester Boulevard in 2022, does that prove that any of the controls were working?
No. I can tell you that there were times when the lights did not work but drivers exercised appropriate caution.
While detecting that there were incidents may indicate that controls were not working (more work needs to be done to confirm that), the lack of exceptions does not provide assurance that controls were in place, adequately designed, or operating effectively.
I hope that helps.
By the way, the intersection example illustrates another issue that many don’t understand.
The system of internal control only provides reasonable assurance. It does not provide absolute or perfect assurance.
COSO’s Internal Control Framework provides some examples of the limitations, but there is more.
When you test internal controls, you may find exceptions.
For example, you inspect the traffic lights and find that they were inoperative for a few hours on one day.
If that only happened once over a period of a year, I would call that an “isolated incident”. It is reasonable to accept the occasional breakdown.
But if it happened several times in a month, I would call it a “control breakdown”.
You can have effective internal controls despite isolated incidents, but not when there have been control breakdowns.
That is why when we find exceptions we need to expand the sample size to determine whether we have an isolated incident, which would acceptable, or a control breakdown – when we would assess that the control has failed to operate effectively as designed.
I welcome your comments.
Leave a comment
Norman Marks on Governance, Risk Management, and Internal Audit 
What is sad is so many external and internal auditors do not know and understand these concepts. It seems like executive presence, ability to communicate and network, getting exposure for the next job, and a multitude of other attributes overwhelm the need to be technically competent. Technical competency is an attribute I see as insufficient by it self to make a great auditor, but it is a required baseline.
Excellent reasoning why a detailed transaction/data test with no/few exceptions would not mean the Control is effective or even there. I remember well the very good analogy with the burgling, and now note the equally to-the-point traffic light one.
But let us separate the issue in 2:
-A. Transaction/Data testing is NOT Control(Effectiveness) Testing
-B. The Testing effected by/with Software (Analytics/AI/RPA/…) applications is largely/mostly Transaction testing*
*NM: “Consultants are pushing the notion that you can use analytics and other methods like AI and RPA to test controls.
There are very few opportunities to do so, as these techniques may provide some level of assurance that the data is free of error (if not always omissions). But they rarely provide acceptable evidence that the controls management have in place even exist, let alone are adequately designed and operating effectively”.
On (B), is it self evident, something to be taken for granted, an indisputable & undisputed fact, or a discuss-ible view, an open question ?
I ask ‘neutrally’ and innocently, as one who does not know, have the experience.Is it so, and if yes why ? Why do software developers/sellers say their solutions test Controls when in fact they test Transactions/Data ? And why, in the first place, they make their software – purported to be SW for testing Controls – test Transactions/Data (T/D), rather than Controls ? Is it a case of trying, knowingly, to present applications made principally for ‘substantive’ (T/D) testing as also SW for Control Testing, i.e. two/both in one ? Is there an inherent difficulty in producing SW that can test Controls ? If so which/what ? – something to do with the nature of Control Testing, lending itself less for computer testing … , or … ?
I would appreciate comments elucidating this aspect (B). I.e. that applications that can/do test T/D are being presented by their suppliers as applications that test/also test Controls.
I mean, Norman, this is the US, and we are talking SarOx, which has been around decades and – as I understand, from my so different other side of the world (in more than one senses, size of country, as well as place on planet) – is a very important matter; and Control-Effectiveness testing is in its own right, with or without SarOx, before & after it, a very important notion, conceptually but more matteringly practically. And the US is the country where “solution” became literally synonymous with IT/SW a very long time ago.
HOW COME, then, software houses, in the most mature/advanced market in the world, Auditing & IT-wise, and Control-mentality-wise, still commit such a major blunder/misrepresentation … ?
Thanks.
So much. For the above post already, heeding my ‘request’.
And any further comments you may kindly make, in response to my present comment/request.
Joseph, these people don’t understand what they are saying. There are a few situations where you can use software to test controls. For example, you can use it to examine configuration settings. The trouble is you also need to know whether there are controls over the configuration settings.
Bottom line is to use your common sense and judgment. Do I have the required assurance on the existence, design, and operation of the controls?
I can see where this can get complex. But if you don’t lock your doors and windows you don’t have a control or risk problem. You simply don’t have an objective to prevent break ins. Whether that’s good or bad depends on circumstances. But it is useless to consider risks and controls if an objective is absent. Report the missing objective as evidence of failure.
If an objective exists, performance against the objective is assurance isn’t it? Incident reporting, would be necessary for such a conclusion. Any conclusion on control effectiveness based only on evidence that a control exists at a point in time is risky. Then there is the fact that most errors are a result of human failure. Controls must be linked to root cause of failure which is absent from most analysis. I’ve come to the conclusion that only reliable performance in value adding activities is useful assurance and that data is indeed control. The audit universe should be rated by performance, not risk.
Wow, didn’t expect to disagree with you so much!
1. You can have an objective without controls or any assurance of achieving it.
2. You can be successful with no controls.
3. You do need assurance of continuing operation of controls – agree
4. Data is certainly not control – unless management is monitoring it a a detective control
I’d also add that (successful) past performance against an objective is not necessarily sufficient assurance over the effectiveness of controls. It may simply be as a result of the threat not being experienced and a control not needing to operate.
If businesses want absolute assurance, then it is imperative that they have to test or review all the data. Who wants to do that? Even no errors or omissions upon reviewing all the transactions, would it mean that the systems of internal controls are reasonably adequate? Now, there were changes in the staffing of those involved in the processing of transactions and limited testings were done on the premise that previous testings proved reasonable assurance. Would that reduced level of testing, be these prove no errors or omissions, provide reasonable assurance by merely testing the data and NOT the systems of internal control?
It depends on what you are trying to do. Are you trying to provide assurance on the data or on the controls?
Thanks for this post Norman. Personally, I think there are still many, many people, both in management and in auditing, that do not truly understand control testing versus substantive testing. I have seen CPA firm work-papers that use management inquiry to define the control and then substantively test a sample of the data and call it effective (not to mention testing where inquiry was the only step used and then deemed the control effective). On the internal side, I’ve also heard numerous times from managers that if no errors were found then why do we have an issue? I assume that for public companies SOX and the requirements therein improve this, but for private companies, non-profits, etc. I don’t think much true control testing is even occurring. Even at the highest levels (board/audit committee) it seems to be human nature to assume that if no specific errors were found then it indicates everything (controls) is humming along just as planned. I also think that the software issue adds to the confusion. What so many of these AI/RPA functions are actually doing is automating your control, not testing it. I would love for management to put in place a fancy automated monitoring control and then I can simply test that it was properly configured and functioning during the period under scope, but I can’t run the control for management and then call it testing. I also found your two examples to be very useful; I would love to borrow those in the future when trying to explain internal controls and control testing.
One last thought to provoke some additional conversations… If management and the board don’t understand control testing and are okay with getting their assurance through substantive testing only (and assuming there are no regulatory requirements like SOX in play), then does it matter?
Thanks, IA Paul.
On your last thought, two points to consider:
1. Their assurance is a chimera. An unwelcome surprise is coming!
2. Internal audit is taking a huge risk by informing the board and management that controls that may not even exist are operating effectively.
I’m surprised as well. I’m not sure I understand your responses but as I indicated it can get complex and nuanced. My point is that decision makers make decisions by relying on data all the time. If a pilot looks at his/her dashboard and sees the plane is at 30,000 feet, and should be at 20,000 feet, he/she makes a decision. I was not responding to a situation in the parallel universe of SOX account balances. Its 2022. Data drives decisions. Decisions drive outcomes. Achieving expected performance consistently is evidence of good data the data drives decisions. I’m all for periodic testing of instruments or using parallel systems. But if signed SOX type certificates are mandated in my seat back pockets, I will worry. I’d love a debate but this isn’t the best forum.
All the best.
Bruce, my post was about obtaining assurance on controls for SOX. I think we are in different debates.
Best
It is possible to make (accounting)systems and reports that make the impression all figures are correct. But if you never check that the goods represented in stock are physically there (and not transported between locations all the time), all can be fake. So both kinds of testing – data and controls – are needed. See Enron.
What is the root cause of that problem and what “control” will solve it? Not detect. Solve. I’m not sure I have a complete answer. I am sure SOX is at best an incomplete and unreliable one.
Bruce, for SOX purposes, a control that detects a material error so it can be corrected before filing the financials is sufficient.
That is why you need controls over the existence of the assets. No problem here, guys.
Here is what External Audit Standards (ISA) (ISA 330) say, about the matter:
“Evaluating the Operating Effectiveness of Controls
16. When evaluating the operating effectiveness of relevant controls, the auditor shall evaluate whether misstatements that have been detected by substantive procedures indicate that controls are not operating effectively. The absence of misstatements detected by substantive procedures, however, does not provide audit evidence that controls related to the assertion being tested are effective.”
I guess, given the much longer history and thus depth of EA Standards, IA Standards could hardly argue otherwise.
Hello, Norman,
Enjoyed the article, thank you.
I would like to draw your attention to a certain methodological aspect. I might overestimate its importance but still. I leave it up to your consideration.
The thing is that you as well as many others use the word “control” a bit too carelessly. For example, in this article you say:
“The city of San Jose, my hometown, has implemented a number of controls to limit accidents at busy intersections. They include:
– Traffic lights.
– Lane and other street markings.
– Periodic police visits.
– Reliance on controls performed by others, such as DMV’s driver licensing controls”.
In fact, this list contains not controls but process elements to successfully achieve the goal of the process of passing intersection. The most common goal for this process may be put as to pass the intersection in a reasonable time, safely (no injury as a mark of quality), at a minimal cost. Traffic lights, lanes and other street markings are a part of the process infrastructure (the enablers to be exact). Periodic police visits are a part of the process management infrastructure (the guides to be exact). I have no comments on the “controls performed by others” as I have no idea what it’s all about.
To summarize.
You say “control”, but depending on the context this may mean different things:
1. Measures to manage a risk (in the example above this could be fines for using phones while driving).
2. Business process elements and improvements to ensure its goal achieved in a preferred way (as in the example above; we may add some more – regular cleaning to make the infrastructure clearly visible or teaching kids at schools on how to properly pass intersections).
3. The act of matching the actual result with the planned/expected result (in the example above the police could perform the visual inspection to spot damaged or malfunctional infrastructure).
The third variant is the only true meaning of the word “control” in respect to management as a science and art.
I am not fully aware whether this distinguishment is worth considering in a native English speaking and professional environment. But for other environments juggling with the word “control” really creates a great deal of mess and confusion multiplied by translation quirks.
Would appreciate your comment.