Testing the Design Effectiveness of Internal Controls: Test the Control Before You Test the Transactions
A Practical CPE Webinar for Internal Auditors, SOX Professionals, Compliance Teams and Internal Control Specialists
One of the most expensive mistakes in internal control testing is also one of the most common:
Testing whether a control operated before determining whether the control was properly designed in the first place.
An auditor can select 25 transactions, obtain 25 pieces of evidence, document 25 successful control performances—and still reach the wrong conclusion if the underlying control was never capable of reducing the identified risk to an acceptable level.
That is the fundamental distinction between design effectiveness and operating effectiveness.
Before asking:
Did the control operate?
the auditor should first ask:
If this control operates exactly as management designed it, will it actually address the risk?
Corporate Compliance Seminars' Testing the Design Effectiveness of Internal Controls CPE training focuses on developing that capability. The program addresses COSO, Sarbanes-Oxley compliance, entity-level controls, financial transaction processes, IT general controls, risk assessment, control design and the development of stronger testing methodologies.
Design Effectiveness Comes Before Operating Effectiveness
Consider a simple Accounts Payable control:
"The Accounts Payable Manager reviews the weekly payment report."
That sounds like a control.
But is it an effectively designed control?
Before testing whether the manager actually performed the review, the auditor needs considerably more information.
What exactly does the manager review?
What risks is the review intended to address?
How does the manager identify an exception?
What information is contained in the report?
Is the report complete and accurate?
What dollar threshold is used?
What evidence demonstrates that the review occurred?
What happens when an exception is identified?
Could the individual preparing the payment also manipulate the report?
Could management override the process?
Those are design-effectiveness questions.
Until the auditor understands the answers, selecting samples may accomplish very little.
What Does "Design Effectiveness" Really Mean?
At its core, evaluating design effectiveness means determining whether a control—individually or together with other controls—is capable of preventing, or detecting and correcting, the risk it was established to address.
The sequence should therefore begin with the risk.
Objective → Risk → Control → Control Design → Operating Effectiveness
That order matters.
Weak control programs frequently reverse it:
Existing Control → Testing → Exception → Remediation
The organization becomes very good at testing controls without first asking whether it has the right controls.
That can turn SOX compliance into an expensive annual testing exercise rather than an effective system of internal control.
Start With the Objective
A control should exist for a reason.
Before evaluating a control, identify the objective.
For example:
Objective: Vendor payments are made only for authorized goods and services received by the company.
Now identify what could prevent achievement of that objective.
Potential risks might include:
Payments to fictitious vendors
Duplicate invoices
Unauthorized purchases
Payments for goods never received
Incorrect quantities
Incorrect prices
Fraudulent changes to vendor banking information
Employee conflicts of interest
Management override
Only after identifying those risks can the auditor intelligently evaluate whether the controls are appropriately designed.
One Control Rarely Addresses Every Risk
Suppose management tells you:
"All invoices require approval."
That may sound reassuring.
But approval by whom?
Approval of what?
At what point?
Based on what information?
A manager signing an invoice may confirm that the invoice exists.
It may not establish that:
The vendor is legitimate
The goods were received
The quantity is correct
The price agrees with the purchase order
The invoice hasn't already been paid
The vendor's banking information is valid
The approver has appropriate authority
The word "approval" does not establish control effectiveness.
Auditors must understand precisely what the control performer does.
The Walkthrough Is One of the Auditor's Most Powerful Tools
A walkthrough should not be a meeting where the auditor reads last year's narrative to management and asks whether anything changed.
A meaningful walkthrough follows the transaction.
The PCAOB describes a walkthrough as following a transaction from its origination through the company's processes and information systems until it reaches the financial records, using the same documents and technology company personnel use.
Walkthroughs generally combine inquiry, observation, document inspection and reperformance. The PCAOB also notes that probing questions during a walkthrough can help identify important points where a necessary control is missing or is not designed effectively.
That makes the walkthrough a critical design-effectiveness procedure.
Stop Asking Yes-or-No Questions
Consider these two approaches.
Weak Question
"Do you review new vendors before they're approved?"
Management answers:
"Yes."
You learned almost nothing.
Better Question
"Show me the last vendor you added to the system and walk me through everything that happened from the original request until the vendor became available for payment."
Now the auditor can observe the process.
Follow with questions such as:
"Who requested the vendor?"
"What information did you independently verify?"
"How did you verify the banking information?"
"Who approved the vendor?"
"Could the person creating the vendor also approve it?"
"What happens when something doesn't match?"
"Show me the evidence."
That is how auditors move from inquiry to evidence.
Design Effectiveness Requires Understanding the Precision of the Control
Not all reviews are equal.
Consider:
"The Controller reviews monthly financial results."
That could be a strong management-review control.
It could also be virtually meaningless.
The auditor needs to determine the precision at which the control operates.
Does the Controller investigate:
A $5 million variance?
A $500,000 variance?
A 10% variance?
Every unusual account?
Only items that "look wrong"?
What information is reviewed?
How reliable is that information?
What constitutes an exception?
What follow-up occurs?
How is resolution documented?
A control described as a "monthly review" may be designed with sufficient precision to detect a material error—or it may be incapable of doing so.
The title of the control tells the auditor very little.
Who Performs the Control?
Design effectiveness also depends on the control owner.
Ask:
Does the person have appropriate authority?
Does the person understand the process?
Does the person possess sufficient competence?
Is the individual independent of the activity being reviewed?
Does the person have access to reliable information?
Could the person override the process?
Is there an inappropriate segregation-of-duties conflict?
A beautifully documented control assigned to the wrong person can still be poorly designed.
Frequency Matters
A control may be appropriate but operate too infrequently to address the risk.
Suppose a high-volume process generates thousands of transactions every week.
Management performs a detailed review once each quarter.
The question is not simply whether the quarterly review occurs.
The auditor should determine whether a quarterly control can identify the relevant errors early enough to prevent or detect the potential misstatement or operational problem.
Control frequency must match risk.
Information Used in the Control Matters
Auditors increasingly need to evaluate information produced by the entity and other reports used by control owners.
Suppose management reviews an exception report.
That review is only as effective as the report.
The auditor should ask:
Where does the data originate?
Is the population complete?
Are parameters correct?
Can users modify the report?
Are excluded transactions visible?
Who controls report logic?
Is the report generated from a controlled system?
A strong review performed over unreliable information is not a strong control.
Entity-Level Controls Can Change Everything
Design-effectiveness testing should not begin and end at the transaction level.
CCS's program specifically addresses the evaluation of entity-level controls, including their relationship to risk management and governance.
Auditors should consider areas such as:
Management's philosophy
Organizational accountability
Ethics
Board oversight
Audit Committee oversight
Risk assessment
Authority and responsibility
Competence
Communication
Monitoring
Corrective action
A transaction-level control exists inside this larger control environment.
If senior management routinely overrides established controls, beautifully designed process controls may provide substantially less assurance than they appear to provide on paper.
COSO Gives Auditors the Architecture
The COSO Internal Control Framework provides the structure for thinking about controls as a system, rather than as an inventory of unrelated activities.
CCS's course specifically emphasizes applying COSO when designing and evaluating internal controls and understanding the relationship among internal controls, organizational objectives and risk.
This is an important conceptual shift.
The question isn't:
"Do we have 200 controls?"
The question is:
"Do we have a system of controls appropriately designed to address the risks threatening our objectives?"
More controls do not necessarily mean better control.
Sometimes they simply mean more testing.
SOX Programs Can Become Bloated When Design Is Ignored
One of the major opportunities in mature SOX programs is rationalization.
Organizations accumulate controls.
A new control is added after an audit finding.
Another is added after a system implementation.
Another appears following a management request.
Five years later, the company may be testing hundreds of controls without determining whether they remain necessary.
A design-effectiveness review can identify:
Duplicate controls
Redundant controls
Controls addressing insignificant risks
Controls operating at insufficient precision
Controls that no longer address the process
Controls superseded by technology
Missing key controls
Unnecessary manual controls
The objective should not be to test as many controls as possible.
The objective should be to identify and test the controls that matter.
IT General Controls Are Part of the Equation
Modern financial and operational controls depend heavily on technology.
CCS therefore incorporates IT general controls and technology considerations into the program.
Auditors need to consider controls involving:
User access
Privileged access
Program changes
System development
Interfaces
Automated controls
Report generation
Data integrity
A business control may appear well designed until the auditor discovers that users have inappropriate system access or that the underlying application can be modified without appropriate change controls.
Business controls and IT controls increasingly have to be evaluated together.
Design Deficiencies Should Be Fixed Before Extensive Operating Testing
This is where organizations can save substantial time.
Suppose the auditor determines during design testing that a control cannot adequately address the identified risk.
What value is created by testing 25 samples to prove that the ineffective control operated exactly as designed?
Very little.
The better sequence is:
Identify the risk.
Evaluate the control design.
Identify the design deficiency.
Redesign the control.
Implement the improved control.
Allow the control to operate.
Then test operating effectiveness.
This is both better auditing and better compliance management.
Design Effectiveness and Operating Effectiveness Are Different Conclusions
Auditors should be able to explain the distinction clearly.
Design Effectiveness
Could the control, if performed as designed by appropriately qualified personnel, adequately address the identified risk?
Operating Effectiveness
Did the appropriately designed control actually operate as intended during the period being evaluated?
A control can therefore be: Well designed but not operating effectively.
Or:
Operating exactly as designed but still poorly designed.
That second possibility is why design testing must come first.
CCS offers separate training addressing operating effectiveness, reinforcing the distinction between these two stages of internal control evaluation.
Artificial Intelligence Creates New Opportunities for Design Testing
AI can help auditors become more effective during design evaluations.
An auditor can use an approved AI tool to help:
Develop walkthrough questions
Analyze process narratives
Compare narratives with policies
Identify potential control gaps
Analyze segregation of duties
Develop risk-and-control matrices
Summarize walkthrough notes
Identify inconsistencies
Develop follow-up questions
Compare controls against identified risks
But AI should not make the design-effectiveness conclusion.
That remains an auditor judgment.
AI can ask:
"Have you considered whether the person approving the vendor can also change the vendor's bank account?"
The auditor must determine whether that risk actually exists and whether the available evidence supports the conclusion.
A Better Way to Think About Internal Control Testing
The strongest auditors do not begin with the control.
They begin with the objective.
Then they ask:
What could go wrong?
How significant could it be?
What prevents it?
What detects it?
Is that control appropriately designed?
Who performs it?
How frequently?
Using what information?
At what level of precision?
What evidence demonstrates performance?
Can management override it?
What happens when an exception occurs?
Only then should the auditor ask: How should we test whether the control operated effectively?
That is risk-based internal control testing.
Who Should Attend?
CCS designed this program for professionals involved in designing, evaluating, testing or overseeing internal controls, including internal auditors, audit staff, compliance officers, risk management professionals and finance professionals. The program is classified at the Basic level, with no prerequisites or advance preparation required.
The subject matter is particularly valuable for professionals working with:
COSO
SOX compliance
ICFR
Internal Audit
Risk management
Compliance
Entity-level controls
Financial transaction controls
IT general controls
Build Better Controls Before You Build Bigger Testing Programs
Internal control testing should never become an exercise in accumulating samples.
The number of controls tested does not determine the quality of an internal control program.
The number of samples selected does not determine the quality of an audit.
The real question is whether the organization has identified the important risks and designed controls capable of addressing them.
That requires judgment.
It requires risk assessment.
It requires good walkthroughs.
It requires understanding COSO.
And it requires auditors who know how to evaluate why a control exists, what it is supposed to accomplish and whether its design can actually accomplish it.
That is what Testing the Design Effectiveness of Internal Controls is designed to teach.
Comments