Testing the Design Effectiveness of Internal Controls: Test the Control Before You Test the Transactions

A Practical CPE Webinar for Internal Auditors, SOX Professionals, Compliance Teams and Internal Control Specialists

One of the most expensive mistakes in internal control testing is also one of the most common:

Testing whether a control operated before determining whether the control was properly designed in the first place.

An auditor can select 25 transactions, obtain 25 pieces of evidence, document 25 successful control performances—and still reach the wrong conclusion if the underlying control was never capable of reducing the identified risk to an acceptable level.

That is the fundamental distinction between design effectiveness and operating effectiveness.

Before asking:

Did the control operate?

the auditor should first ask:

If this control operates exactly as management designed it, will it actually address the risk?

Corporate Compliance Seminars' Testing the Design Effectiveness of Internal Controls CPE training focuses on developing that capability. The program addresses COSO, Sarbanes-Oxley compliance, entity-level controls, financial transaction processes, IT general controls, risk assessment, control design and the development of stronger testing methodologies.

Testing the Design Effectiveness of Internal Controls — Course Information and Registration

Design Effectiveness Comes Before Operating Effectiveness

Consider a simple Accounts Payable control:

"The Accounts Payable Manager reviews the weekly payment report."

That sounds like a control.

But is it an effectively designed control?

Before testing whether the manager actually performed the review, the auditor needs considerably more information.

What exactly does the manager review?

What risks is the review intended to address?

How does the manager identify an exception?

What information is contained in the report?

Is the report complete and accurate?

What dollar threshold is used?

What evidence demonstrates that the review occurred?

What happens when an exception is identified?

Could the individual preparing the payment also manipulate the report?

Could management override the process?

Those are design-effectiveness questions.

Until the auditor understands the answers, selecting samples may accomplish very little.

What Does "Design Effectiveness" Really Mean?

At its core, evaluating design effectiveness means determining whether a control—individually or together with other controls—is capable of preventing, or detecting and correcting, the risk it was established to address.

The sequence should therefore begin with the risk.

Objective → Risk → Control → Control Design → Operating Effectiveness

That order matters.

Weak control programs frequently reverse it:

Existing Control → Testing → Exception → Remediation

The organization becomes very good at testing controls without first asking whether it has the right controls.

That can turn SOX compliance into an expensive annual testing exercise rather than an effective system of internal control.

Start With the Objective

A control should exist for a reason.

Before evaluating a control, identify the objective.

For example:

Objective: Vendor payments are made only for authorized goods and services received by the company.

Now identify what could prevent achievement of that objective.

Potential risks might include:

• Payments to fictitious vendors

• Duplicate invoices

• Unauthorized purchases

• Payments for goods never received

• Incorrect quantities

• Incorrect prices

• Fraudulent changes to vendor banking information

• Employee conflicts of interest

• Management override

Only after identifying those risks can the auditor intelligently evaluate whether the controls are appropriately designed.

One Control Rarely Addresses Every Risk

Suppose management tells you:

"All invoices require approval."

That may sound reassuring.

But approval by whom?

Approval of what?

At what point?

Based on what information?

A manager signing an invoice may confirm that the invoice exists.

It may not establish that:

• The vendor is legitimate

• The goods were received

• The quantity is correct

• The price agrees with the purchase order

• The invoice hasn't already been paid

• The vendor's banking information is valid

• The approver has appropriate authority

The word "approval" does not establish control effectiveness.

Auditors must understand precisely what the control performer does.

The Walkthrough Is One of the Auditor's Most Powerful Tools

A walkthrough should not be a meeting where the auditor reads last year's narrative to management and asks whether anything changed.

A meaningful walkthrough follows the transaction.

The PCAOB describes a walkthrough as following a transaction from its origination through the company's processes and information systems until it reaches the financial records, using the same documents and technology company personnel use.

Walkthroughs generally combine inquiry, observation, document inspection and reperformance. The PCAOB also notes that probing questions during a walkthrough can help identify important points where a necessary control is missing or is not designed effectively.

That makes the walkthrough a critical design-effectiveness procedure.

Stop Asking Yes-or-No Questions

Consider these two approaches.

Weak Question

"Do you review new vendors before they're approved?"

Management answers:

"Yes."

You learned almost nothing.

Better Question

"Show me the last vendor you added to the system and walk me through everything that happened from the original request until the vendor became available for payment."

Now the auditor can observe the process.

Follow with questions such as:

"Who requested the vendor?"

"What information did you independently verify?"

"How did you verify the banking information?"

"Who approved the vendor?"

"Could the person creating the vendor also approve it?"

"What happens when something doesn't match?"

"Show me the evidence."

That is how auditors move from inquiry to evidence.

Design Effectiveness Requires Understanding the Precision of the Control

Not all reviews are equal.

Consider:

"The Controller reviews monthly financial results."

That could be a strong management-review control.

It could also be virtually meaningless.

The auditor needs to determine the precision at which the control operates.

Does the Controller investigate:

• A $5 million variance?

• A $500,000 variance?

• A 10% variance?

• Every unusual account?

• Only items that "look wrong"?

What information is reviewed?

How reliable is that information?

What constitutes an exception?

What follow-up occurs?

How is resolution documented?

A control described as a "monthly review" may be designed with sufficient precision to detect a material error—or it may be incapable of doing so.

The title of the control tells the auditor very little.

Who Performs the Control?

Design effectiveness also depends on the control owner.

Ask:

• Does the person have appropriate authority?

• Does the person understand the process?

• Does the person possess sufficient competence?

• Is the individual independent of the activity being reviewed?

• Does the person have access to reliable information?

• Could the person override the process?

• Is there an inappropriate segregation-of-duties conflict?

A beautifully documented control assigned to the wrong person can still be poorly designed.

Frequency Matters

A control may be appropriate but operate too infrequently to address the risk.

Suppose a high-volume process generates thousands of transactions every week.

Management performs a detailed review once each quarter.

The question is not simply whether the quarterly review occurs.

The auditor should determine whether a quarterly control can identify the relevant errors early enough to prevent or detect the potential misstatement or operational problem.

Control frequency must match risk.

Information Used in the Control Matters

Auditors increasingly need to evaluate information produced by the entity and other reports used by control owners.

Suppose management reviews an exception report.

That review is only as effective as the report.

The auditor should ask:

• Where does the data originate?

• Is the population complete?

• Are parameters correct?

• Can users modify the report?

• Are excluded transactions visible?

• Who controls report logic?

• Is the report generated from a controlled system?

A strong review performed over unreliable information is not a strong control.

Entity-Level Controls Can Change Everything

Design-effectiveness testing should not begin and end at the transaction level.

CCS's program specifically addresses the evaluation of entity-level controls, including their relationship to risk management and governance.

Auditors should consider areas such as:

• Management's philosophy

• Organizational accountability

• Ethics

• Board oversight

• Audit Committee oversight

• Risk assessment

• Authority and responsibility

• Competence

• Communication

• Monitoring

• Corrective action

A transaction-level control exists inside this larger control environment.

If senior management routinely overrides established controls, beautifully designed process controls may provide substantially less assurance than they appear to provide on paper.

COSO Gives Auditors the Architecture

The COSO Internal Control Framework provides the structure for thinking about controls as a system, rather than as an inventory of unrelated activities.

CCS's course specifically emphasizes applying COSO when designing and evaluating internal controls and understanding the relationship among internal controls, organizational objectives and risk.

This is an important conceptual shift.

The question isn't:

"Do we have 200 controls?"

The question is:

"Do we have a system of controls appropriately designed to address the risks threatening our objectives?"

More controls do not necessarily mean better control.

Sometimes they simply mean more testing.

SOX Programs Can Become Bloated When Design Is Ignored

One of the major opportunities in mature SOX programs is rationalization.

Organizations accumulate controls.

A new control is added after an audit finding.

Another is added after a system implementation.

Another appears following a management request.

Five years later, the company may be testing hundreds of controls without determining whether they remain necessary.

A design-effectiveness review can identify:

• Duplicate controls

• Redundant controls

• Controls addressing insignificant risks

• Controls operating at insufficient precision

• Controls that no longer address the process

• Controls superseded by technology

• Missing key controls

• Unnecessary manual controls

The objective should not be to test as many controls as possible.

The objective should be to identify and test the controls that matter.

IT General Controls Are Part of the Equation

Modern financial and operational controls depend heavily on technology.

CCS therefore incorporates IT general controls and technology considerations into the program.

Auditors need to consider controls involving:

• User access

• Privileged access

• Program changes

• System development

• Interfaces

• Automated controls

• Report generation

• Data integrity

A business control may appear well designed until the auditor discovers that users have inappropriate system access or that the underlying application can be modified without appropriate change controls.

Business controls and IT controls increasingly have to be evaluated together.

Design Deficiencies Should Be Fixed Before Extensive Operating Testing

This is where organizations can save substantial time.

Suppose the auditor determines during design testing that a control cannot adequately address the identified risk.

What value is created by testing 25 samples to prove that the ineffective control operated exactly as designed?

Very little.

The better sequence is:

• Identify the risk.

• Evaluate the control design.

• Identify the design deficiency.

• Redesign the control.

• Implement the improved control.

• Allow the control to operate.

• Then test operating effectiveness.

This is both better auditing and better compliance management.

Design Effectiveness and Operating Effectiveness Are Different Conclusions

Auditors should be able to explain the distinction clearly.

Design Effectiveness

Could the control, if performed as designed by appropriately qualified personnel, adequately address the identified risk?

Operating Effectiveness

Did the appropriately designed control actually operate as intended during the period being evaluated?

A control can therefore be: Well designed but not operating effectively.

Or:

Operating exactly as designed but still poorly designed.

That second possibility is why design testing must come first.

CCS offers separate training addressing operating effectiveness, reinforcing the distinction between these two stages of internal control evaluation.

Artificial Intelligence Creates New Opportunities for Design Testing

AI can help auditors become more effective during design evaluations.

An auditor can use an approved AI tool to help:

• Develop walkthrough questions

• Analyze process narratives

• Compare narratives with policies

• Identify potential control gaps

• Analyze segregation of duties

• Develop risk-and-control matrices

• Summarize walkthrough notes

• Identify inconsistencies

• Develop follow-up questions

• Compare controls against identified risks

But AI should not make the design-effectiveness conclusion.

That remains an auditor judgment.

AI can ask:

"Have you considered whether the person approving the vendor can also change the vendor's bank account?"

The auditor must determine whether that risk actually exists and whether the available evidence supports the conclusion.

A Better Way to Think About Internal Control Testing

The strongest auditors do not begin with the control.

They begin with the objective.

Then they ask:

• What could go wrong?

• How significant could it be?

• What prevents it?

• What detects it?

• Is that control appropriately designed?

• Who performs it?

• How frequently?

• Using what information?

• At what level of precision?

• What evidence demonstrates performance?

• Can management override it?

• What happens when an exception occurs?

Only then should the auditor ask: How should we test whether the control operated effectively?

That is risk-based internal control testing.

Who Should Attend?

CCS designed this program for professionals involved in designing, evaluating, testing or overseeing internal controls, including internal auditors, audit staff, compliance officers, risk management professionals and finance professionals. The program is classified at the Basic level, with no prerequisites or advance preparation required.

The subject matter is particularly valuable for professionals working with:

• COSO

• SOX compliance

• ICFR

• Internal Audit

• Risk management

• Compliance

• Entity-level controls

• Financial transaction controls

• IT general controls

Build Better Controls Before You Build Bigger Testing Programs

Internal control testing should never become an exercise in accumulating samples.

The number of controls tested does not determine the quality of an internal control program.

The number of samples selected does not determine the quality of an audit.

The real question is whether the organization has identified the important risks and designed controls capable of addressing them.

That requires judgment.

It requires risk assessment.

It requires good walkthroughs.

It requires understanding COSO.

And it requires auditors who know how to evaluate why a control exists, what it is supposed to accomplish and whether its design can actually accomplish it.

That is what Testing the Design Effectiveness of Internal Controls is designed to teach.