ISACA Advanced in AI Audit

Governance sets the conditions for everything else. You assess the structures, roles and policies that govern AI, test whether they operate as described and advise management on implementation gaps.

• Audit criteria for AI governance
• Roles, committees and decision rights
• AI policy and ethical policy review
• Alignment of AI with strategy
• Inventory completeness of AI systems
• Advising management on implementation

You build An audit program for AI governance, with criteria, tests and expected evidence.

Audit plans follow risk. You identify the risks that AI brings, test whether ethical and regulatory obligations are met, and consider privacy and security as part of the same picture.

• AI risk identification for audit planning
• Ethical principles and how to test them
• AI regulation and compliance audits
• Privacy risks in AI systems
• Security risks in AI systems
• Rating AI audit risk

You build An AI risk assessment that drives an audit plan, with ratings and reasoning.

Data quality limits what a model can do. You audit how training and operating data is sourced, labeled, protected and retained, and test for quality, lineage and bias in what the model learned from.

• Data sourcing and rights to use data
• Data quality and labeling controls
• Lineage and provenance evidence
• Bias and representativeness testing
• Privacy and retention of AI data
• Access controls over data and features

You build A data audit checklist and sample test results for a training dataset.

How an AI system was built shapes its reliability. You audit development methods, model selection, documentation, change control and the use of third-party models, from requirements to release.

• Requirements and design documentation
• Development methods and model selection
• Versioning, reproducibility and change control
• Third-party and open-source components
• Release approval and segregation of duties
• Decommissioning and records

You build A life cycle audit program with tests for each stage and the evidence each should produce.

Assurance requires examining behavior, not only paperwork. You review validation reports, re-perform key tests, check robustness and fairness, and decide whether testing was enough for the use.

• Reviewing validation and test reports
• Re-performing performance tests
• Robustness and adversarial testing
• Fairness and explainability testing
• Test data adequacy
• Judging whether testing matched the risk

You build A test plan that re-performs validation on a sample AI classifier and records the results.

Controls after release decide whether problems are caught. You audit live monitoring, drift handling, human review and override, and the logs that show whether supervision actually happens.

• Monitoring of live AI performance
• Drift detection and retraining control
• Human review and override evidence
• Logs and traceability of decisions
• Handling complaints and appeals
• Production change control

You build An audit of a monitoring and oversight process for one live AI system, with findings.

AI systems attract attack and fail in new ways. You recognize threats such as poisoning, prompt injection and model theft, test the controls against them and assess incident readiness.

• AI-specific threats and vulnerabilities
• Access control and secrets around models
• Supply chain and dependency risks
• Incident response readiness for AI
• Resilience and fallback processes
• Testing security controls on AI systems

You build A security and incident readiness review of an AI application, with control tests.

An AI audit needs particular scoping decisions. You define objectives and criteria, plan a risk-based program, decide what skills and experts are required, and agree access to data and models.

• Audit objectives and criteria for AI
• Risk-based audit planning
• Scoping systems, models and data
• Skills, experts and independence
• Access to models, data and vendors
• Writing the audit program

You build An audit plan for a customer-facing AI system, with scope, criteria and resources.

Evidence from AI is unusual in kind. You collect and judge documentation, system outputs and logs, sample model decisions properly and document the work so another auditor could repeat it.

• Types and reliability of AI audit evidence
• Inquiry, observation, inspection and re-performance
• Sampling model outputs and decisions
• Testing data and model artifacts
• Working papers for AI audits
• Evidence when the vendor will not share

You build A sampling plan and working paper set for testing decisions made by an AI system.

Auditors can use AI themselves. You apply data analytics, continuous auditing and language-model assistants to audit work, and control the risks of confidentiality, error and over-reliance.

• Data analytics for audit populations
• Continuous auditing and monitoring
• Language-model assistants for audit tasks
• Validating AI-produced audit output
• Confidentiality and data handling
• Documenting the use of AI in an audit

You build A protocol for using AI in an audit, with validation steps and a worked example.

A finding helps only if the reader acts on it. You write findings with cause and consequence, rate them, agree actions with management and follow up until the risk is addressed.

• Structuring AI audit findings
• Root cause and impact
• Ratings and recommendations
• Reporting to management and audit committees
• Advisory and assurance engagements compared
• Follow-up and closure

You build An AI audit report with findings, ratings and agreed management actions.

One engagement covers every part. You plan an audit, test governance, data, development and operations, use analytics where they help and report findings to the audit committee.

• Scoping and planning the audit
• Testing governance and data controls
• Testing development, operations and security
• Using analytics and AI-enabled tools
• Reporting to the audit committee

You build A completed AI audit file: plan, working papers, findings and final report.