AI-Specific Audit Evidence: What Documentation Should Auditors Request Under ISO 42001?

ISO 42001 audits require evidence types that differ significantly from traditional IT audits. Understanding what documentation to request—and what constitutes sufficient evidence—is essential for auditors assessing AI Management Systems.

Categories of AI-Specific Evidence

Evidence for ISO 42001 audits falls into several categories:

• Governance Documentation: Policies, procedures, and organizational structures
• Technical Artifacts: Model cards, test results, system documentation
• Operational Records: Logs, metrics, incident reports
• Assessment Records: Risk assessments, impact assessments, vendor evaluations
• Training and Competence: Training materials, attendance records, competence evaluations

Model Cards and System Documentation

Model cards are a critical evidence type unique to AI audits. They provide standardized documentation about ML models.

What Model Cards Should Include:

• Model Details: Name, version, type, architecture
• Intended Use: Primary use cases, intended users, out-of-scope uses
• Training Data: Data sources, preprocessing steps, data characteristics
• Performance Metrics: Accuracy, precision, recall, F1 scores
• Limitations: Known weaknesses, failure modes, edge cases
• Ethical Considerations: Fairness metrics, bias assessment results, potential harms
• Maintenance: Update schedule, monitoring approach, deprecation criteria

Audit Questions:

• Do model cards exist for all production AI systems?
• Are they maintained and version-controlled?
• Do they reflect current model state?
• Are they accessible to appropriate stakeholders?

AI Risk Registers

AI risk registers document identified risks, assessments, and treatment decisions.

Required Elements:

• Risk Identification: Comprehensive catalog of AI-related risks
• Risk Assessment: Likelihood and impact ratings for each risk
• Risk Owner: Assigned accountability for each risk
• Treatment Decision: Accept, mitigate, transfer, or avoid
• Treatment Actions: Specific controls or measures applied
• Residual Risk: Remaining risk after treatment
• Review Status: Last review date and next review scheduled

AI-Specific Risk Categories to Verify:

• Algorithmic bias and discrimination
• Model drift and performance degradation
• Lack of explainability
• Data quality and provenance issues
• Adversarial attacks
• Unintended consequences
• Third-party AI dependencies
• Regulatory compliance risks

AI System Impact Assessments

Impact assessments evaluate potential consequences of AI deployment beyond traditional risk assessment.

Required Documentation:

• System Description: What the AI system does and how
• Intended Use: Primary purposes and beneficiaries
• Foreseeable Misuse: How the system could be misused
• Affected Parties: Individuals, groups, or communities affected
• Impact Categories: Societal, environmental, ethical, economic impacts
• Mitigation Measures: Controls to reduce negative impacts
• Monitoring Approach: How impacts will be tracked post-deployment

Bias Testing Documentation

Evidence of bias testing and fairness assessment is crucial for demonstrating responsible AI practices.

Documentation Requirements:

• Fairness Metrics: Which metrics are used and why
• Protected Attributes: Demographics or characteristics assessed
• Thresholds: Acceptable limits for bias metrics
• Test Results: Actual measurements with timestamps
• Remediation: Actions taken when bias detected
• Ongoing Monitoring: How bias is tracked in production

Common Fairness Metrics:

• Demographic parity
• Equalized odds
• Predictive parity
• Individual fairness
• Calibration across groups

Data Provenance Records

Data provenance documentation traces data from source through transformation to model training.

Required Information:

• Data Sources: Where data originated
• Collection Methods: How data was gathered
• Legal Basis: Authority for data use
• Transformations: All processing steps applied
• Quality Checks: Validation performed
• Approvals: Authorization for use in AI training
• Retention: How long data is kept

Human Oversight Evidence

Documentation demonstrating appropriate human involvement in AI operations.

Required Records:

• Intervention Triggers: Documented thresholds and conditions
• Override Logs: Records of human interventions
• Decision Rationale: Why overrides were made
• Outcome Tracking: Results of human decisions
• Authority Matrix: Who can override what
• Training Records: Evidence of oversight training

Third-Party AI Assessment Records

Documentation of due diligence for external AI components.

Assessment Documentation:

• Vendor Questionnaires: AI governance questions and responses
• Security Assessments: Technical security evaluation
• Bias Assessments: Fairness documentation from vendor
• Contract Terms: AI-specific clauses and SLAs
• Ongoing Monitoring: Vendor performance tracking
• Incident Coordination: Communication procedures

Event Logs

Comprehensive logging across the AI system lifecycle.

Required Log Attributes:

• Timestamp: When the event occurred (tamper-proof)
• Actor: Who or what initiated the action
• Action: What was done
• Object: What was affected
• Outcome: Result of the action
• Context: Relevant circumstances

Lifecycle Phases to Log:

• Design decisions and approvals
• Data collection and processing
• Model training and validation
• Testing results
• Deployment approvals
• Production changes
• Incidents and responses

Evidence Collection Best Practices

For Auditors:

1. Request in Advance: Provide evidence list before audit
2. Sample Systematically: Select representative samples
3. Verify Authenticity: Check timestamps, signatures, version control
4. Cross-Reference: Validate consistency across documents
5. Interview Corroboration: Confirm documents match practice

For Organizations:

1. Centralize Documentation: Maintain organized, accessible repository
2. Version Control: Track document history and changes
3. Regular Updates: Keep documentation current
4. Access Controls: Protect sensitive evidence
5. Retention Policies: Maintain required retention periods

Conclusion

AI-specific audit evidence differs substantially from traditional IT audit documentation. Understanding what to request and how to evaluate it enables auditors to effectively assess ISO 42001 compliance while helping organizations prepare appropriate documentation.