As ISO 42001 certifications become more common, patterns are emerging in the types of nonconformities auditors frequently identify. Understanding these common findings helps organizations proactively address gaps before certification audits.
Classification of Nonconformities
- Major Nonconformity: Absence or complete breakdown of a required system element that affects the ability of the AIMS to achieve intended outcomes
- Minor Nonconformity: Single lapse or partial implementation that doesn't systematically affect AIMS effectiveness
- Observation: Area of potential risk that hasn't yet manifested as a nonconformity
Top 10 Most Common Nonconformities
1. Incomplete AI System Inventory
Finding: Organizations fail to identify all AI systems within scope, often missing embedded AI in third-party tools or legacy systems with algorithmic decision-making.
Evidence of Issue: AI risk assessment only covers obvious ML models, no discovery process for identifying AI across the organization, third-party SaaS tools with AI features not assessed.
How to Fix: Conduct systematic AI discovery audit, include criteria for what constitutes an AI system, review all vendor tools for AI capabilities, establish process for identifying new AI systems.
2. Missing or Inadequate AI System Impact Assessments
Finding: AI systems are deployed without formal impact assessments or assessments lack required depth.
Evidence of Issue: No documented impact assessments for production AI systems, assessments don't consider societal or ethical impacts, no process for reassessing when systems change.
How to Fix: Develop impact assessment template aligned with ISO 42001, require assessment before deployment authorization, include triggers for reassessment, document assessment methodology.
3. Insufficient Bias Testing Documentation
Finding: Organizations claim to test for bias but lack documented evidence, defined metrics, or acceptable thresholds.
Evidence of Issue: No defined fairness metrics, no documented thresholds for acceptable bias levels, testing performed but results not recorded, no remediation records when bias detected.
How to Fix: Define fairness metrics appropriate for each AI system, establish documented thresholds, create standardized testing and documentation procedures, implement bias monitoring in production.
4. Weak Human Oversight Protocols
Finding: Human oversight exists conceptually but lacks defined triggers, procedures, and documentation.
Evidence of Issue: No confidence thresholds defined for human review, override procedures undocumented, no records of human interventions, personnel lack training on oversight responsibilities.
How to Fix: Define specific triggers for human intervention, document override procedures, implement logging of all human interventions, train and assess personnel competence.
5. Incomplete Event Logging
Finding: Logging doesn't cover all required AI system lifecycle phases or lacks required attributes.
Evidence of Issue: Design decisions not logged, development changes missing attribution, logs lack tamper-proof timestamps, no defined retention periods.
How to Fix: Map required logging across all lifecycle phases, ensure logs include who, what, when, and why, implement tamper-evident logging mechanisms, define and enforce retention policies.
6. Data Provenance Gaps
Finding: Training data sources are poorly documented with incomplete lineage information.
Evidence of Issue: Unknown data sources for training sets, transformation steps undocumented, no approval records for data usage, missing data quality assessments.
How to Fix: Implement data lineage tracking from source to model, document all data transformations, establish data approval workflows, conduct and record data quality assessments.
7. Third-Party AI Components Without Due Diligence
Finding: Pre-trained models, APIs, or AI services used without vendor assessment.
Evidence of Issue: Using foundation models without vendor evaluation, no AI-specific clauses in vendor contracts, missing documentation of third-party AI capabilities and limitations.
How to Fix: Develop AI-specific vendor assessment criteria, require assessments before using any third-party AI, include AI governance requirements in contracts, monitor third-party AI performance and updates.
8. Inadequate Management Review
Finding: Management reviews don't adequately cover AI-specific topics or lack evidence of improvement decisions.
Evidence of Issue: AI governance not on management review agenda, no review of AI-specific metrics, decisions not documented or followed through.
How to Fix: Define AI-specific inputs for management review, include AI metrics, incidents, and audit results, document decisions and track implementation.
9. Training and Competence Gaps
Finding: Personnel lack required competence for AI-related responsibilities or training is undocumented.
Evidence of Issue: No defined competence requirements for AI roles, missing training records, no competence assessment evidence, awareness training doesn't cover AI policies.
How to Fix: Define competence requirements by role, develop and deliver appropriate training, assess and document competence, include AI governance in awareness programs.
10. Documentation-Reality Disconnect
Finding: Documented procedures don't reflect actual practices or aren't followed consistently.
Evidence of Issue: Procedures describe processes that don't exist, staff describe different practices than documented, records don't support documented processes.
How to Fix: Review procedures with process owners, update documentation to reflect reality, implement regular documentation reviews, train staff on documented procedures.
Prevention Strategies
Before Certification:
- Conduct thorough gap analysis against all requirements
- Perform internal audits using the same criteria as certification audits
- Address all findings before Stage 2
- Conduct mock interviews with key personnel
Ongoing Maintenance:
- Regular internal audits covering all AIMS elements
- Prompt corrective action for identified issues
- Continuous documentation updates
- Regular training refreshers
Responding to Nonconformities
- Accept and Understand: Don't argue—understand the gap
- Root Cause Analysis: Identify why the gap exists
- Corrective Action: Address both the specific instance and systemic cause
- Implementation: Execute corrective actions with documented evidence
- Verification: Confirm actions are effective
- Prevention: Implement controls to prevent recurrence
Conclusion
Understanding common nonconformities enables proactive prevention rather than reactive correction. By addressing these typical findings before certification audits, organizations can streamline their certification journey and build more robust AI governance systems.