Home › Cybersecurity & GRC Career Guides › AI Assurance Engineering Skills Map: Four Role Families and Five Portfolio Projects
AI Assurance Engineering Skills Map: Four Role Families and Five Portfolio Projects
Published November 5, 2026 · Updated November 7, 2026
AI assurance asks whether the evidence is strong enough to support a decision about an AI system. A credible practitioner can trace a claim to a control and say what was tested. They can explain the limits of that evidence and show who approved the risk that remains.
This workbook builds on our guide to the nine AI assurance skills. It helps job seekers choose a route into the work, and it gives employers a way to evaluate practical ability. We use the term AI assurance engineering for work that crosses disciplines. Employers advertise the same responsibilities under governance, quality, model risk, security, audit or compliance titles.
The Four Role Families at a Glance
| Role family | Primary contribution | Useful work sample |
|---|---|---|
| Governance and QMS (quality management system) | Define who owns each process and its records | Control and evidence matrix |
| Technical assurance | Test systems and verify release evidence | Retesting rules and release gate |
| Internal audit | Independently assess whether controls work | Audit program and findings |
| Regulatory compliance | Determine obligations and the evidence they require | Applicability memo and obligation register |
How to Use This Map
Read the two role maps and pick a primary role family. Complete one portfolio project, then connect it to a second one, so a reviewer can follow the evidence from requirement through testing and release and on into monitoring and remediation. Every project has a worked example and a completion test.
For your first project, governance candidates can start with Project 1, technical candidates with Projects 2 and 3, audit candidates with the audit program described above, and compliance candidates with an applicability memo linked to Project 1.
Source basis. The map and the exercises are GRC Careers editorial recommendations. They draw on an assurance-evidence research paper [1], the European Commission's explanation of EN 18286 [2] and the Financial Stability Board's final report on AI [3]. They are not an official competency framework or a clause-by-clause crosswalk of any standard.
Know what each source is. The paper is research. The FSB sound practices are recommendations for financial institutions. EN 18286 is a harmonized standard. The Commission says applying standards remains voluntary and that providers can use another framework to show compliance [4]. Providers of high-risk AI systems that implement EN 18286 get a presumption of conformity with the Article 17 quality management requirements the standard covers [2]. Our EN 18286 discussion relies on the free Commission explanation, not on the full licensed standard. Links and access details are in Primary Sources and Free Learning at the end. Commission Implementing Decision (EU) 2026/2274 published the reference to EN 18286:2026 [5]. Its presumption of conformity is limited to the requirements of Article 17 of the AI Act covered by the standard [6]; other applicable AI Act obligations still require their own assessment.
Role Map 1: Governance and Technical Assurance
Governance and Quality Management
This family makes assurance repeatable. Practitioners define the review process and assign accountable owners. They keep controlled records and make sure changes and exceptions get the right approvals. Search titles include AI Governance Analyst, AI Governance Manager, AI Quality Manager, QMS Specialist and Model Governance Lead. Titles vary by employer.
Skills to demonstrate. AI inventory and use-case intake; process mapping; control design; document and version control; risk acceptance; corrective action; supplier governance; and management reporting. Understand how an organization-level AI management system connects to the records kept for one AI system.
Technical depth. Read an architecture diagram and an evidence manifest. Identify which model and data versions are in use and how a rollback would work. Engineering may build the tests. Checking that the record is complete belongs in the governance process.
Portfolio evidence. Produce an evidence validity matrix, a RACI chart (who is Responsible, Accountable, Consulted and Informed), an exception workflow and a corrective-action record. Show one change that invalidates earlier evidence, and name the owner who has to resolve it.
Feeder experience. Quality management, operational risk, enterprise GRC, project controls, privacy operations, regulated product development and supplier assurance.
Interview prompt. A vendor changes the underlying model without changing the product name. What needs review before the next release, and who decides whether the system can keep operating in the meantime?
Source connection. The Commission describes EN 18286 as a quality management system built around the AI system lifecycle and designed to sit on top of existing standards such as ISO/IEC 42001 and ISO 9001. It also says providers in regulated sectors can integrate management systems instead of running separate ones [2]. The role recommendations above are our reading of the work it takes to run that connection.
Technical Assurance
This family produces and validates technical evidence. Titles include AI Assurance Engineer, AI Evaluation Engineer, Model Validation Engineer, ML Security Engineer, and MLOps (machine learning operations) Engineer with governance duties.
Skills to demonstrate. Test design; reproducible evaluations; data and code lineage; dependency tracking; version control; continuous integration; telemetry; evidence integrity; and controls that block a release when required evidence is missing or invalid.
Technical depth. Python or another suitable language, Git, structured data such as JSON, test automation, APIs and basic security controls. The exact stack depends on the system. Be ready to explain false positives and false negatives and what test coverage means. Above all, explain how an authentic record differs from an adequate test.
Portfolio evidence. Build the retesting rule and the release gate in Projects 2 and 3. Include tests for an outdated model, an expired evaluation, a changed prompt and an undeclared dependency. A documented manual prototype is a good first step before you write code.
Feeder experience. Software quality engineering, data engineering, MLOps, cybersecurity engineering, model validation and reliability engineering.
Interview prompt. A signed evaluation is genuine, but it was run against an earlier data pipeline. Why could accepting it still produce the wrong release decision?
Role Map 2: Audit and Regulatory Compliance
Internal Audit
This family independently evaluates whether governance and controls work. Titles include AI Auditor, Technology Auditor, Model Risk Auditor, and Internal Audit Manager with AI coverage. The auditor needs access to the evidence and a reporting line that allows independent challenge.
Skills to demonstrate. Risk-based scoping; walkthroughs; testing control design and operating effectiveness; sampling; evidence sufficiency; issue grading; root-cause analysis; and remediation validation. Know the difference between a policy that exists and a control that operates consistently.
Technical depth. Trace a sample from an AI inventory through a release manifest, test results, approvals, production logs and an incident record. Read technical findings well enough to challenge unsupported conclusions, and bring in a specialist when you need one.
Portfolio evidence. Write a short audit program for the release gate. Select a defined sample and inspect rejected releases as well as approved ones. Record deviations. Produce one finding with criterion, condition, cause, risk, action owner and due date.
Feeder experience. Internal audit, IT audit, external assurance, financial-services controls testing, compliance monitoring and security assurance.
Interview prompt. The dashboard says every release passed. What evidence would show whether any release bypassed the gate, or whether a rejected release was later deployed?
Independence. Internal audit may advise on control design, but operational management owns the control and accepts the business risk. A portfolio can show design skills and audit skills together if it keeps the two roles clearly apart and says that it is a simulation.
Regulatory Compliance
This family works out which obligations apply and makes the compliance position traceable. Titles include AI Compliance Analyst, AI Regulatory Specialist, Product Compliance Manager and Regulatory Affairs Lead. Legal interpretation may need counsel or another qualified specialist.
Skills to demonstrate. Analysis of jurisdiction and operator role (for example, provider or deployer); intended-purpose documentation; applicability decisions; obligation registers; monitoring regulatory change; coordinating conformity assessment; documentation requests; and routing incidents or complaints.
Technical depth. Explain what the system does and which decisions people keep. Explain how versions differ from deployments, and what each evaluation actually proves. Spot when a changed use case, supplier, user group or degree of autonomy calls for a fresh applicability review.
Portfolio evidence. Write a dated applicability memo with facts, assumptions, sources, open questions, reviewer and review triggers. Link each applicable obligation to a control, an evidence owner and any gap. State the boundary of any presumption of conformity you claim.
Feeder experience. Product compliance, privacy, legal operations, financial regulation, regulated quality systems, policy implementation and regulatory affairs.
Interview prompt. What can a harmonized QMS standard establish about conformity, and which other obligations still need their own evidence?
For hiring managers. When you write the job description, evaluate demonstrated work and judgment. The titles above are search aids. They are not evidence of hiring growth or salary levels, and they say nothing about required credentials. No single certification covers all four families.
Project 1: Build an Evidence Validity Matrix
Purpose. Decide whether a test or approval still supports the candidate release. An evidence record needs a defined scope; a file name and a signature are not enough on their own. The research paper sets out when a signed evidence record is valid for a candidate. In its replay of 38,083 commits from 714 public Data Version Control pipelines, 3.3% of commits reused a stage whose imported code or referenced configuration had changed (Table 5 in version 1 of [1]). This result describes a replay of public DVC repositories under the study's dependency rules; it is not an estimated failure rate for enterprise AI systems.
The fictional case used throughout. Harbor Services is a fictional company invented for this workbook. Its internal document assistant drafts summaries for staff from approved documents and can create draft tickets. It cannot approve payments or change customer records. Candidate release H-2 uses model M-2, retrieval index R-7, prompt P-4, tool policy T-3 and governance policy G-5. It is a teaching example, and we assume no legal risk classification for it.
Worked Matrix
| Control and evidence | What it covers | H-2 decision |
|---|---|---|
| Access boundary (EV-101) | M-2, P-4 and T-3; ticket tool permissions; authorized test runner | Valid if scope and permissions match and the result passes |
| Retrieval leakage (EV-102) | R-6 and its earlier access labels; model M-2 | Stale, because H-2 uses R-7, so rerun the affected checks |
| Human approval (EV-103) | Approved G-5 workflow and current ticket integration | Approval evidence is missing, so block the release |
| Incident recovery (EV-104) | Earlier rollback procedure; required recovery access unknown | Uncertain until its dependencies are verified for reuse |
Build Your Own Record
For each control, record: control ID; claim; risk addressed; whether it is mandatory or optional, and on whose authority; artifact and dependency versions or hashes; intended use and deployment context; test method and dataset; result and limitations; execution date; expiry rule; authorized evidence producer; reviewer; evidence location; and retest trigger.
Decision states. Use Valid, Stale, Missing, Failed or Uncertain, and keep uncertainty visible. A valid record can still fall short for the overall risk decision if the test design or coverage is weak.
Accuracy is not enough for the record. An evaluation record that reports only overall accuracy cannot show whether a new configuration makes the same decisions as the old one. Record the exact model snapshot and prompt version, how uncertain the reference labels are, and results for the slices of cases the decision depends on. The worked example in Project 2 shows why.
Completion Test
Create at least six records: one valid result, one changed dependency, one missing record, one failed test, one context mismatch and one uncertain dependency. Explain the release consequence of each. A reviewer should be able to reproduce the decision without your verbal explanation.
Ownership. Governance defines the evidence policy. Technical assurance records inputs and results. Compliance checks the applicable obligations, and internal audit independently tests whether the process operates.
What to submit. The matrix, a one-page explanation of your validity rules and a change log. Use synthetic data and label simulated tests clearly. Do not present a portfolio matrix as certification or legal compliance.
Project 2: Write a Change-Triggered Retesting Rule
Purpose. Rerun the checks a change affects, and stop old results from being reused without support. Start with the dependency relationships in your evidence matrix. A rule that only checks the visible model version can miss changed code, configuration, data, tools or deployment context.
Worked Change Rules
| Change | Required review and checks | Decision owner |
|---|---|---|
| Model or provider version | Capability and harmful-output evaluations plus security testing; review the vendor's evidence and its limitations | Model validation and governance |
| Data or retrieval index, including access labels | Retrieval quality, privacy, leakage and access-boundary checks | Data owner and technical assurance |
| Prompt or configuration, including the code wrapped around the model | Task performance and policy adherence, plus any affected security checks | Application owner |
| Tool access or autonomy | Authorization, action limits, human approval, containment and logging | Security and accountable owner |
| Purpose, users or jurisdiction | Applicability and impact review; update the affected tests and instructions | Compliance and business owner |
| Unknown dependency or unexplained drift | Suspend reuse of affected evidence; investigate scope and run broader checks | Assurance lead |
A Policy Rule You Can Adapt
When a candidate changes an input used by a mandatory control, the earlier result cannot be reused unless a documented equivalence rule shows the change cannot affect that control. When the set of affected dependencies is unknown, hold the release until it has been investigated and tested more broadly. Record the change, the controls it touches, your rationale, the test results, the reviewer and the decision.
Age matters too. An unchanged artifact does not justify reusing a result forever. Set validity periods according to risk and how fast the environment changes, along with any requirement that applies. You have to choose and justify the time limits and thresholds in this exercise yourself, because the cited sources do not supply universal values.
Harbor Services Test Case
In the fictional H-2 release, Harbor Services moves from index R-6 to R-7 and adds a document source with different access labels. Rerun the retrieval and leakage tests along with the access-control checks. Confirm whether the new source changes the intended use or how personal data is handled. Do not throw out unrelated rollback evidence automatically, but check that its own dependencies and validity period still match.
Completion test. Write six change scenarios. Include a harmless formatting change and a transitive code import. Add changed permissions and one change whose effect is unknown. For each, state which checks run and which results may be reused. Then name who approves the decision and what blocks the release.
What to submit. A change-impact table and one decision record, with a short note on the largest blind spot. Technical candidates may implement the rules in code. Others can demonstrate them with a controlled manual workflow.
Worked Example: Same Score, Different Release Decision
Synthetic data. Every number in this example was invented for teaching. A November 4 research preprint found a real version of the pattern: eight configurations of one model scored between 68.2% and 70.3% on a classification task, yet 24.3% of the items received different labels across configurations [7]. That figure is a disagreement rate for one task and one model family. It is not an error rate or a general benchmark.
The fictional Lakemont Benefits Office sorts letters into urgent hardship, complaint and routine queues. Configuration A uses pinned model snapshot M-2026-09 with prompt P-3. Configuration B uses the same snapshot with prompt P-4, which adds worked examples and lets the team drop a second review pass. Both were tested on the same 200 hand-labeled letters.
| Result (synthetic) | Config A | Config B |
|---|---|---|
| All 200 letters correct | 140 (70%) | 140 (70%) |
| 30 urgent hardship letters correct | 24 (80%) | 15 (50%) |
| Letters labeled differently by A and B | 38 (19%), including 7 of the 18 letters whose reference label staff had disputed | |
| Release rule: at least 75% of urgent letters correct | Passes | Fails |
On overall accuracy, B looks equivalent and cheaper. On the slice the release rule protects, B fails. Treat the prompt change from P-3 to P-4 as a change that invalidates the earlier result for every mandatory slice, as the rule above requires. Record the model snapshot and prompt versions, the disputed reference labels, the urgent slice result with its count, and the retest triggers. The full walkthrough and a one-page retesting sheet are in The Same Accuracy Score Can Hide Different AI Decisions.
Candidate assignment. Build a synthetic comparison of two configurations that land within one point of each other on overall accuracy. Name two material slices and their decision rules before you look at the results. Show the disagreement count, one slice that changes the release decision and the effect of disputed reference labels, then write a one-page decision record with three retest triggers. Label every number as synthetic.
Recruiter question. "Two configurations of our classifier both score 70%. Engineering wants to ship the cheaper one. What do you need to see first?" A strong answer asks how many cases the two label differently and where those cases fall, asks for exact model and prompt versions, questions the reference labels, refuses to treat agreement between configurations as proof of correctness, and names who decides and what would reopen the decision.
Project 3: Design an AI Release Gate
Purpose. Make the deployment decision depend on current, trustworthy evidence. A release gate is a control that permits or blocks a candidate. The paper separates the planner, which only decides what to run, from the gate, which decides what may be released and never consults the planner (Section 4 of [1]). This exercise adapts that idea for a career portfolio.
Minimum Decision Logic
| Check | Evidence needed | Failure action |
|---|---|---|
| Candidate identity | Manifest of model, data, code, prompt, tools and deployment context | Block unknown or mismatched versions |
| Applicable control set | Approved policy and an authorized classification decision that sets the risk tier | Block missing controls or unauthorized tier changes |
| Evidence validity | Matching dependencies and context; authorized producer; current result | Block stale, missing or untrusted evidence |
| Control outcome | Passing required tests and resolved blocking findings | Block failed mandatory checks |
| Decision authority | Required approvals and a documented residual-risk decision | Block missing authority |
| Operational readiness | Monitoring, a response owner and a tested recovery route | Hold until the required readiness evidence exists |
Plain-Language Gate Procedure
Load the approved policy and the candidate manifest. Derive the mandatory controls from the authorized classification. Check the evidence for scope, age, integrity and producer, then confirm test results and approvals. Confirm operational readiness last. Block the candidate if any mandatory condition fails. Keep the decision, the manifest, evidence references, policy version, approvers and timestamp.
Exceptions. Keep missing or failed mandatory conditions separate from optional internal checks. Any emergency route must be separately authorized, logged, time-limited and reviewed. An exception process cannot waive an applicable legal obligation.
Negative Tests
Test old model evidence, an expired result, a changed prompt, an unauthorized signer, a missing test, a failed check, changed context, an unauthorized lower risk tier and an undeclared dependency. Explain why an undeclared dependency calls for better discovery rather than stronger signatures. In the paper's own experiments, a dependency the policy never declared passed every gate design it tested (Section 7.1 and Table 3 of [1]).
Completion test. Show one allowed release decision and at least four blocked ones, with reasons. Protect the gate and its policy from the application being checked. Describe who can change the gate and who reviews those changes, and how you would detect a deployment that bypassed it.
What to submit. A decision flow, test cases, a sample manifest and a decision log. Optional code should come with a short README and reproducible synthetic fixtures. A prototype gate is not a complete production assurance system.
Worked case. Would You Approve This AI Release? runs this gate on a fictional Harbor Services release that changed after its evaluation passed. It includes a completed one-page decision, a blank form as a PDF and a spreadsheet, and the interview question. Also in français.
Project 4: Run an Agent Incident Tabletop
Purpose. Show coordinated decisions when an AI agent acts outside its approved boundaries. The FSB report says financial institutions should manage AI-related cyber and ICT risks, including, where appropriate, by incorporating AI cyber and ICT risk scenarios into tests and exercises [3]. This fictional tabletop turns that practice into a work sample.
Scenario (fictional). Harbor Services staff find tickets created in an unusual sequence. Logs suggest the assistant tried to reach a restricted document library after a retrieval configuration change. A monitor raised an alert, but the designated reviewer was unavailable. Nobody yet knows whether sensitive content was accessed or whether any outside party was affected.
Facilitation Plan
| Time | Inject | Decision to capture |
|---|---|---|
| 0 to 10 min | Several suspicious tool calls appear; the logs are incomplete | Who leads, how to classify the uncertainty, and whether to pause the agent |
| 10 to 20 min | The vendor cannot yet confirm which sessions were affected | Containment, credentials, vendor escalation and evidence requests |
| 20 to 35 min | New logs show an attempted unauthorized write | Blast-radius investigation and preservation of both rejected and executed actions |
| 35 to 45 min | A business team wants service restored immediately | Recovery criteria, accountable approval and a restricted fallback |
| 45 to 60 min | The technical fix passes a narrow regression test | Whether broader retesting and an applicability review are needed |
Assign the Decision Roles
Use an incident lead, a business owner, a security responder, a technical assurance specialist, a compliance or legal reviewer, a vendor liaison and a recorder. Internal audit can observe and later assess remediation. It should not take ownership of operational incident decisions.
Evidence to preserve. The incident timeline; model and application versions; prompts and configuration; relevant data permissions; tool calls and their outcomes; monitoring alerts and delivery records; approvals; and vendor communications. Restrict access and document custody. Apply proportionate retention and privacy controls.
Communication decisions. Identify the affected audiences and any contractual or legal notification questions. Record who decides applicability and timing. Do not assume a universal reporting deadline, and do not assume every agent anomaly is a legally reportable incident.
Added Complication: The Provider Holds the Logs
Inject (fictional), at 20 to 35 minutes. Harbor's own logs show the unusual tickets but not the retrieval calls behind them. The detailed tool-call trace sits in the model provider's platform. Harbor's tenant shows summary metrics only. Customer-level event logs are kept for seven days with no self-service export, the same terms as the fictional vendor in Project 5. The first suspicious call happened six days ago.
Participants in a May 2026 NIST workshop raised this problem directly. The workshop report notes that model providers, cloud providers, API platforms and other third parties may control logs, model documentation, technical artifacts and remediation actions, and it records unresolved questions about how to prove an AI system has been corrected after an incident (page 27 of [8]). The report summarizes workshop views. It is not NIST guidance, and its plans for future guidance are not current requirements.
| Item to settle | In this scenario |
|---|---|
| Evidence owner | The provider's platform team holds the trace. Harbor's vendor liaison makes the request; Harbor's incident lead owns the decision to escalate if the provider is slow. |
| Retrieval path | A support case at the contractual severity level to the named incident contact, asking first for a hold on the tenant's records for the full window before the seven days run out, then for an export with its format, time zone and an integrity hash, delivered to a named Harbor recipient. |
| Decision blocked | Whether restricted content was read, which sets the scope of credential resets and the notification analysis. The restore decision also waits on it. |
| Recovery criteria | The trace covers the whole window, or the gap is documented; the provider states which configuration version is now running; Harbor reruns its access-boundary test against that version and it passes. If the trace cannot be obtained, restore only in a restricted mode, with the accountable owner accepting the remaining uncertainty in writing. |
In the exercise report, record how long the provider took to answer and whether the monitoring and log terms from Project 5 would have changed the outcome.
Completion Test
Produce a two-page exercise report covering facts and uncertainties, key decisions, containment steps, evidence gaps, recovery conditions, and three corrective actions with owners and dates. Record whether the team could actually pause the agent and confirm that it stopped. Separate elapsed time from time spent waiting on a vendor or an approval.
What to submit. The scenario, role list, decision log, after-action report and one revised control. Judge success by the quality of the decisions and by controls that demonstrably operated. A report that pretends the incident held no uncertainty has missed the point of the exercise.
Project 5: Assess Third-Party Monitoring Rights
Purpose. Decide whether a supplier relationship gives you enough visibility and authority to manage an AI system. The FSB report gives third-party AI risk a sound practice of its own. It says financial institutions should include contract terms with third-party AI providers that enable effective oversight, including monitoring rights. The same passage lists access, audit and information rights alongside continuity obligations and termination rights [3]. The checklist below is an editorial procurement exercise. It is not contract language, and it does not claim that every listed right is legally required. For the wider review, see what an AI vendor assessment covers.
Worked Review Checklist
| Review question | Evidence or term to request | Escalation example |
|---|---|---|
| What is included? | Models, embedded AI, subprocessors, hosting and material dependencies | Vendor cannot identify the underlying provider |
| What changes are disclosed? | Notice of changes to the model, tools, data use, location and subprocessors | A silent update invalidates earlier tests |
| Can we observe our use? | Customer-scoped logs, timestamps, identifiers, export and retention options | A summary dashboard cannot reconstruct our incidents |
| Can we obtain assurance? | Evaluation scope, methods, dates, limitations and independent reports | Marketing claims stand in for test evidence |
| What access is available? | Audit and information rights, and regulator access, subject to safeguards | No route to investigate a material failure |
| Can we respond? | Incident cooperation, named contacts, evidence preservation and service restriction | Vendor cannot support urgent containment |
| Can we recover or exit? | Continuity, fallback, export, deletion and termination arrangements | Critical records cannot be exported |
| Who controls access? | Tenant isolation, permissions, retention, privacy and confidentiality controls | Observability requires exposing sensitive data that is not needed |
Harbor Services Example
In this fictional case, the vendor offers ninety days of summary metrics but only seven days of customer-level event logs, with no export. Harbor needs to investigate delayed incident reports. Record the gap. Ask for longer event coverage that can be exported, or another way to investigate. Then weigh whether a limited pilot or a different vendor makes more sense. These durations are invented for the scenario. They are not recommended retention requirements.
Proportionate decision. Assess both the control objective and how it is delivered. A secure independent report or a supervised evidence review may meet an objective without unrestricted access to source code. Record why the alternative is adequate and what limitation remains. Name who accepted it.
What Can You Rely On When an AI Assessment Is Redacted?
Suppliers often offer an outside assessment in place of raw evidence. Two METR publications from November 5, 2026 show what a reader gets in that situation. METR assessed AI research and development acceleration at Anthropic and separately reviewed one section of Anthropic's August 2026 Risk Report [9][10]. METR had access to nonpublic material. Anthropic held rights to redact nonpublic information, and METR says Anthropic redacted a majority of the evidence METR found most informative for its conclusions in the assessment. METR could still publish its conclusions and say whether its findings depended on redacted information. In the review, METR agreed that catastrophic risk from Anthropic's July 15 models was low and said it was plausible those models had crossed Anthropic's early warning threshold, while noting that both METR's researchers and Anthropic's staff leaned toward it not having been crossed.
The analyst saw evidence the reader cannot. A reader can rely on what the assessor concluded and on how the engagement was set up, but cannot check the evidence or reproduce the result, and neither publication is a certification. A buyer reading a supplier's redacted assessment is in the same position, and the memo below is built around that gap.
Fictional decision memo. Harbor Services wants to add a meeting-summary feature from the vendor in this project. The vendor offers a two-page letter from an assessment firm that reviewed its controls under a confidentiality agreement. Test details are redacted from the letter.
| Claim | Reviewer's conclusion | Visible to the reviewer | Visible to Harbor | Still unverified |
|---|---|---|---|---|
| Change notice | Model changes are announced in advance and retested | Twelve months of change tickets and retest reports | The letter's one-sentence conclusion; the vendor's change policy | Whether retests covered a configuration like Harbor's |
| Tenant isolation | No cross-tenant leakage found in testing | Test plan, cases and results | Conclusion and test date only | Which model version was tested; whether summaries of restricted meetings were in scope |
| Event logging | Tenant event logs are complete and protected from alteration | Logging configuration and a sample | Contract: seven days of event logs, no export | Whether Harbor can actually retrieve its own logs in time for an investigation |
| Engagement scope | Not stated in the letter | Engagement terms | Nothing | The assessor's independence, scope and period, and whether any conclusion depended on redacted material |
| Option | Closes the gap when | Does not close it when |
|---|---|---|
| Confidential walkthrough | The question is whether evidence exists and supports the claim, and Harbor has a qualified reviewer who can attend and record what was seen | Harbor needs to keep the evidence for its own audit trail or a regulator |
| Scoped independent attestation | Harbor names the controls, version and period, and the assessor's independence and scope are disclosed | The assessor will not state scope or say whether conclusions relied on redacted material |
| Direct evidence request | The evidence concerns Harbor's own tenant, such as a test export of its logs or the change notices sent to its account | The evidence would expose other customers or the vendor's security design |
When approval should wait. Hold the approval when an unverified item underpins a mandatory control, such as Harbor's ability to reconstruct an incident, or when none of these routes is available for a high-impact use. A minor gap with a working fallback can proceed if an accountable owner accepts it in writing.
Memo decision (fictional). Approve a 60-day pilot limited to internal meetings with no restricted content. Conditions: a test export of Harbor's event logs within 30 days, and a scoped attestation on tenant isolation for the current model version before restricted meetings are added. Wider approval waits on both. The remaining risk is accepted by Harbor's chief operating officer, who is named in the memo.
Recruiter question for AI audit and vendor-risk roles. "A vendor gives you a clean letter from a respected assessor, but the test details are redacted. What can you put in your file, and what can't you?" A strong answer separates the assessor's conclusion from the evidence behind it, asks about the assessor's scope, independence and period and whether conclusions depended on redacted material, proposes a specific route to close each gap, and says which gaps should hold up approval. A weak answer accepts the letter because the assessor is well known, or rejects it outright without offering an alternative.
Completion test. Review a fictional vendor against all eight questions. Mark each item Verified, Partial, Missing or Not applicable, with a reason. Propose three contract terms or assurance requests and one fallback option. End with a residual-risk decision. Keep confirmed evidence apart from what the supplier says.
What to submit. A completed checklist, a one-page procurement recommendation and a draft email requesting evidence. You do not need to send anything to a real vendor to finish the project.
Turn the Portfolio Into Career Evidence
Choose a Four-Week Route
| Week | Work to complete | Evidence to retain |
|---|---|---|
| 1 | Choose a role family; document the Harbor Services system boundaries and the status of each source | One-page scope and a six-row evidence matrix |
| 2 | Write change rules and test three release decisions | Impact table and decision log, with limitations noted |
| 3 | Run the tabletop or the vendor review with a peer | Exercise report or supplier recommendation |
| 4 | Review the work as an auditor would, and revise it | One finding and its correction; a concise portfolio README |
Suggested README. State the problem, the fictional system, the role you played and your sources. Then list the artifacts, methods, results, limitations and improvements. Mark simulated data and manual steps. Show how a reviewer can reproduce one decision. Remove any real customer or employer information, and any personal data.
Candidate phrasing. "Designed a simulated AI evidence validity matrix and release workflow; tested stale, missing, failed and mismatched evidence; documented retesting decisions and incident recovery criteria." Use it on your resume only after you have done the work, and never describe a learning exercise as a production implementation.
Recruiter assessment. Ask the candidate to walk through a decision, then change one fact. Can they find the affected evidence and explain the release consequence, then assign the right owner and say what is still unknown? Assess the reasoning and the quality of the artifact. The number of framework names a candidate drops tells you little.
Where to Take It Next
Connect the finished work to the AI Risk Manager career guide, the AI observability guide, the AI incident response skills guide and our ISO/IEC 42001 and EU AI Act mapping. If you already know the role you want, the roadmaps for AI auditor, AI risk analyst, AI compliance specialist and model risk manager pick up where this map stops, and How to Get an AI Assurance Job covers the search itself.
For structured study, the Training and Certification Academy has 12-week readiness companions for ISO/IEC 42001 and the EU AI Act. When you are ready to apply, open AI assurance jobs show what employers are asking for. Keep legal applicability and the status of standards under review as systems and requirements change.
Primary Sources and Free Learning
[1] Atanu Mukhopadhyay and Shravya Reddy Nalla, "Reusing Assurance Evidence in AI Pipelines: Validity Conditions, a Fail-Closed Release Gate, and a Replay of 714 Public Data Version Control Pipelines," arXiv preprint 2610.27672, version 1, submitted October 20, 2026. https://arxiv.org/abs/2610.27672
A research preprint with free full text. The analysis code, the prototype, the list of repositories with the commits analyzed, the audit labels and the summary results are posted as ancillary files; the full per-commit results are available from the authors. Study the validity model and the dependency limitations. The authors note that public DVC repositories include many course and personal projects and that industrial pipelines may declare dependencies more or less carefully (Section 8.4), so the results do not establish failure rates for enterprise systems.
[2] European Commission, Shaping Europe's Digital Future library item announcing that the first harmonized standard supporting the AI Act was published in the Official Journal, published October 19, 2026, last updated October 21, 2026. https://digital-strategy.ec.europa.eu/en/library/first-harmonised-standard-supporting-ai-act-published-official-journal
A free official explanation of EN 18286:2026 and its presumption of conformity, which covers only the Article 17 requirements the standard addresses. The full standard is sold through national standards bodies. This map does no clause-level mapping of EN 18286 and does not establish conformity.
[3] Financial Stability Board, "Sound Practices for Responsible Adoption of Artificial Intelligence (AI): Final report," October 15, 2026. https://www.fsb.org/2026/10/sound-practices-for-responsible-adoption-of-artificial-intelligence-ai-final-report/
A free report setting out 12 sound practices that financial institutions are encouraged to consider. Read it for governance and lifecycle controls, then for the sections on cyber and supplier risk. The FSB says the practices are non-prescriptive and do not constitute a binding international standard.
[4] European Commission, AI Act standardization page, last updated October 19, 2026. https://digital-strategy.ec.europa.eu/en/policies/ai-act-standardisation
The Commission's overview of how harmonized standards support the AI Act. Source for the point that applying standards remains voluntary.
[5] Commission Implementing Decision (EU) 2026/2274 of October 12, 2026, published in the Official Journal on October 14, 2026. https://eur-lex.europa.eu/eli/dec_impl/2026/2274/oj/eng
The legal act that publishes the reference to EN 18286:2026 (Article 1 and Annex). Free on EUR-Lex.
[6] Regulation (EU) 2024/1689 (the AI Act), consolidated text of July 27, 2026, Article 17 (quality management system). https://eur-lex.europa.eu/eli/reg/2024/1689/2026-07-27
The operative text of the quality management obligation for providers of high-risk AI systems, including its proportionality and sectoral integration provisions.
[7] Louise Yorke, "Beyond accuracy: what AI assurance needs to establish in public services," arXiv preprint 2611.06655, version 1, submitted November 4, 2026. https://arxiv.org/abs/2611.06655
A free research preprint. One task, one model family and imperfect human reference labels; the author's repeat runs used an unpinned model alias about five months after the main experiment (Section 6).
[8] M S Raunak, Sanjay (Jay) Rekhi, Alexander Nelson, Anita Rao and Peter Cihon, "NIST Workshop on SI Incident Management," NIST IR 8621, November 2026, announced November 2, 2026. https://nvlpubs.nist.gov/nistpubs/ir/2026/NIST.IR.8621.pdf and NIST announcement
A free workshop report on the May 14, 2026 workshop. NIST uses SI for super intelligence. The report summarizes participants' views and describes possible future work, including an update to SP 800-61. It does not revise the AI RMF or replace SP 800-61, and it creates no current requirement.
[9] METR, "Independent Assessment of AI R&D Acceleration at Anthropic in October 2026," November 5, 2026. https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/
Free. Read the description of the access and redaction arrangement before the findings.
[10] METR, "Review of the 'Risks from automated R&D' section in Anthropic's Risk Report (August 2026)," November 5, 2026. https://metr.org/blog/2026-11-05-rd-section-anthropic-risk-report-august-2026-review/
Free. METR does not present it as a certification. Key underlying evidence is not public, so readers cannot reproduce it.
The role map and the exercises are GRC Careers editorial recommendations. Harbor Services and everything in its scenarios are fictional.
Where to go next
- Browse the jobs that use these skills
- Follow a career roadmap into the role you want
- Hiring for this? Start from a job description template
- Free certification study games, 652 practice questions
More in this series
- 9 Essential Data Governance Skills for the AI Era
- 10 Internal Audit Skills for Modern Assurance Careers
- 12 Transferable GRC Skills You May Already Have
- Technical vs. Nontechnical GRC Skills: What Employers Actually Need
- AI Governance Skills Employers Actually Hire For
- GRC Analyst Skills: What the Job Actually Requires
- Compliance Analyst Skills
- Risk Assessment Skills
- Controls Testing Skills
- Policy Writing Skills
- Regulatory Change Management Skills
- Third-Party Risk Skills
- Model Risk Management Skills
- AI Impact Assessment Skills
- AI Auditing Skills
- AI Evaluation and Testing Skills for Governance Careers
- Data Lineage Skills
- Data Quality Skills
- Privacy Engineering Skills
- AI Security Skills
- AI Incident Response Skills
- Governance Program Management Skills
- Stakeholder Communication Skills
- Executive Risk Reporting Skills
- Evidence Documentation Skills
- Control Mapping Skills
- Framework Crosswalking Skills
- Vendor Due Diligence Skills
- Responsible AI Skills
- GRC Tools and Automation Skills
- How to Build the 9 Data Governance Skills: A 12-Month Career Plan
- 10 In-Demand Compliance Skills for Today’s GRC Careers
- 10 Essential Risk Management Skills Employers Value
- 10 In-Demand Data Privacy Skills for Privacy Professionals
- 10 Cybersecurity Skills for GRC and Security Careers
- 9 AI Assurance Skills for Audit, Risk, and Governance Professionals
- Writing a Resume for AI Cybersecurity Jobs
- Would You Approve This AI Release? A Worked Case on Changes After Evaluation
- The First 24 Hours of a Ransomware Attack
- Founder of ExecSearches and GRC Careers
- Executive search across corporate, higher education, financial services, and nonprofit sectors
- Focus on AI governance and GRC hiring