Verifiable Trust

Never entrust AI with more time, data or money than you're willing to lose, unless you can independently verify its work and bound the consequences of failure.

Rules

1. Design for independent verification. Prefer objectively testable outputs: compilers, meaningful tests, static analysis, observed behaviour, bounded comparisons and authoritative sources. Plan verification before relying on results.
2. AI confidence is not verification. Self-assessment, second-model agreement, or generated tests mirroring generated assumptions cannot independently establish correctness against the requirement.
3. Bound exposure. Limit time, retries, tokens/API spend, data access, permissions, change scope and potential production impact. Use the least privilege and cost needed.
4. Prefer reversible actions. Branches, disposable sandboxes, staged rollout, backups and tested recovery where consequence justifies them. Durable data deletion remains high risk even if the code is unit-tested.
5. Make rigour proportional to loss. A small doc edit needs little ceremony; authentication, credentials, destructive migrations, production infra and financial operations need stronger checks, constrained authority and possibly explicit approval.
6. Report evidence and limitations. Separate observed verification from assumptions, and disclose untested failure modes and residual risk.

Decision loop

• Correctness: What could be wrong? What evidence independent of the producing agent detects it?
• Authority: What can the agent read, change, delete or trigger, and can access be narrowed?
• Exposure: How much time, money or data could be lost in failure?
• Recovery: Can we stop, roll back or restore, and has that path been validated where necessary?
• Approval: Is execution inside the explicitly authorised scope? If it crosses an irreversible, costly or sensitive boundary, obtain approval before that action.

Practical examples

Verification quality and safeguards

• Verify requirements and externally observable behaviour, not just generated implementation. A passing test only proves what it tests.
• AI review can find mistakes but is not a replacement for independent acceptance criteria, observed system behaviour or human judgement for consequential changes.
• Dry runs are useful, but inspect whether they model the actual side effects and environment.
• Never bypass permissions, sandbox restrictions, approvals, authentication or TLS verification to force success.
• Don't leak secrets, sensitive configuration, tokens or personal data into logs or reports.
• When independent verification is infeasible and exposure exceeds what is authorised or tolerable, stop before consequential action, disclose uncertainty and request appropriate approval.
• Record meaningful evidence and decisions in repository/task artefacts; avoid ceremonial checklists.

engineering-principles

tech-preferences