If you are building self-improving agents that rewrite their own skill files or prompts, then you are prone to the classic Ship of Theseus problem.
Every self-edit replaces one more “plank” of the original spec, and after enough edits, you cannot tell if the agent is still doing the job it was built for.
The failure mode is drift.
The agent does not crash or throw errors. It also keeps updating checks along the way. Each patch looks fine in isolation, but when you look at it after n iterations, you see it might have just drifted from the original task.
Here are a few guardrails that would help:
-
Keep an immutable core spec separate from the mutable parts. Let the agent edit examples and phrasing, never the goal or constraints.
-
Diff every edit against the original spec, not just the previous version. Comparing to the last state only hides compounding drift.
-
Version every edit like a commit, with a rollback path to any earlier version.
We should not stop agents from improving, but do make sure the task’s identity is retained; the same reason that Ship of Theseus is only interesting because we care whether it is still the same ship.