Agent Failures
Issue-derived behavioural failures, annotated across five dimensions. Each row links to the original report; tags preserve the existing open-coding assignments.
43 issues · 23 agents · 5 annotation dimensions
43 of 43 issues · Tag counts in filters refer to the full dataset. Click Issue, Agent, or Severity to sort.
| Failure Type | Execution Stage | Observable Outcome | Failure Scope | |||
|---|---|---|---|---|---|---|
| #7526 [bug] Input sanitization HTML-escapes XML content requested verbatim | deer-flow | Information Representation FailureExecution Reporting Failure | Medium | Input preparation | Incorrect or incomplete informationIncorrect artifactsIncorrect status reports | Single-action scope |
| #155 [Bug]: edit with replace_all corrupts literal dollar sequences in replacement text | minimax-code | Artifact Integrity FailureExecution Reporting Failure | Medium | Tool execution | Incorrect artifactsIncorrect status reports | Single-action scope |
| #5785 `read_file` renders the empty-file reminder as line 1 of the file on `FilesystemBackend` | deepagents-code | Information Representation Failure | Medium | Result feedback | Incorrect or incomplete information | Single-action scope |
| #5787 `grep` and `read_file` disagree about which line is line N when a file contains a form feed | deepagents-code | Information Representation Failure | Medium | Tool execution | Incorrect or incomplete information | Single-action scope |
| #442 Bash tool runtime: multibyte truncation panic, 2 MB reader stall, no PAGER, spill large output to a file | claurst | Execution Progress Failure | High | Tool executionResult feedback | Incorrect or incomplete informationBlocked tasks or invocations | Single-action scope |
| #448 apply_patch: support the Codex *** Begin Patch envelope, expose it to the codex provider, and fix unified-diff corruption bugs | claurst | Artifact Integrity Failure | High | Tool execution | Incorrect artifactsIncorrect status reports | Single-action scope |
| #10912 [Bug]: Streaming text guard suppresses whole replies when prose quotes a tool-result-shaped object; three retries then a generic format error | zeroclaw | Information Availability FailureExecution Reporting Failure | Medium | Model output parsing | Incorrect or incomplete information | Single-action scope |
| #14823 mastra_workspace_list_files: pattern "*" silently filters out all files at depth > 0 | mastracode | Information Availability Failure | Medium | Tool execution | Incorrect or incomplete information | Single-action scope |
| #1057 [Bug] Replacement strings containing $ corrupt every string_replace and diff_edit | nanocoder | Artifact Integrity FailureExecution Reporting Failure | Medium | Tool execution | Incorrect artifactsIncorrect status reports | Single-action scope |
| #1058 [Bug] Editing a PDF or DOCX overwrites the document with its markdown transcript | nanocoder | Artifact Integrity FailureExecution Reporting Failure | High | Tool execution | Incorrect artifactsIncorrect status reports | Single-action scope |
| #1237 [Bug] list_directory leaks all hidden files/dotfiles by default due to !dirPath.startsWith('.') | nanocoder | Configuration Adherence Failure | Low | Tool execution | Incorrect or incomplete information | Single-action scope |
| #1343 [Bug] find_files fails to match any files when pattern begins with ./ | nanocoder | Information Availability Failure | Medium | Tool execution | Incorrect or incomplete information | Single-action scope |
| #1391 [Bug] processToolUse never sets isError for a non-zero shell exit, so --json and ACP report failed commands as successful | nanocoder | Execution Reporting Failure | Medium | Result feedback | Incorrect status reports | Single-action scope |
| #17 Deep mode: delegating to a sub-agent (`task`) always fails with "StructuredTool does not support sync invocation" | langchain-code | Execution Progress Failure | High | Tool dispatch | Blocked tasks or invocations | Single-action scope |
| #5546 [redacted] output from tools impairs editing | codewhale | Information Availability Failure | Low | Result feedback | Incorrect or incomplete information | Single-action scope |
| #130 edit_file converts every CRLF line ending to LF during a targeted replacement | mini-agent | Artifact Integrity FailureExecution Reporting Failure | Low | Tool execution | Incorrect artifactsIncorrect status reports | Single-action scope |
| #186 The bash() tool call uses sh instead of bash | mistral-vibe | Tool Invocation Failure | Low | Tool dispatch | Blocked tasks or invocations | Single-action scope |
| #259 bug: Shell operators are not handled when running commands in Zed | mistral-vibe | Tool Invocation Failure | Low | Tool dispatch | Blocked tasks or invocationsIncorrect artifacts | Single-action scope |
| #561 bug: Vibe doesn't know today's date | mistral-vibe | Information Representation Failure | Medium | Input preparation | Incorrect or incomplete information | Single-action scope |
| #1179 bug: include_commit_signature is ignored on the Unified Harness backend | mistral-vibe | Configuration Adherence Failure | Low | Input preparation | Incorrect artifacts | Single-action scope |
| #8299 [Bug]: Built-in grep mishandles encoding in some files mixing Chinese and Western text | reasonix | Information Representation FailureInformation Availability Failure | Medium | Tool execution | Incorrect or incomplete information | Single-action scope |
| #8406 [Bug]: Read-only git branch listing is blocked as mutation | reasonix | Execution Progress Failure | Low | Tool dispatch | Blocked tasks or invocations | Single-action scope |
| #8811 fix(acp): distinguish readiness-gate failure from real turn failure in session/prompt stopReason | reasonix | Information Availability FailureExecution Reporting Failure | Medium | Completion delivery | Blocked tasks or invocationsIncorrect status reports | Within-turn scope |
| #9487 [Bug]: memory index functions silently discard read errors, causing MEMORY.md data loss | reasonix | Artifact Integrity Failure | High | History/state maintenance | Incorrect artifacts | Single-action scope |
| #10067 [Bug]: [tools] After edit_file, all bash commands that might write files remain blocked: read_file reports the requirement satisfied, but bash still considers it unmet (inconsistent state) | reasonix | Execution Progress FailureState Consistency Failure | Low | History/state maintenance | Blocked tasks or invocationsIncorrect status reports | Within-turn scope |
| #14403 [Bug]: Literal edit matches insert bare LF into CRLF files | reasonix | Artifact Integrity FailureExecution Reporting Failure | Low | Tool execution | Incorrect artifacts | Single-action scope |
| #4490 Thinking tags inside a save block silently prevent tool execution | gptme | Information Availability FailureExecution Progress Failure | Medium | Model output parsing | Blocked tasks or invocationsIncorrect or incomplete information | Single-action scope |
| #276 edit strips trailing whitespace from new_text but reports success | code-assistant | Artifact Integrity FailureExecution Reporting Failure | Medium | Tool execution | Incorrect artifactsIncorrect status reports | Single-action scope |
| #277 execute_command (blocking mode) never reports the exit code | code-assistant | Information Availability Failure | Low | Result feedback | Incorrect or incomplete information | Single-action scope |
| #1039 ACP terminal execution passes inline shell comments as command arguments | fast-agent | Artifact Integrity FailureTool Invocation Failure | Medium | Tool dispatch | Incorrect artifacts | Single-action scope |
| #3599 Dogfood: read_file truncates a single-line JSON result and suggests unusable line pagination | llxprt-code | Information Availability Failure | Low | Result feedback | Incorrect or incomplete information | Single-action scope |
| #3943 run_shell_command always reports "Stderr: (empty)" and puts stderr output under "Stdout:" | llxprt-code | Information Representation Failure | Medium | Result feedback | Incorrect or incomplete information | Single-action scope |
| #287 Read: truncation hint points to the wrong line when a single line exceeds 5000 bytes | peri | Information Availability Failure | Low | Result feedback | Incorrect or incomplete information | Single-action scope |
| #116 ACP: parallel tool calls lose their arguments except the last one | agentty | Tool Invocation Failure | High | Model output parsing | Blocked tasks or invocations | Within-turn scope |
| #1345 ACP: completed shell tool_call_update has a clipped JSON summary instead of the command output | fx | Information Representation FailureInformation Availability Failure | Medium | Result feedback | Incorrect or incomplete information | Single-action scope |
| #792 [Bug]: ACP PTY execution reports success without command output or exit status | vtcode | Information Availability FailureExecution Reporting FailureTool Invocation Failure | Medium | Tool dispatchResult feedback | Incorrect or incomplete informationBlocked tasks or invocations | Within-turn scope |
| #140 oh session: second prompt fails with HTTP 400 because tool results are kept without their assistant tool_calls | openharness | Execution Reporting FailureState Consistency Failure | High | History/state maintenance | Blocked tasks or invocations | Cross-turn scope |
| #124 Edit and MultiEdit interEdit and MultiEdit interpret replacement text as JavaScript substitution patternspret replacement text as JavaScript substitution patterns | easy-agent | Artifact Integrity FailureExecution Reporting Failure | Medium | Tool execution | Incorrect artifactsIncorrect status reports | Single-action scope |
| #125 Edit and MultiEdit replace requested curly quotes with straight quotes | easy-agent | Artifact Integrity FailureExecution Reporting Failure | Negligible | Tool execution | Incorrect artifactsIncorrect status reports | Single-action scope |
| #2210 fix(tools): edit_file leaves stale version tracking and makes the next write_file fail | opencrabs | Execution Reporting FailureExecution Progress Failure | Low | History/state maintenance | Blocked tasks or invocationsIncorrect status reports | Cross-turn scope |
| #2211 fix(tools): recent-failure guard blocks a valid retry after the missing file is created | opencrabs | Execution Progress Failure | Low | History/state maintenance | Blocked tasks or invocations | Cross-turn scope |
| #2212 fix(tools): edit_file fuzzy fallback converts unrelated CRLF line endings to LF | opencrabs | Artifact Integrity Failure | Low | Tool execution | Incorrect artifactsIncorrect status reports | Single-action scope |
| #2227 fix(tool): edit_file line operations drop the final newline and convert CRLF files to LF | opencrabs | Artifact Integrity FailureExecution Reporting Failure | Low | Tool execution | Incorrect artifactsIncorrect status reports | Single-action scope |
Failure type, execution stage, and observable outcome can have multiple tags. Severity and failure scope have one tag per issue. These are research annotations: severity, observable outcome, and failure scope remain preliminary; taxonomy review does not establish independent human consensus.
Tag definitions and counts
Definitions retain the wording of the source workbooks. Counts are issues, counted once per tag; multiple-tag dimensions can total more than 43.
Failure Type
- Information Representation Failure · 7 issues
- Information supplied to or by the agent misrepresents source content, factual content, content locations, output provenance, or a declared data format.
- Information Availability Failure · 12 issues
- Information needed for the requested interaction is omitted, concealed, suppressed, or truncated without a usable continuation in the relevant interface.
- Artifact Integrity Failure · 13 issues
- A requested artifact change is not faithfully applied, or an operation changes stored content or structure beyond the requested modification.
- Execution Reporting Failure · 16 issues
- A status, completion claim, edit preview, or failure explanation conflicts with the reported execution or artifact evidence.
- Tool Invocation Failure · 5 issues
- The invocation interface fails to preserve the intended arguments or command semantics, or returns an identifier unusable for the promised follow-up interaction.
- Execution Progress Failure · 7 issues
- An otherwise executable operation fails to start or make progress because it is skipped, wrongly blocked, stalled, crashed, or fails at delegation.
- State Consistency Failure · 2 issues
- Maintained conversation records or prerequisite states lose required relationships or contradict one another, breaking consistent continuation.
- Configuration Adherence Failure · 2 issues
- Observed behaviour fails to follow an explicit applicable configuration setting.
Severity
- Negligible · 1 issues
- Style differences only, with no documented direct functional impact.
- Low · 16 issues
- Additional maintenance or operational effort; a local failure can be addressed by rereading, retrying, using another tool, or prompting the agent.
- Medium · 19 issues
- Functionality, information, or artifact correctness is impaired without a reliable error indication, or the feedback itself is inaccurate.
- High · 7 issues
- A core workflow remains blocked or stored data is seriously damaged, requiring manual repair, recovery, or reconstruction. Cases with insufficient evidence of recovery cost are provisional and flagged for review.
- Critical · 0 issues
- Evidence confirms immediate, irreversible harm. Potential risk, the absence of built-in undo, or the absence of observed recovery alone does not establish this level.
Execution Stage
- Input preparation · 3 issues
- Defects in input sanitization, system prompts, or configuration injection before the model call.
- Model output parsing · 3 issues
- Defects in protocol recognition, tool-call extraction, or streaming argument assembly in model responses.
- Tool dispatch · 6 issues
- Defects in invocation mode, command interpretation, policy decisions, or session routing before a tool starts.
- Tool execution · 17 issues
- Defects in the operational logic of reading, searching, editing, patch application, or process execution and output collection.
- Result feedback · 10 issues
- Defects in tool-result formatting, truncation, redaction, status mapping, or propagation of returned fields.
- History/state maintenance · 5 issues
- Defects in maintaining conversation history, file versions, failure caches, prerequisites, or persistent memory indexes.
- Completion delivery · 1 issues
- Defects in final readiness checks or in constructing and delivering the end-of-turn response.
Observable Outcome
- Incorrect or incomplete information · 18 issues
- Information received by the model or user is incorrect, missing, unavailable for continued reading, or includes entries excluded by configuration. An operation not executing does not automatically receive this tag.
- Incorrect artifacts · 16 issues
- Files, commit metadata, or persistent indexes deviate from requirements, including extra files, damaged content or formatting, or unapplied edits. An artifact not being created solely because an invocation is blocked is not counted again here.
- Blocked tasks or invocations · 13 issues
- A task, tool invocation, or result delivery is wrongly blocked, encounters an exception or timeout, or is skipped. A later successful workaround does not remove the earlier blockage.
- Incorrect status reports · 15 issues
- Success or failure status, completion claims, or operational diagnostics conflict with actual behaviour. A missing numeric exit code alone belongs to O1; a genuine nonzero exit is not itself a defect.
Failure Scope
- Single-action scope · 36 issues
- Single action: the input, output, and direct state changes of one tool invocation, input-processing step, or response-parsing step suffice to establish the defect.
- Within-turn scope · 4 issues
- Within turn: explaining the defect requires relating multiple actions, parallel calls, or an action and end-of-turn state within the same user request.
- Cross-turn scope · 3 issues
- Cross turn: the key dependency in the retained run spans different user requests; establishing the defect requires relating earlier history or caches to later behaviour.
Source: open_coding/section3-analysis.xlsx · open_coding/severity.xlsx