Compare commits

..
Author SHA1 Message Date
weichen b6d87638cc Pin the nested agent-native to the M5 fix
Why: composition consistency - Dev pins agent-native at the M5
preflight tip; the squad container's nested copy must match
(check-pins).
Evidence: pin only; adapters/native tests green on the new pin.
Impact: nested gitlink pointer change.
Refs: feature/audit-followups-pin-gate-m5
2026-10-22 06:41:21 +00:00
weichen 9edb6ba865 Preserve frame arrival order in the native event probe
Why: with a native that populates CallID/Status/ErrorReason the probe
path regrouped concurrent tool frames instead of forwarding them in
arrival order; the tap contract requires one frame per event.
Evidence: concurrent-identity test passes on the new pin and a legacy
passthrough case was added (verify-u11br.json).
Impact: probe enriches frames in place; order and grouping unchanged.
Refs: feature/agent-stack-audit-remediation-b2 U11b repair
2026-10-22 03:59:55 +00:00
weichen 676a1ede8c Carry tool-call identity and outcome through squad events
Why: F14 squad half - the event port dropped CallID/Status/ErrorReason,
so downstream hosts paired by tool name and showed failures as success.
Evidence: bridge and process-event tests pass (verify-u11b.json);
taps/wrappers preserve the fields; concurrent callbacks copy per-event.
Impact: new omitempty fields only; legacy events byte-identical.
Refs: feature/agent-stack-audit-remediation-b2 U11b
2026-10-22 03:25:28 +00:00
weichen 12a53b0285 Enforce the runtime concurrency cap at queue claim time
Why: F12 - claims counted only per-agent in-flight tasks, so two agents
on one runtime could exceed the runtime cap; count errors were ignored
and silently allowed claims.
Evidence: queue tests pass incl. dual-bound, concurrent-claim and
forced-count-failure cases (verify-u9.json, exit=0).
Impact: single atomic conditional UPDATE; lease and reclaim rules kept.
Refs: feature/agent-stack-audit-remediation-b2 U9
2026-10-22 02:57:17 +00:00
weichen cf83d30743 Persist collaboration state and surface persistence failures
Why: F11 - SQLite runs lost mailbox/handoffs/board across restarts,
identical JSON entries collapsed during decode, and checkpoint or
terminal-save failures were swallowed behind success events.
Evidence: store and checkpoint tests pass incl. injected-failure and
round-trip tables (verify-u8.json, exit=0); schema migration preserves
rows.
Impact: runs survive restarts with collaboration state; save failures
are visible before any completion event; async/on_exit semantics kept.
Refs: feature/agent-stack-audit-remediation-b2 U8
2026-10-22 02:57:17 +00:00
weichen 0f1daa8b89 Align the SQLite run store test with honest step failures
Why: the U3B semantics change removed the failed-with-patches
promotion; TestEngineRunsOnSQLiteStore still encoded the old behavior
and failed under the fixed engine.
Evidence: repair receipt success; sqlite store tests pass
(verify-u3br2.json).
Impact: test-only change.
Refs: feature/agent-stack-audit-remediation-b1 U3B repair
2026-10-21 18:05:51 +00:00
weichen cdd51c8678 Pin nested agent-intent and agent-native to the batch-1 fixes
Why: composition consistency - Dev pins agent-intent at 765a7e7 and
agent-native at the U3A result-fidelity tip; the squad container's
nested copies must match (check-pins).
Evidence: pin only; both libraries verified per-unit.
Impact: nested gitlink pointer changes.
Refs: feature/agent-stack-audit-remediation-b1
2026-10-21 17:45:01 +00:00
weichen 23f161bfb3 Honor executor terminals in retries and interrupted recovery
Why: F05 - files-presence could promote failures to success, recovery
marked crashed steps completed on files alone or reran them blindly,
and retries minted a fresh ExecutionID each attempt.
Evidence: squad tests pass incl. executor result contract and recovery
tables (verify-u3b.json); (nil,nil), canceled and side-effect-possible
semantics handled; interrupted steps without a trusted terminal now
hold for reconciliation instead of auto-advancing.
Impact: retries reuse the step ExecutionID and shared deadline; no
auto-replay of unverified external actions.
Refs: feature/agent-stack-audit-remediation-b1 U3B
2026-10-21 17:20:40 +00:00
weichen 4e6557782a Temporarily disable the cursor profile
Why: U3B hit two consecutive pre-session cursor infrastructure
failures (TLS destruction, then model auto rejection consistent with
the documented monthly quota exhaustion); both runs fail-stopped
before any session started with zero workspace changes.
Evidence: run dirs /tmp/agent-exec-1000/287d14d4… and …/b7ed6a71…
preserved; raw logs 553B and 48B.
Impact: routing only; candidates become antigravity then pi then codex.
Refs: Agent-System-Dev feature/agent-stack-audit-remediation-b1
2026-10-21 16:44:24 +00:00
weichen aa15a5d164 Version resolver configs and invalidate executor caches
Why: F06 - the wrapped-executor cache keyed only backend/role/model, so
allow->deny permission tightening and factory replacement never took
effect on subsequent resolves.
Evidence: squad and adapters/native tests pass (verify-u4.json); config
identity helper covers permission/tool/limiter changes.
Impact: stale executors can no longer widen capabilities; in-flight
tasks are not rolled back (documented).
Refs: feature/agent-stack-audit-remediation-b1 U4
2026-10-21 15:51:53 +00:00
39 changed files with 4058 additions and 172 deletions
+1 -1
View File
@@ -24,7 +24,7 @@ jobs:
echo "SUBMODULE_TOKEN is not configured" >&2
exit 1
fi
git config --global url."https://weichen:${SUBMODULE_TOKEN}@gitea.naitangbaobao.cn/".insteadOf "ssh://git@gitea.naitangbaobao.cn:52222/"
git config --global url."https://weichen:${SUBMODULE_TOKEN}@git-home.u1680832.nyat.app:30319/".insteadOf "ssh://git@git-ssh.u1680832.nyat.app:31411/"
git -c fetch.recurseSubmodules=false submodule update --init --recursive
- name: Install Go 1.27.1
run: |
+1 -1
View File
@@ -31,7 +31,7 @@ jobs:
parsed_server = urlsplit(SERVER)
if parsed_server.scheme not in {"http", "https"} or not parsed_server.netloc:
sys.exit("Checkout server must be the platform-injected http(s) URL")
SSH = "ssh://git@gitea.naitangbaobao.cn:52222/weichen/"
SSH = "ssh://git@git-ssh.u1680832.nyat.app:31411/weichen/"
# 审查过的白名单,独立于候选 .gitmodules:仅列出两个治理 gitlink。
REPOSITORIES = {
"deps/development-guidelines": "Development-Guidelines",
+6 -6
View File
@@ -1,18 +1,18 @@
[submodule "deps/agent-native"]
path = deps/agent-native
url = ssh://git@gitea.naitangbaobao.cn:52222/weichen/Agent-Native.git
url = ssh://git@git-ssh.u1680832.nyat.app:31411/weichen/Agent-Native.git
[submodule "deps/development-guidelines"]
path = deps/development-guidelines
url = ssh://git@gitea.naitangbaobao.cn:52222/weichen/Development-Guidelines.git
url = ssh://git@git-ssh.u1680832.nyat.app:31411/weichen/Development-Guidelines.git
[submodule "deps/error-code-system"]
path = deps/error-code-system
url = ssh://git@gitea.naitangbaobao.cn:52222/weichen/Error-Code-System.git
url = ssh://git@git-ssh.u1680832.nyat.app:31411/weichen/Error-Code-System.git
[submodule "deps/logging-system"]
path = deps/logging-system
url = ssh://git@gitea.naitangbaobao.cn:52222/weichen/Logging-System.git
url = ssh://git@git-ssh.u1680832.nyat.app:31411/weichen/Logging-System.git
[submodule "deps/agent-intent"]
path = deps/agent-intent
url = ssh://git@gitea.naitangbaobao.cn:52222/weichen/Agent-Intent.git
url = ssh://git@git-ssh.u1680832.nyat.app:31411/weichen/Agent-Intent.git
[submodule "deps/development-manager"]
path = deps/development-manager
url = ssh://git@gitea.naitangbaobao.cn:52222/weichen/Development-Manager.git
url = ssh://git@git-ssh.u1680832.nyat.app:31411/weichen/Development-Manager.git
+8 -2
View File
@@ -36,13 +36,19 @@ profiles:
adapter: cursor
executable: "cursor-agent-stable"
model: auto
enabled: true
# 2026-10-21 临时禁用:U3B 连续两次 fail-stop(首次 TLS 会话前故障 553B 日志、
# 二次 "Cannot use this model: auto",与 Backend 配置已记录的月配额耗尽一致),
# 两次均未建立会话、零改动(现场保留)。恢复时改回 enabled: true。
enabled: false
antigravity:
adapter: antigravity
executable: agy
model: auto
enabled: true
# 2026-10-21 临时禁用:U3B 尝试被 400 FAILED_PRECONDITION "User location is
# not supported for the API use" 拒绝,0 token、未建立会话(现场保留);
# 与 Backend 配置已记录的 Gemini 端点不稳一致。恢复时改回 enabled: true。
enabled: false
pi:
adapter: pi
+3 -3
View File
@@ -23,9 +23,9 @@
历史文档与旧链接里的 `git-panel.u1680832.nyat.app:30319` 已失效(TLS 握手失败),
见到一律按过期处理,不要再试。当前权威通道:
- **Web / API(HTTPS)**:`https://gitea.naitangbaobao.cn`(Gitea 1.27.3,
- **Web / API(HTTPS)**:`https://git-home.u1680832.nyat.app:30319`(Gitea 1.27.3,
API 前缀 `/api/v1`;偶发 broken pipe / 超时,重试即可)。
- **Git over SSH**:`ssh://git@gitea.naitangbaobao.cn:52222/weichen/<Repo>.git`
- **Git over SSH**:`ssh://git@git-ssh.u1680832.nyat.app:31411/weichen/<Repo>.git`
(各仓 origin 与 gitlink 均为此格式)。
- **API token**:`~/.secrets/gitea`,两行键值 `URL_PREFIX=…` 与
`ACCESS_TOKEN=<40 位 hex>`;取值 `grep ACCESS_TOKEN ~/.secrets/gitea | cut -d= -f2-`,
@@ -33,7 +33,7 @@
(baseline 硬边界 2:只记录来源、字段与可用状态)。
- **SSH key**:`~/.ssh/home_gitea`(ED25519,公钥指纹
`SHA256:GYXaOXIMeloQ1fKSfdJ/ndgPQzs/Ix7Hk62ekfkfQ20`,Gitea 侧密钥名 GMKtec);
`~/.ssh/config` 已有 `gitea.naitangbaobao.cn(端口 52222)` 主机条目,克隆/推送直接可用。
`~/.ssh/config` 已有 `git-ssh.u1680832.nyat.app` 主机条目,克隆/推送直接可用。
## Agent 执行与新项目准入(强制)
+2 -2
View File
@@ -6,7 +6,7 @@
- 目标/产品边界:通用智能体小队编排底座。角色、蓝图、规划、执行引擎、黑板、HITL 与 HTTP/CLI 组合根。不含预设医疗器械角色或任务模板。不实现 Agent。
- 源码、资产、文档、机器契约位置:Go 模块路径 `agent-squad`。包为 `squad`、`store`、`server`、`cli`、`textextract`、`conformance`。人类文档在根 `README.md`、`policy/`;过程记录在 `docs/records/`。对外 HTTP 错误合同以错误码库信封为准。
- 权威远端、集成基线、受保护分支:`ssh://git@gitea.naitangbaobao.cn:52222/weichen/Agent-Squad.git`,集成基线 `main`。
- 权威远端、集成基线、受保护分支:`ssh://git@git-ssh.u1680832.nyat.app:31411/weichen/Agent-Squad.git`,集成基线 `main`。
- 并行共同基线记录位置、合入通知/主动检查方式:Gitea Issue/PR。跨仓共同起点是本仓提交加上它锁定的 Agent-Native、错误码库、日志库与规范库 SHA。
- 发布线与批准版本/实际制品查询入口:无独立制品库。可执行文件由 `go build` 从固定 gitlink 组合产出。
- 提交/审核/合并/发布身份与权限:仓库所有者 weichen。
@@ -33,7 +33,7 @@
克隆:
```bash
git clone --recurse-submodules ssh://git@gitea.naitangbaobao.cn:52222/weichen/Agent-Squad.git
git clone --recurse-submodules ssh://git@git-ssh.u1680832.nyat.app:31411/weichen/Agent-Squad.git
# 已克隆时:git submodule update --init --recursive
```
+1 -1
View File
@@ -21,7 +21,7 @@ Agent-Native 自己的 `deps/` 里还有一份错误码库、日志库和规范
克隆时带上子模块,否则无法编译。已有克隆若仍留着根目录 `Agent-Native`,先删掉该残留目录再初始化:
```bash
git clone --recurse-submodules ssh://git@gitea.naitangbaobao.cn:52222/weichen/Agent-Squad.git
git clone --recurse-submodules ssh://git@git-ssh.u1680832.nyat.app:31411/weichen/Agent-Squad.git
# 已克隆时:
git submodule sync --recursive
git submodule update --init --recursive
+184 -6
View File
@@ -3,8 +3,13 @@ package native
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"reflect"
"strings"
"sync"
"time"
nativexec "agent-native/adapters/native"
@@ -52,12 +57,58 @@ type Config struct {
PermissionAsk func(ctx context.Context, call llm.ToolCall) runtime.AskVerdict
}
// Identity 返回塑造 Executor 的配置摘要(模型/客户端/工具/Exec 等)。
// RegisterFull 经 BackendResolver.Register 替换工厂时会失效缓存;角色侧
// BashPermission 等由 squad.ConfigIdentity 纳入 Resolve 缓存键,故 allow→deny
// 在下一次 Resolve 即生效,无需重启进程。
func (c Config) Identity() string {
var b strings.Builder
b.WriteString("model=")
b.WriteString(c.DefaultModel)
b.WriteString("\x00turns=")
b.WriteString(fmt.Sprintf("%d", c.MaxTurns))
b.WriteString("\x00timeout=")
b.WriteString(c.Timeout.String())
if c.Client != nil {
b.WriteString("\x00base=")
b.WriteString(c.Client.BaseURL)
b.WriteString("\x00key=")
b.WriteString(c.Client.APIKey)
}
if c.Exec != nil {
b.WriteString("\x00exec.bash=")
b.WriteString(c.Exec.Bash)
b.WriteString("\x00exec.python=")
b.WriteString(c.Exec.Python)
}
if c.Search != nil {
b.WriteString("\x00search=1")
}
if c.Fetch != nil {
b.WriteString("\x00fetch=1")
}
if c.MCP != nil {
b.WriteString("\x00mcp=1")
}
if len(c.SkillsRoots) > 0 {
b.WriteString("\x00skills=")
b.WriteString(strings.Join(c.SkillsRoots, ","))
}
if c.CodeOrchestration {
b.WriteString("\x00code=1")
}
sum := sha256.Sum256([]byte(b.String()))
return hex.EncodeToString(sum[:16])
}
// Register 把 "agent-native" 工厂注册进解析器。
func Register(r *squad.BackendResolver, client *llm.Client, defaultModel string, maxTurns int, timeout time.Duration) {
RegisterFull(r, Config{Client: client, DefaultModel: defaultModel, MaxTurns: maxTurns, Timeout: timeout})
}
// RegisterFull 注册带工具装配的 "agent-native" 工厂。
// 再次调用会替换工厂并失效该后端缓存(见 BackendResolver.Register);
// 配置摘要见 Config.Identity,角色侧见 squad.ConfigIdentity。
func RegisterFull(r *squad.BackendResolver, cfg Config) {
r.Register("agent-native", func(role squad.RoleSpec) (squad.Executor, error) {
opts, err := buildOptions(cfg, role)
@@ -134,12 +185,19 @@ func toNative(req squad.ExecRequest) exec.Request {
}
if req.OnEvent != nil {
prev := req.OnEvent
// 一帧一回调:同步转发,不缓存、不批量重发、不按调用分组——下游看到的
// 就是 native 的到达序。富化在原帧上就地完成(逐帧独立取值),
// 并行工具调用互不串字段,也就不会把交错帧重排成按调用分组。
out.OnEvent = func(ev exec.ProcessEvent) {
// CallID/Status/ErrorReason 是 native 侧的可选字段(U11a 落地):
// 缺席 = legacy,取零值透传,绝不伪造。
extras := probeNativeEventExtras(ev)
prev(squad.ProcessEvent{
Type: ev.Type, Text: ev.Text, Tool: ev.Tool, Detail: ev.Detail,
PromptTokens: ev.PromptTokens, CompletionTokens: ev.CompletionTokens,
LastPromptTokens: ev.LastPromptTokens, LastCompletionTokens: ev.LastCompletionTokens,
CacheReadTokens: ev.CacheReadTokens, CacheWriteTokens: ev.CacheWriteTokens,
CallID: extras.CallID, Status: extras.Status, ErrorReason: extras.ErrorReason,
})
}
}
@@ -147,28 +205,148 @@ func toNative(req squad.ExecRequest) exec.Request {
}
func fromNative(res *exec.Result) *squad.ExecResult {
return &squad.ExecResult{
Success: res.Success, Patches: res.Patches, Logs: res.Logs, ErrorReason: res.ErrorReason,
Retryable: res.Retryable, PromptTokens: res.PromptTokens, CompletionTokens: res.CompletionTokens,
if res == nil {
return nil
}
out := &squad.ExecResult{
Success: res.Success,
Patches: res.Patches,
Logs: res.Logs,
ErrorReason: res.ErrorReason,
Retryable: res.Retryable,
PromptTokens: res.PromptTokens,
CompletionTokens: res.CompletionTokens,
}
// F05/U3b(Squad 侧 contract):
// - native exec.Result 在后续 release 会新增 SideEffects、Canceled、Aborted 字段;
// 本地 deps gitlink 若尚未跟进,从 JSON 探测拿不到这些字段时回落到
// 「legacy reading: side effects = possible」,与既有调用方旧行为一致。
// - 在不影响 ExecResult 默认值的前提下,用 JSON 探测而非硬性修改 deps。
probe, err := probeNativeResultExtras(res)
if err == nil {
out.SideEffects = probe.SideEffects
out.Canceled = probe.Canceled
out.Aborted = probe.Aborted
}
return out
}
// nativeResultExtras 探测 native exec.Result 上的可选字段(结构化 JSON)。
// 缺省(字段不存在或为零值时)按 legacy reading:SideEffects 空串视为 "possible";
// 调用方侧 ExecResult.IsRetryableSideEffectReports() 已经处理该回退。
type nativeResultExtras struct {
SideEffects squad.ExecSideEffects `json:"side_effects,omitempty"`
Canceled bool `json:"canceled,omitempty"`
Aborted bool `json:"aborted,omitempty"`
}
// probeNativeResultExtras 通过 JSON 探测 native Result 上的可选字段。
// 探测出 cancel/abort/side_effects 任何一个就使用它们;探测失败或全部为零值视为
// legacy reading(SideEffects 空 = possible 兜底)。
func probeNativeResultExtras(res *exec.Result) (nativeResultExtras, error) {
if res == nil {
return nativeResultExtras{}, fmt.Errorf("nil result")
}
v := reflect.ValueOf(res).Elem()
if !v.IsValid() {
return nativeResultExtras{}, fmt.Errorf("invalid result")
}
// 用 JSON 做结构化探测:这一路兼容 deps 升级(增字段)、不增字段两种情形。
data, err := json.Marshal(v.Interface())
if err != nil {
return nativeResultExtras{}, err
}
var out nativeResultExtras
if err := json.Unmarshal(data, &out); err != nil {
return nativeResultExtras{}, err
}
return out, nil
}
// nativeEventExtras 是 native exec.ProcessEvent 上的可选字段:工具调用身份与
// 真实结局(U11b,语义与 native 侧一致:空串 = legacy/未知)。
type nativeEventExtras struct {
CallID string
Status string
ErrorReason string
}
// probeNativeEventExtras 读取 native 过程帧上的 CallID/Status/ErrorReason。
//
// 与 probeNativeResultExtras 同一取法(可选字段缺席 = legacy 零值),但改用
// 反射按字段名读取而不是 JSON:本地 deps gitlink 上的 exec.ProcessEvent 尚未
// 跟进这三个字段(U11a 落地),硬写 ev.CallID 会编译不过;反射让同一份代码
// 两侧都成立——字段缺席取零值(既有行为逐字节不变),字段落地后无需改本文件
// 即生效,也不受 native 是否给字段加 JSON tag 影响。每次过程帧只做三次字段
// 定位,不构造中间对象;native 可并行发帧时读到的仍是该帧自己的值。
func probeNativeEventExtras(v any) nativeEventExtras {
rv := reflect.ValueOf(v)
for rv.Kind() == reflect.Pointer || rv.Kind() == reflect.Interface {
if rv.IsNil() {
return nativeEventExtras{}
}
rv = rv.Elem()
}
if rv.Kind() != reflect.Struct {
return nativeEventExtras{}
}
return nativeEventExtras{
CallID: optionalStringField(rv, "CallID"),
Status: optionalStringField(rv, "Status"),
ErrorReason: optionalStringField(rv, "ErrorReason"),
}
}
// optionalStringField 按字段名取字符串字段;缺席、嵌套或类型不符都返回 ""
// (legacy 零值)。只取直接字段,不做嵌套指针下钻。
func optionalStringField(rv reflect.Value, name string) string {
field, ok := rv.Type().FieldByName(name)
if !ok || len(field.Index) != 1 {
return ""
}
value := rv.Field(field.Index[0])
if value.Kind() != reflect.String {
return ""
}
return value.String()
}
// eventTap 在组合根侧再挂一层过程事件观察。帧按值传递(squad.ProcessEvent
// 只含值类型字段),两个下游各自拿到整帧副本,并行工具调用不会互相串字段;
// 回调保持同步,不排队、不缓存。
//
// 扇出以 mu 串行:「一帧 → 全部下游」是一个整体。这不是可有可无的细节:
// native 对并行工具调用是每调用一个 goroutine 发帧,两次帧到达可以真正并发。
// 不串行时,帧 F 交给组合根侧 sink 之后、在交给请求侧 sink 之前,可能被后到
// 的帧 G 抢先递完,于是同一串到达帧在两个下游呈现出不同的顺序——实测就是
// 组合根侧看到 [start(err), start(ok), end(ok), end(err)](到达序),而请求侧
// 看到 [start(ok), end(ok), start(err), end(err)](被按调用重排/分组)。
// 串行扇出后每个下游看到的都是同一串到达序:一个上游帧 → 每个下游恰好一次、
// 顺序不变,且事件富化(CallID/Status/ErrorReason)在桥里就地完成、不在这里
// 重新分帧。mu 只在本次扇出期间持有,不跨帧持有。
type eventTap struct {
next squad.Executor
on func(squad.ProcessEvent)
mu sync.Mutex
}
func (b *eventTap) Name() string { return b.next.Name() }
func (b *eventTap) Execute(ctx context.Context, req squad.ExecRequest) (*squad.ExecResult, error) {
prev := req.OnEvent
req.OnEvent = func(ev squad.ProcessEvent) {
req.OnEvent = b.hook(req.OnEvent)
return b.next.Execute(ctx, req)
}
// hook 把一帧扇出到组合根侧 sink 与请求侧 sink,顺序不变。
func (b *eventTap) hook(prev func(squad.ProcessEvent)) func(squad.ProcessEvent) {
return func(ev squad.ProcessEvent) {
b.mu.Lock()
defer b.mu.Unlock()
b.on(ev)
if prev != nil {
prev(ev)
}
}
return b.next.Execute(ctx, req)
}
// LLM 把 llm.Client 适配成 ModelClient。
+427
View File
@@ -0,0 +1,427 @@
package native
import (
"context"
"reflect"
"sync"
"testing"
"time"
"agent-native/exec"
"agent-squad/squad"
)
// nativeEventWithIDs 模拟 U11a 落地后的 native exec.ProcessEvent 形态:
// 多出的 CallID / Status / ErrorReason,且带 snake_case JSON tag
// (证明 squad 侧读取不受 native 的 tag 写法影响)。
type nativeEventWithIDs struct {
Type string `json:"type"`
Text string `json:"text"`
Tool string `json:"tool"`
Detail string `json:"detail"`
CallID string `json:"call_id,omitempty"`
Status string `json:"status,omitempty"`
ErrorReason string `json:"error_reason,omitempty"`
}
func TestBridgeMapsLegacyNativeEventWithoutFabricatingFields(t *testing.T) {
var got []squad.ProcessEvent
out := toNative(squad.ExecRequest{OnEvent: func(ev squad.ProcessEvent) { got = append(got, ev) }})
if out.OnEvent == nil {
t.Fatal("bridge must install its own event hook when the request has one")
}
out.OnEvent(exec.ProcessEvent{
Type: "tool_execution_end", Text: "found", Tool: "grep", Detail: `{"q":"x"}`,
PromptTokens: 11, CompletionTokens: 22,
LastPromptTokens: 3, LastCompletionTokens: 4, CacheReadTokens: 5, CacheWriteTokens: 6,
})
if len(got) != 1 {
t.Fatalf("want 1 squad event, got %d", len(got))
}
want := squad.ProcessEvent{
Type: "tool_execution_end", Text: "found", Tool: "grep", Detail: `{"q":"x"}`,
PromptTokens: 11, CompletionTokens: 22,
LastPromptTokens: 3, LastCompletionTokens: 4, CacheReadTokens: 5, CacheWriteTokens: 6,
}
if !reflect.DeepEqual(got[0], want) {
t.Fatalf("legacy native event must map byte-identically:\n got %+v\nwant %+v", got[0], want)
}
// 无请求侧回调时不装钩子(既有行为不变)。
if toNative(squad.ExecRequest{}).OnEvent != nil {
t.Fatal("bridge must not install an event hook when the request has none")
}
}
func TestProbeNativeEventExtrasReadsNewFieldsWhenNativeHasThem(t *testing.T) {
// legacy 帧:字段缺席(老 pin)或在场但未填(当前 pin 的零值)→ 一律零值,
// 不臆造身份与结局。
if extras := probeNativeEventExtras(exec.ProcessEvent{Type: "tool_execution_end", Tool: "grep"}); extras != (nativeEventExtras{}) {
t.Fatalf("legacy native event must yield zero extras, got %+v", extras)
}
// U11a 落地后:按字段名读到真实值(与 JSON tag 写法无关)。
future := nativeEventWithIDs{Type: "tool_execution_end", Tool: "grep", CallID: "call_err",
Status: "error", ErrorReason: "grep: permission denied"}
want := nativeEventExtras{CallID: "call_err", Status: "error", ErrorReason: "grep: permission denied"}
if extras := probeNativeEventExtras(future); extras != want {
t.Fatalf("identity/outcome not carried through: %+v", extras)
}
if extras := probeNativeEventExtras(&future); extras != want {
t.Fatalf("pointer and value frames must probe identically: %+v", extras)
}
// 缺席/非字符串字段都回落到零值。
odd := struct {
CallID int
Status string
ErrorReason bool
}{CallID: 7, Status: "ok", ErrorReason: true}
if extras := probeNativeEventExtras(odd); extras != (nativeEventExtras{Status: "ok"}) {
t.Fatalf("non-string/absent fields must stay zero-valued, got %+v", extras)
}
var nilFrame *nativeEventWithIDs
if extras := probeNativeEventExtras(nilFrame); extras != (nativeEventExtras{}) {
t.Fatalf("nil frame must stay zero-valued, got %+v", extras)
}
if extras := probeNativeEventExtras("not a frame"); extras != (nativeEventExtras{}) {
t.Fatalf("non-struct input must stay zero-valued, got %+v", extras)
}
// 嵌套(嵌入)字段不探测:不沿 nil 嵌入指针下钻,也不当作直接字段读取。
nested := struct {
*nativeEventWithIDs
Status string
}{Status: "ok"}
if extras := probeNativeEventExtras(nested); extras != (nativeEventExtras{Status: "ok"}) {
t.Fatalf("nested optional fields must stay zero-valued, got %+v", extras)
}
}
// interleavedToolFrames 是并行工具调用的固定交错帧序(squad 侧形态):
//
// start(call_err) → start(call_ok) → end(call_ok, ok) → end(call_err, error)
//
// 按到达序转发时逐帧等值;任何「按调用分组重排」「按调用缓存后再发」都会把它
// 变成 [start, end, start, end],因此它能把重排/重发类缺陷钉成硬失败。
func interleavedToolFrames() []squad.ProcessEvent {
return []squad.ProcessEvent{
{Type: "tool_execution_start", Tool: "grep", CallID: "call_err"},
{Type: "tool_execution_start", Tool: "grep", CallID: "call_ok"},
{Type: "tool_execution_end", Tool: "grep", CallID: "call_ok",
Status: squad.ToolStatusOK, Text: "found"},
{Type: "tool_execution_end", Tool: "grep", CallID: "call_err",
Status: squad.ToolStatusError, ErrorReason: "grep: permission denied"},
}
}
// interleavedNativeFrames 是 interleavedToolFrames 的 native 侧形态:U11a 落地后
// (当前 pin 即此形态)的帧自带 CallID/Status/ErrorReason,桥读到的就是它们。
func interleavedNativeFrames() []exec.ProcessEvent {
return []exec.ProcessEvent{
{Type: "tool_execution_start", Tool: "grep", CallID: "call_err"},
{Type: "tool_execution_start", Tool: "grep", CallID: "call_ok"},
{Type: "tool_execution_end", Tool: "grep", CallID: "call_ok",
Status: squad.ToolStatusOK, Text: "found"},
{Type: "tool_execution_end", Tool: "grep", CallID: "call_err",
Status: squad.ToolStatusError, ErrorReason: "grep: permission denied"},
}
}
// legacyToolFrames 是同一个交错序列的 legacy squad 形态:身份与结局字段为空串
// (上游未填充),失败真相只在 Text 里——空串 = 未知,不是「无名调用」。
func legacyToolFrames() []squad.ProcessEvent {
return []squad.ProcessEvent{
{Type: "tool_execution_start", Tool: "grep"},
{Type: "tool_execution_start", Tool: "grep"},
{Type: "tool_execution_end", Tool: "grep", Text: "found"},
{Type: "tool_execution_end", Tool: "grep", Text: "grep: permission denied"},
}
}
// legacyNativeFrames 是 legacy native 事件流:帧上没有 CallID/Status/ErrorReason
// (字段不存在,或存在但未填——对探测而言都是零值)。桥必须原样透传。
func legacyNativeFrames() []exec.ProcessEvent {
return []exec.ProcessEvent{
{Type: "tool_execution_start", Tool: "grep"},
{Type: "tool_execution_start", Tool: "grep"},
{Type: "tool_execution_end", Tool: "grep", Text: "found"},
{Type: "tool_execution_end", Tool: "grep", Text: "grep: permission denied"},
}
}
// assertArrivalOrder 断言帧序逐帧等于到达序:一个上游帧对应下游一帧,顺序、
// 身份与结局全部逐帧等值(squad.ProcessEvent 只含值类型字段,可直接比较)。
func assertArrivalOrder(t *testing.T, tag string, got, want []squad.ProcessEvent) {
t.Helper()
if len(got) != len(want) {
t.Fatalf("%s: frames = %d, want %d (每帧一次转发,不多不少不重发):\n%+v", tag, len(got), len(want), got)
}
for i := range want {
if got[i] != want[i] {
t.Fatalf("%s: frame %d = %+v\nwant %+v\n(帧序必须与到达序逐帧一致)", tag, i, got[i], want[i])
}
}
}
// concurrentToolExecutor 并发发两次同名工具调用的过程帧(CallID 不同、一成一败),
// 模拟 native 侧并行工具执行时的帧序交错。
type concurrentToolExecutor struct{}
func (concurrentToolExecutor) Name() string { return "concurrent-tool" }
func (concurrentToolExecutor) Execute(ctx context.Context, req squad.ExecRequest) (*squad.ExecResult, error) {
calls := []struct {
callID, status, errorReason, text string
}{
{callID: "call_ok", status: squad.ToolStatusOK, text: "found"},
{callID: "call_err", status: squad.ToolStatusError, errorReason: "grep: permission denied"},
}
var wg sync.WaitGroup
for _, call := range calls {
wg.Add(1)
go func(callID, status, errorReason, text string) {
defer wg.Done()
if req.OnEvent == nil {
return
}
req.OnEvent(squad.ProcessEvent{Type: "tool_execution_start", Tool: "grep", CallID: callID})
req.OnEvent(squad.ProcessEvent{Type: "tool_execution_end", Tool: "grep", CallID: callID,
Status: status, ErrorReason: errorReason, Text: text})
}(call.callID, call.status, call.errorReason, call.text)
}
wg.Wait()
return &squad.ExecResult{Success: true}, nil
}
// assertPaired 断言按 CallID 配对:每个 call_id 一帧 start、一帧 end,
// 结局属于该调用自身(并行工具调用不串字段)。
func assertPaired(t *testing.T, tag string, frames []squad.ProcessEvent) {
t.Helper()
starts := map[string]int{}
ends := map[string]squad.ProcessEvent{}
for _, ev := range frames {
if ev.CallID == "" {
t.Fatalf("%s: frame without CallID: %+v", tag, ev)
}
switch ev.Type {
case "tool_execution_start":
starts[ev.CallID]++
case "tool_execution_end":
if _, dup := ends[ev.CallID]; dup {
t.Fatalf("%s: duplicate end for %s: %+v", tag, ev.CallID, ev)
}
ends[ev.CallID] = ev
}
}
if len(ends) != 2 {
t.Fatalf("%s: distinct CallIDs = %d, want 2 (%+v)", tag, len(ends), frames)
}
for _, want := range []struct {
callID, status, errorReason string
}{
{callID: "call_ok", status: squad.ToolStatusOK},
{callID: "call_err", status: squad.ToolStatusError, errorReason: "grep: permission denied"},
} {
if starts[want.callID] != 1 {
t.Fatalf("%s: call %s starts = %d, want 1", tag, want.callID, starts[want.callID])
}
end := ends[want.callID]
if end.Status != want.status || end.ErrorReason != want.errorReason {
t.Fatalf("%s: call %s outcome = %s/%q, want %s/%q", tag, want.callID,
end.Status, end.ErrorReason, want.status, want.errorReason)
}
}
}
// TestEventTapKeepsConcurrentToolCallIdentityAndOutcome 在真实并发帧源下验证三件事:
// 身份与结局不串(严格按 CallID 配对)、帧不多发不少发(每个调用一 start 一 end)、
// 所有下游观察到同一串到达序(并发帧源下到达序本身不确定,但必须唯一且一致)。
func TestEventTapKeepsConcurrentToolCallIdentityAndOutcome(t *testing.T) {
var tapped, requested []squad.ProcessEvent
var mu sync.Mutex
tap := &eventTap{next: concurrentToolExecutor{}, on: func(ev squad.ProcessEvent) {
mu.Lock()
defer mu.Unlock()
tapped = append(tapped, ev)
}}
_, err := tap.Execute(context.Background(), squad.ExecRequest{
OnEvent: func(ev squad.ProcessEvent) {
mu.Lock()
defer mu.Unlock()
requested = append(requested, ev)
},
})
if err != nil {
t.Fatal(err)
}
assertPaired(t, "tap", tapped)
assertPaired(t, "request", requested)
if !reflect.DeepEqual(tapped, requested) {
t.Fatalf("every sink must observe the same arrival order (one frame per upstream frame, unbuffered):\ntap %+v\nrequest %+v", tapped, requested)
}
}
// TestNativeFramesReachBothTapSinksInArrivalOrder 走完整一跳:native 帧 → 桥的
// OnEvent(就地富化 CallID/Status/ErrorReason)→ 事件 tap → 组合根侧 sink 与
// 请求侧 sink。两种 native(当前 pin 的带字段帧 / legacy 无字段帧)都要逐帧
// 等值、顺序不变:既不能被按调用分组重排,也不能缓存后再批量发。
func TestNativeFramesReachBothTapSinksInArrivalOrder(t *testing.T) {
for _, tc := range []struct {
name string
native []exec.ProcessEvent
want []squad.ProcessEvent
}{
{name: "current-native", native: interleavedNativeFrames(), want: interleavedToolFrames()},
{name: "legacy-native", native: legacyNativeFrames(), want: legacyToolFrames()},
} {
t.Run(tc.name, func(t *testing.T) {
var mu sync.Mutex
var tapped, requested []squad.ProcessEvent
tap := &eventTap{on: func(ev squad.ProcessEvent) {
mu.Lock()
defer mu.Unlock()
tapped = append(tapped, ev)
}}
nativeHook := toNative(squad.ExecRequest{OnEvent: tap.hook(func(ev squad.ProcessEvent) {
mu.Lock()
defer mu.Unlock()
requested = append(requested, ev)
})}).OnEvent
if nativeHook == nil {
t.Fatal("bridge must install its own event hook when the request has one")
}
for _, ev := range tc.native {
nativeHook(ev)
}
assertArrivalOrder(t, tc.name+"/tap", tapped, tc.want)
assertArrivalOrder(t, tc.name+"/request", requested, tc.want)
if tc.name == "current-native" {
assertPaired(t, tc.name, tapped)
assertPaired(t, tc.name, requested)
}
})
}
}
// TestBridgeForwardsNativeFramesInArrivalOrder 直接钉住桥这一跳的帧序与富化:
// 当前 pin(帧带 CallID/Status/ErrorReason)就地把值带上来,legacy 帧(无这三个
// key)就以空串原样透传——两种情况都必须逐帧等值、顺序不变。
func TestBridgeForwardsNativeFramesInArrivalOrder(t *testing.T) {
for _, tc := range []struct {
name string
native []exec.ProcessEvent
want []squad.ProcessEvent
}{
{name: "current-native", native: interleavedNativeFrames(), want: interleavedToolFrames()},
{name: "legacy-native", native: legacyNativeFrames(), want: legacyToolFrames()},
} {
t.Run(tc.name, func(t *testing.T) {
var got []squad.ProcessEvent
hook := toNative(squad.ExecRequest{OnEvent: func(ev squad.ProcessEvent) {
got = append(got, ev)
}}).OnEvent
if hook == nil {
t.Fatal("bridge must install its own event hook when the request has one")
}
for _, ev := range tc.native {
hook(ev)
}
assertArrivalOrder(t, tc.name, got, tc.want)
if tc.name == "current-native" {
assertPaired(t, tc.name, got)
}
})
}
}
// fanoutWindow 制造扇出竞态窗口:支线 A 逐帧发 call_ok 的两帧;当 A 的第二帧进入
// 组合根侧 sink 时(entered 关闭)支线 B 才发 call_err 的两帧,并在自己的扇出
// 全部返回后关闭 finished。A 在 sink 里等 finished,于是「A 已收下第二帧、还没
// 交给请求侧 sink」这个窗口被固定复现——扇出不串行时 B 会抢先穿过这个窗口。
type fanoutWindow struct {
entered chan struct{}
finished chan struct{}
}
func (fanoutWindow) Name() string { return "fanout-window" }
func (w fanoutWindow) Execute(ctx context.Context, req squad.ExecRequest) (*squad.ExecResult, error) {
if req.OnEvent == nil {
return &squad.ExecResult{Success: true}, nil
}
var wg sync.WaitGroup
wg.Add(2)
go func() {
defer wg.Done()
req.OnEvent(squad.ProcessEvent{Type: "tool_execution_start", Tool: "grep", CallID: "call_ok"})
req.OnEvent(squad.ProcessEvent{Type: "tool_execution_end", Tool: "grep", CallID: "call_ok",
Status: squad.ToolStatusOK, Text: "found"})
}()
go func() {
defer wg.Done()
<-w.entered
req.OnEvent(squad.ProcessEvent{Type: "tool_execution_start", Tool: "grep", CallID: "call_err"})
req.OnEvent(squad.ProcessEvent{Type: "tool_execution_end", Tool: "grep", CallID: "call_err",
Status: squad.ToolStatusError, ErrorReason: "grep: permission denied"})
close(w.finished)
}()
wg.Wait()
return &squad.ExecResult{Success: true}, nil
}
// TestEventTapFansOutOneFramePerSinkInOneOrder 把「扇出是一个整体」钉成确定性失败:
// 组合根侧 sink 在第二帧上主动等另一条支线走完自己的扇出。没有扇出串行化时,
// 请求侧 sink 会先收到 call_err 的两帧,组合根侧 sink 却先收到 call_ok 的两帧——
// 同一串到达帧在两个下游呈现为两种顺序(这正是并发下把帧重排的入口)。
func TestEventTapFansOutOneFramePerSinkInOneOrder(t *testing.T) {
window := fanoutWindow{entered: make(chan struct{}), finished: make(chan struct{})}
var mu sync.Mutex
var tapped, requested []squad.ProcessEvent
tap := &eventTap{next: window, on: func(ev squad.ProcessEvent) {
mu.Lock()
tapped = append(tapped, ev)
seen := len(tapped)
mu.Unlock()
if seen == 2 {
// 不持有采集锁去等:让另一条支线真的能穿过扇出窗口(否则本测试
// 只会测到自己)。
close(window.entered)
select {
case <-window.finished:
case <-time.After(250 * time.Millisecond):
}
}
}}
_, err := tap.Execute(context.Background(), squad.ExecRequest{
OnEvent: func(ev squad.ProcessEvent) {
mu.Lock()
defer mu.Unlock()
requested = append(requested, ev)
},
})
if err != nil {
t.Fatal(err)
}
want := []squad.ProcessEvent{
{Type: "tool_execution_start", Tool: "grep", CallID: "call_ok"},
{Type: "tool_execution_end", Tool: "grep", CallID: "call_ok", Status: squad.ToolStatusOK, Text: "found"},
{Type: "tool_execution_start", Tool: "grep", CallID: "call_err"},
{Type: "tool_execution_end", Tool: "grep", CallID: "call_err",
Status: squad.ToolStatusError, ErrorReason: "grep: permission denied"},
}
assertArrivalOrder(t, "tap", tapped, want)
assertArrivalOrder(t, "request", requested, want)
assertPaired(t, "tap", tapped)
assertPaired(t, "request", requested)
}
func TestEventTapWithoutRequestHookStillTaps(t *testing.T) {
var mu sync.Mutex
var got []squad.ProcessEvent
tap := &eventTap{next: concurrentToolExecutor{}, on: func(ev squad.ProcessEvent) {
mu.Lock()
defer mu.Unlock()
got = append(got, ev)
}}
if _, err := tap.Execute(context.Background(), squad.ExecRequest{}); err != nil {
t.Fatal(err)
}
assertPaired(t, "tap-only", got)
}
+82
View File
@@ -47,3 +47,85 @@ func TestBuildOptionsBashPermissionGating(t *testing.T) {
t.Fatal("missing model must fail-closed")
}
}
func TestConfigIdentityReflectsToolAndClient(t *testing.T) {
base := Config{
Client: &llm.Client{BaseURL: "http://x", APIKey: "k"},
DefaultModel: "m1", MaxTurns: 5, Timeout: time.Minute,
Exec: &tools.ExecConfig{Bash: "/bin/bash"},
}
otherExec := base
otherExec.Exec = &tools.ExecConfig{Bash: "/bin/zsh"}
if base.Identity() == otherExec.Identity() {
t.Fatal("Exec tool-config change must change Config.Identity")
}
otherClient := base
otherClient.Client = &llm.Client{BaseURL: "http://y", APIKey: "k"}
if base.Identity() == otherClient.Identity() {
t.Fatal("client change must change Config.Identity")
}
same := base
if base.Identity() != same.Identity() {
t.Fatal("identical config must keep Identity stable")
}
}
func TestRegisterFullAllowDenyTakesEffectOnNextResolve(t *testing.T) {
r := squad.NewBackendResolver()
cfg := Config{
Client: &llm.Client{BaseURL: "http://x", APIKey: "k"},
DefaultModel: "m1", MaxTurns: 1, Timeout: time.Second,
Exec: &tools.ExecConfig{Bash: "/bin/bash"},
}
RegisterFull(r, cfg)
allowRole := squad.RoleSpec{Role: "eng", Backend: "agent-native", Model: "m1", BashPermission: squad.BashPermissionAllow}
denyRole := squad.RoleSpec{Role: "eng", Backend: "agent-native", Model: "m1", BashPermission: squad.BashPermissionDeny}
eAllow, _, err := r.Resolve(allowRole, "")
if err != nil {
t.Fatal(err)
}
eDeny, _, err := r.Resolve(denyRole, "")
if err != nil {
t.Fatal(err)
}
if eAllow == eDeny {
t.Fatal("allow→deny must not reuse cached executor on next Resolve")
}
// Identity of role configs must diverge so cache keys differ.
if squad.ConfigIdentity(allowRole) == squad.ConfigIdentity(denyRole) {
t.Fatal("role ConfigIdentity must include BashPermission")
}
}
func TestRegisterFullToolConfigReplacementTakesEffect(t *testing.T) {
r := squad.NewBackendResolver()
cfg1 := Config{
Client: &llm.Client{BaseURL: "http://x", APIKey: "k"},
DefaultModel: "m1", MaxTurns: 1, Timeout: time.Second,
Exec: &tools.ExecConfig{Bash: "/bin/bash"},
}
cfg2 := Config{
Client: &llm.Client{BaseURL: "http://x", APIKey: "k"},
DefaultModel: "m1", MaxTurns: 1, Timeout: time.Second,
Exec: &tools.ExecConfig{Bash: "/bin/zsh"},
}
if cfg1.Identity() == cfg2.Identity() {
t.Fatal("precondition: tool configs must differ in Identity")
}
RegisterFull(r, cfg1)
role := squad.RoleSpec{Role: "eng", Backend: "agent-native", Model: "m1", BashPermission: squad.BashPermissionAllow}
e1, _, err := r.Resolve(role, "")
if err != nil {
t.Fatal(err)
}
RegisterFull(r, cfg2)
e2, _, err := r.Resolve(role, "")
if err != nil {
t.Fatal(err)
}
if e1 == e2 {
t.Fatal("RegisterFull tool-config replacement must invalidate cache")
}
}
+35 -10
View File
@@ -189,8 +189,33 @@ func (s *Store) PromoteDeferred(at time.Time) (int, error) {
return int(n), nil
}
// claimSQL is the whole claim: one atomic conditional UPDATE whose two
// guarded counts bound the agent by its own max_concurrency and the runtime by
// max_concurrency over every in-flight task of that runtime (an agent bound to
// it, or a task this runtime already leased). A count that cannot be read
// fails the statement, so the claim is refused instead of allowed.
const claimSQL = `UPDATE task SET status='dispatched', runtime_id=?, lease_until=?, updated_at=?
WHERE id=? AND status='queued'
AND (SELECT COUNT(*) FROM task WHERE agent_id=? AND status IN ('dispatched','running')) < ?
AND (SELECT COUNT(*) FROM task t WHERE t.status IN ('dispatched','running')
AND (t.runtime_id=? OR EXISTS (SELECT 1 FROM agent a WHERE a.id=t.agent_id AND a.runtime_id=?))) < ?`
// claimCapacitySQL counts the agent's own and the runtime's in-flight tasks in
// one read, so the pre-check below matches the guards of claimSQL.
const claimCapacitySQL = `SELECT
(SELECT COUNT(*) FROM task WHERE agent_id=? AND status IN ('dispatched','running')),
(SELECT COUNT(*) FROM task t WHERE t.status IN ('dispatched','running')
AND (t.runtime_id=? OR EXISTS (SELECT 1 FROM agent a WHERE a.id=t.agent_id AND a.runtime_id=?)))`
// Claim takes one queued task whose agent and runtime are under their caps.
// The task becomes dispatched with a five-minute lease.
//
// Both caps live inside the single conditional UPDATE, so concurrent claims
// can overshoot neither the agent cap nor the runtime total: the agent count
// is bounded by agent.max_concurrency and the runtime count (every in-flight
// task whose agent belongs to this runtime, plus every task this runtime
// already leased) by runtime.max_concurrency. A read error is a refusal, never
// a silent allow: the candidate is skipped, and no task is handed out.
func (s *Store) Claim(runtimeID string, at time.Time) (Task, bool, error) {
rt, err := s.getRuntime(runtimeID)
if err != nil {
@@ -223,20 +248,20 @@ func (s *Store) Claim(runtimeID string, at time.Time) (Task, bool, error) {
if agent.RuntimeID != "" && agent.RuntimeID != runtimeID {
continue
}
cap := agent.MaxConcurrency
if rt.MaxConcurrency < cap {
cap = rt.MaxConcurrency
// Pre-check only: the UPDATE below re-counts both caps under the write
// lock. A count that cannot be read means unknown capacity, so refuse.
var agentBusy, runtimeBusy int
err = s.db.QueryRow(claimCapacitySQL, cand.agentID, runtimeID, runtimeID).Scan(&agentBusy, &runtimeBusy)
if err != nil {
return Task{}, false, fmt.Errorf("claim capacity count: %w", err)
}
var running int
_ = s.db.QueryRow(`SELECT COUNT(*) FROM task WHERE agent_id=? AND status IN ('dispatched','running')`, cand.agentID).Scan(&running)
if running >= cap {
if agentBusy >= agent.MaxConcurrency || runtimeBusy >= rt.MaxConcurrency {
continue
}
lease := at.UTC().Add(5 * time.Minute).Format(time.RFC3339Nano)
res, err := s.db.Exec(`UPDATE task SET status='dispatched', runtime_id=?, lease_until=?, updated_at=?
WHERE id=? AND status='queued'
AND (SELECT COUNT(*) FROM task WHERE agent_id=? AND status IN ('dispatched','running')) < ?`,
runtimeID, lease, now(), cand.id, cand.agentID, cap)
res, err := s.db.Exec(claimSQL,
runtimeID, lease, now(), cand.id, cand.agentID, agent.MaxConcurrency,
runtimeID, runtimeID, rt.MaxConcurrency)
if err != nil {
return Task{}, false, err
}
+325
View File
@@ -0,0 +1,325 @@
package platform
import (
"path/filepath"
"sync"
"testing"
"time"
)
// openQueue opens a hermetic store in a temp dir and returns it with its path.
func openQueue(t *testing.T) (*Store, string) {
t.Helper()
path := filepath.Join(t.TempDir(), "queue.db")
s, err := Open(path)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = s.Close() })
return s, path
}
// newQueueRuntime registers a runtime with the given cap.
func newQueueRuntime(t *testing.T, s *Store, workspaceID, name string, cap int) Runtime {
t.Helper()
rt, err := s.RegisterRuntime(Runtime{
WorkspaceID: workspaceID, Name: name, Command: "in-process",
Protocol: "agent-native", MaxConcurrency: cap,
})
if err != nil {
t.Fatal(err)
}
return rt
}
// newQueueAgent creates an agent with the given cap, pinned to runtimeID
// (an empty runtimeID leaves the agent unbound, claimable by any runtime).
func newQueueAgent(t *testing.T, s *Store, workspaceID, name, runtimeID string, cap int) Agent {
t.Helper()
agent, err := s.CreateAgent(Agent{WorkspaceID: workspaceID, Name: name, RuntimeID: runtimeID, MaxConcurrency: cap})
if err != nil {
t.Fatal(err)
}
return agent
}
// queueTask creates an issue and queues one task for agentID.
func queueTask(t *testing.T, s *Store, workspaceID, agentID, title string) Task {
t.Helper()
issue, err := s.CreateIssue(Issue{WorkspaceID: workspaceID, Title: title, Body: title, Status: "todo"})
if err != nil {
t.Fatal(err)
}
task, err := s.enqueue(issue, agentID, false, "")
if err != nil {
t.Fatal(err)
}
return task
}
// taskStatus reads the durable status of one task.
func taskStatus(t *testing.T, s *Store, id string) string {
t.Helper()
var status string
if err := s.db.QueryRow(`SELECT status FROM task WHERE id=?`, id).Scan(&status); err != nil {
t.Fatal(err)
}
return status
}
// R6: the runtime cap bounds every agent on that runtime, not one agent only.
// A per-agent cap of 4 each still yields a single in-flight task at cap 1.
func TestClaimEnforcesRuntimeCapAcrossAgents(t *testing.T) {
for _, tc := range []struct {
name string
bindToRT bool
}{
{name: "agents bound to the runtime", bindToRT: true},
{name: "agents not bound to the runtime", bindToRT: false},
} {
t.Run(tc.name, func(t *testing.T) {
s, _ := openQueue(t)
ws, err := s.CreateWorkspace("lab")
if err != nil {
t.Fatal(err)
}
rt := newQueueRuntime(t, s, ws.ID, "rt", 1)
runtimeID := ""
if tc.bindToRT {
runtimeID = rt.ID
}
first := newQueueAgent(t, s, ws.ID, "a1", runtimeID, 4)
second := newQueueAgent(t, s, ws.ID, "a2", runtimeID, 4)
t1 := queueTask(t, s, ws.ID, first.ID, "one")
t2 := queueTask(t, s, ws.ID, second.ID, "two")
claimed, ok, err := s.Claim(rt.ID, time.Now())
if err != nil || !ok || claimed.ID != t1.ID || claimed.RuntimeID != rt.ID {
t.Fatalf("first claim %+v ok=%v err=%v want task %s on %s", claimed, ok, err, t1.ID, rt.ID)
}
if _, ok, err := s.Claim(rt.ID, time.Now()); err != nil || ok {
t.Fatalf("runtime cap 1 must refuse the other agent: ok=%v err=%v", ok, err)
}
if got := taskStatus(t, s, t2.ID); got != "queued" {
t.Fatalf("second task %q must stay queued while the runtime is full", got)
}
if err := s.FinishTask(claimed.ID, "completed", ""); err != nil {
t.Fatal(err)
}
next, ok, err := s.Claim(rt.ID, time.Now())
if err != nil || !ok || next.ID != t2.ID {
t.Fatalf("claim after release %+v ok=%v err=%v want task %s", next, ok, err, t2.ID)
}
})
}
}
// The agent cap holds when two runtimes compete for the same agent.
func TestClaimEnforcesAgentCapAcrossRuntimes(t *testing.T) {
s, _ := openQueue(t)
ws, err := s.CreateWorkspace("lab")
if err != nil {
t.Fatal(err)
}
rtA := newQueueRuntime(t, s, ws.ID, "rt-a", 2)
rtB := newQueueRuntime(t, s, ws.ID, "rt-b", 2)
agent := newQueueAgent(t, s, ws.ID, "solo", "", 1)
t1 := queueTask(t, s, ws.ID, agent.ID, "one")
t2 := queueTask(t, s, ws.ID, agent.ID, "two")
claimed, ok, err := s.Claim(rtA.ID, time.Now())
if err != nil || !ok || claimed.ID != t1.ID {
t.Fatalf("rt-a claim %+v ok=%v err=%v want task %s", claimed, ok, err, t1.ID)
}
if _, ok, err := s.Claim(rtB.ID, time.Now()); err != nil || ok {
t.Fatalf("agent cap 1 must bound the second runtime: ok=%v err=%v", ok, err)
}
if got := taskStatus(t, s, t2.ID); got != "queued" {
t.Fatalf("second task %q must stay queued while the agent is busy", got)
}
}
// Concurrent claims never overshoot the runtime cap or an agent cap.
func TestClaimConcurrentNeverExceedsCaps(t *testing.T) {
s, _ := openQueue(t)
ws, err := s.CreateWorkspace("lab")
if err != nil {
t.Fatal(err)
}
rt := newQueueRuntime(t, s, ws.ID, "rt", 2)
tight := newQueueAgent(t, s, ws.ID, "tight", rt.ID, 1)
loose := newQueueAgent(t, s, ws.ID, "loose", rt.ID, 8)
other := newQueueAgent(t, s, ws.ID, "other", rt.ID, 8)
for _, agent := range []Agent{tight, loose, other} {
queueTask(t, s, ws.ID, agent.ID, "task for "+agent.Name)
}
const workers = 12
var wg sync.WaitGroup
var mu sync.Mutex
claimed := map[string]string{} // task id -> agent id
failures := []string{}
for i := 0; i < workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
task, ok, err := s.Claim(rt.ID, time.Now())
mu.Lock()
defer mu.Unlock()
switch {
case err != nil:
failures = append(failures, err.Error())
case ok:
claimed[task.ID] = task.AgentID
}
}()
}
wg.Wait()
if len(failures) > 0 {
t.Fatalf("concurrent claims failed: %v", failures)
}
if len(claimed) == 0 {
t.Fatal("no claim landed")
}
if len(claimed) > rt.MaxConcurrency {
t.Fatalf("runtime cap %d exceeded: %v", rt.MaxConcurrency, claimed)
}
perAgent := map[string]int{}
for _, agentID := range claimed {
perAgent[agentID]++
}
caps := map[string]int{tight.ID: tight.MaxConcurrency, loose.ID: loose.MaxConcurrency, other.ID: other.MaxConcurrency}
for agentID, n := range perAgent {
if cap, known := caps[agentID]; !known {
t.Fatalf("unknown agent %s claimed", agentID)
} else if n > cap {
t.Fatalf("agent cap %d exceeded by %d for %s: %v", cap, n, agentID, claimed)
}
}
if got := perAgent[tight.ID]; got > 1 {
t.Fatalf("agent cap 1 exceeded by %d: %v", got, claimed)
}
}
// A completed or lease-expired task gives its slot back.
func TestClaimCapacityFreedByReleaseAndReclaim(t *testing.T) {
s, _ := openQueue(t)
ws, err := s.CreateWorkspace("lab")
if err != nil {
t.Fatal(err)
}
rt := newQueueRuntime(t, s, ws.ID, "rt", 1)
agent := newQueueAgent(t, s, ws.ID, "solo", rt.ID, 1)
t1 := queueTask(t, s, ws.ID, agent.ID, "one")
t2 := queueTask(t, s, ws.ID, agent.ID, "two")
claimed, ok, err := s.Claim(rt.ID, time.Now())
if err != nil || !ok || claimed.ID != t1.ID {
t.Fatalf("first claim %+v ok=%v err=%v want task %s", claimed, ok, err, t1.ID)
}
if _, ok, err := s.Claim(rt.ID, time.Now()); err != nil || ok {
t.Fatalf("the agent is at its cap: ok=%v err=%v", ok, err)
}
if err := s.FinishTask(claimed.ID, "completed", ""); err != nil {
t.Fatal(err)
}
second, ok, err := s.Claim(rt.ID, time.Now())
if err != nil || !ok || second.ID != t2.ID {
t.Fatalf("claim after release %+v ok=%v err=%v want task %s", second, ok, err, t2.ID)
}
n, err := s.RecoverOrphans(time.Now().Add(10 * time.Minute))
if err != nil || n != 1 {
t.Fatalf("reclaim %d %v", n, err)
}
if got := taskStatus(t, s, t2.ID); got != "queued" {
t.Fatalf("reclaimed task %q must be queued again", got)
}
third, ok, err := s.Claim(rt.ID, time.Now())
if err != nil || !ok || third.ID != t2.ID || third.Attempts != 1 {
t.Fatalf("claim after reclaim %+v ok=%v err=%v want task %s with one attempt", third, ok, err, t2.ID)
}
}
// A capacity read that fails refuses the claim; it never allows it.
func TestClaimRefusesWhenReadFails(t *testing.T) {
t.Run("unreadable store", func(t *testing.T) {
s, path := openQueue(t)
ws, err := s.CreateWorkspace("lab")
if err != nil {
t.Fatal(err)
}
rt := newQueueRuntime(t, s, ws.ID, "rt", 2)
agent := newQueueAgent(t, s, ws.ID, "a", rt.ID, 2)
task := queueTask(t, s, ws.ID, agent.ID, "one")
if err := s.Close(); err != nil {
t.Fatal(err)
}
if claimed, ok, err := s.Claim(rt.ID, time.Now()); ok {
t.Fatalf("an unreadable store handed out %+v (err=%v)", claimed, err)
}
again, err := Open(path)
if err != nil {
t.Fatal(err)
}
defer again.Close()
if got := taskStatus(t, again, task.ID); got != "queued" {
t.Fatalf("task %q after a failed capacity read", got)
}
})
t.Run("agent lookup fails", func(t *testing.T) {
s, _ := openQueue(t)
ws, err := s.CreateWorkspace("lab")
if err != nil {
t.Fatal(err)
}
rt := newQueueRuntime(t, s, ws.ID, "rt", 2)
agent := newQueueAgent(t, s, ws.ID, "a", rt.ID, 2)
task := queueTask(t, s, ws.ID, agent.ID, "one")
if _, err := s.db.Exec(`DROP TABLE agent`); err != nil {
t.Fatal(err)
}
claimed, ok, err := s.Claim(rt.ID, time.Now())
if ok || err != nil {
t.Fatalf("a failed agent read must skip the candidate: %+v ok=%v err=%v", claimed, ok, err)
}
if _, err := s.db.Exec(`CREATE TABLE agent (id TEXT PRIMARY KEY, runtime_id TEXT)`); err != nil {
t.Fatal(err)
}
if got := taskStatus(t, s, task.ID); got != "queued" {
t.Fatalf("task %q after a skipped candidate", got)
}
})
// The guarded count runs inside the atomic claim: when it cannot be read
// the statement must fail rather than hand the task out.
t.Run("guard count fails", func(t *testing.T) {
s, _ := openQueue(t)
ws, err := s.CreateWorkspace("lab")
if err != nil {
t.Fatal(err)
}
rt := newQueueRuntime(t, s, ws.ID, "rt", 2)
agent := newQueueAgent(t, s, ws.ID, "a", rt.ID, 2)
task := queueTask(t, s, ws.ID, agent.ID, "one")
if _, err := s.db.Exec(`DROP TABLE agent`); err != nil {
t.Fatal(err)
}
lease := time.Now().UTC().Add(5 * time.Minute).Format(time.RFC3339Nano)
res, err := s.db.Exec(claimSQL, rt.ID, lease, now(), task.ID, agent.ID, 2, rt.ID, rt.ID, rt.MaxConcurrency)
if err == nil {
n, _ := res.RowsAffected()
t.Fatalf("a failed guard count claimed %d task(s)", n)
}
if _, err := s.db.Exec(`CREATE TABLE agent (id TEXT PRIMARY KEY, runtime_id TEXT)`); err != nil {
t.Fatal(err)
}
if got := taskStatus(t, s, task.ID); got != "queued" {
t.Fatalf("task %q after a failed guard count", got)
}
})
}
+164 -21
View File
@@ -22,13 +22,19 @@ func (s *Store) RunStore() *SQLiteStore {
return &SQLiteStore{db: s.db}
}
// runSchemaVersion 是 squad run 存储的 schema 版本(squad_run_schema_version 记档)。
// v1 起步,v2 补齐协作态列(mailbox/handoffs/board)。迁移只做增量:
// 新库直接建到最新形态,旧库经 ALTER TABLE ADD COLUMN 补列(既有行读出为空默认,不丢数据)。
const runSchemaVersion = 2
const runSchema = `
CREATE TABLE IF NOT EXISTS squad_run_state (
ws_dir TEXT PRIMARY KEY,
meta TEXT NOT NULL, workflow TEXT NOT NULL, squad TEXT NOT NULL, bb TEXT NOT NULL,
template TEXT, approved TEXT, repair_budget_remaining INTEGER NOT NULL DEFAULT 0,
repair_budget_granted INTEGER NOT NULL DEFAULT 0,
clarifications TEXT, steering TEXT, usage TEXT, updated_at TEXT NOT NULL
clarifications TEXT, steering TEXT, usage TEXT, updated_at TEXT NOT NULL,
mailbox TEXT, handoffs TEXT, board TEXT
);
CREATE TABLE IF NOT EXISTS squad_run_gate (
ws_dir TEXT PRIMARY KEY, gate TEXT NOT NULL
@@ -41,13 +47,90 @@ CREATE TABLE IF NOT EXISTS squad_run_checkpoint (
CREATE TABLE IF NOT EXISTS squad_run_leader (
ws_dir TEXT PRIMARY KEY, data TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS squad_run_schema_version (
id INTEGER PRIMARY KEY CHECK (id = 1), version INTEGER NOT NULL
);
`
// runStateV2Columns 是 v2 在 squad_run_state 上新增的协作态列(列名为代码内常量,非外部输入)。
var runStateV2Columns = []string{"mailbox", "handoffs", "board"}
// migrate 建表并按版本步进:老库补列后再记版本号,逐次读写都是幂等的
// (每次 Save/Load 都调;补列只在版本落后时发生,且既有行不被改写)。
func (s *SQLiteStore) migrate() error {
_, err := s.db.Exec(runSchema)
if _, err := s.db.Exec(runSchema); err != nil {
return err
}
cur, err := s.runSchemaVersion()
if err != nil {
return err
}
if cur >= runSchemaVersion {
return nil
}
if err := s.addRunStateColumns(runStateV2Columns); err != nil {
return err
}
_, err = s.db.Exec(`INSERT INTO squad_run_schema_version (id, version) VALUES (1, ?)
ON CONFLICT(id) DO UPDATE SET version = excluded.version`, runSchemaVersion)
return err
}
// runSchemaVersion 读取当前 schema 版本;无记录 = 0(v1 及更早的库)。
func (s *SQLiteStore) runSchemaVersion() (int, error) {
var v int
err := s.db.QueryRow(`SELECT version FROM squad_run_schema_version WHERE id = 1`).Scan(&v)
if err != nil {
if err == sql.ErrNoRows {
return 0, nil
}
return 0, err
}
return v, nil
}
// runStateColumns 读出 squad_run_state 现存列名(迁移判存在性用)。
func (s *SQLiteStore) runStateColumns() (map[string]bool, error) {
rows, err := s.db.Query(`PRAGMA table_info(squad_run_state)`)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]bool{}
for rows.Next() {
var cid, notnull, pk int
var name, ctype string
var dflt sql.NullString
if err := rows.Scan(&cid, &name, &ctype, &notnull, &dflt, &pk); err != nil {
return nil, err
}
out[name] = true
}
return out, rows.Err()
}
// addRunStateColumns 为缺失的列补 ALTER TABLE ADD COLUMN(既有行保持原值,
// 新列读出为 NULL → 空默认)。
func (s *SQLiteStore) addRunStateColumns(cols []string) error {
have, err := s.runStateColumns()
if err != nil {
return err
}
for _, col := range cols {
if have[col] {
continue
}
if _, err := s.db.Exec(`ALTER TABLE squad_run_state ADD COLUMN ` + col + ` TEXT`); err != nil {
// 并发迁移:另一进程可能刚补上同一列,复核后不算失败。
if again, err2 := s.runStateColumns(); err2 == nil && again[col] {
continue
}
return fmt.Errorf("migrate squad_run_state: add column %s: %w", col, err)
}
}
return nil
}
// OpenSQLiteStore 独立打开一个 squad.Store(不经 platform.Store 的场景,如测试)。
func OpenSQLiteStore(path string) (*SQLiteStore, error) {
db, err := sql.Open("sqlite", path+"?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)")
@@ -70,54 +153,114 @@ func j(v any) string {
return string(b)
}
// runStateColumn 是 squad_run_state 的一列:列名 + 待编码值(编码失败即报错,
// 不用空串顶替——空串读回来是「空默认」,等于静默丢数据)。
type runStateColumn struct {
name string
val any
}
// runStateColumnsOf 列出 RunState 的全部持久化字段(含协作态 mailbox/handoffs/board)。
func runStateColumnsOf(st *squad.RunState) []runStateColumn {
return []runStateColumn{
{"meta", st.Meta},
{"workflow", st.Plan.Workflow},
{"squad", st.Plan.Squad},
{"bb", st.BB},
{"approved", st.Approved},
{"clarifications", st.Clarifications},
{"steering", st.Steering},
{"usage", st.Usage},
{"mailbox", st.Mailbox},
{"handoffs", st.Handoffs},
{"board", st.Board},
}
}
func (s *SQLiteStore) SaveRun(st *squad.RunState) error {
if err := s.migrate(); err != nil {
return err
}
encoded := map[string]string{}
for _, col := range runStateColumnsOf(st) {
data, err := json.Marshal(col.val)
if err != nil {
return fmt.Errorf("save run %s: encode column %s: %w", st.Meta.WorkDir, col.name, err)
}
encoded[col.name] = string(data)
}
_, err := s.db.Exec(`INSERT INTO squad_run_state
(ws_dir, meta, workflow, squad, bb, template, approved, repair_budget_remaining, repair_budget_granted, clarifications, steering, usage, updated_at)
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?)
(ws_dir, meta, workflow, squad, bb, template, approved, repair_budget_remaining, repair_budget_granted, clarifications, steering, usage, mailbox, handoffs, board, updated_at)
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
ON CONFLICT(ws_dir) DO UPDATE SET
meta=excluded.meta, workflow=excluded.workflow, squad=excluded.squad, bb=excluded.bb,
template=excluded.template, approved=excluded.approved,
repair_budget_remaining=excluded.repair_budget_remaining,
repair_budget_granted=excluded.repair_budget_granted,
clarifications=excluded.clarifications, steering=excluded.steering,
usage=excluded.usage, updated_at=excluded.updated_at`,
st.Meta.WorkDir, j(st.Meta), j(st.Plan.Workflow), j(st.Plan.Squad), j(st.BB),
st.Template, j(st.Approved), st.RepairBudgetRemaining, st.RepairBudgetGranted,
j(st.Clarifications), j(st.Steering), j(st.Usage), now())
usage=excluded.usage, mailbox=excluded.mailbox, handoffs=excluded.handoffs,
board=excluded.board, updated_at=excluded.updated_at`,
st.Meta.WorkDir, encoded["meta"], encoded["workflow"], encoded["squad"], encoded["bb"],
st.Template, encoded["approved"], st.RepairBudgetRemaining, st.RepairBudgetGranted,
encoded["clarifications"], encoded["steering"], encoded["usage"],
encoded["mailbox"], encoded["handoffs"], encoded["board"], now())
return err
}
// LoadRun 按列位置读回 RunState。解码目标逐列具名指定——绝不用 JSON 文本当键:
// 文本相同的两列(例如两条一模一样的澄清/补充指令)会互相覆盖,把一列丢掉。
func (s *SQLiteStore) LoadRun(wsDir string) (*squad.RunState, error) {
if err := s.migrate(); err != nil {
return nil, err
}
st := &squad.RunState{}
var meta, wf, sq, bb, template, approved, clar, steering, usage string
row := s.db.QueryRow(`SELECT meta, workflow, squad, bb, template, approved,
repair_budget_remaining, repair_budget_granted, clarifications, steering, usage
FROM squad_run_state WHERE ws_dir = ?`, wsDir)
// 列 → 扫描目标(顺序与 SQL 一一对应)。
var tpl sql.NullString
var granted int
err := row.Scan(&meta, &wf, &sq, &bb, &template, &approved,
&st.RepairBudgetRemaining, &granted, &clar, &steering, &usage)
if err != nil {
var meta, wf, sq, bb, approved, clar, steering, usage, mailbox, handoffs, board sql.NullString
row := s.db.QueryRow(`SELECT meta, workflow, squad, bb, template, approved,
repair_budget_remaining, repair_budget_granted, clarifications, steering, usage,
mailbox, handoffs, board
FROM squad_run_state WHERE ws_dir = ?`, wsDir)
if err := row.Scan(&meta, &wf, &sq, &bb, &tpl, &approved,
&st.RepairBudgetRemaining, &granted, &clar, &steering, &usage,
&mailbox, &handoffs, &board); err != nil {
if err == sql.ErrNoRows {
return nil, fmt.Errorf("load run %s: %w", wsDir, os.ErrNotExist)
}
return nil, err
}
st.RepairBudgetGranted = granted
for pair, dst := range map[string]any{
meta: &st.Meta, wf: &st.Plan.Workflow, sq: &st.Plan.Squad, bb: &st.BB,
approved: &st.Approved, clar: &st.Clarifications, steering: &st.Steering, usage: &st.Usage,
for _, dec := range []struct {
col string
raw sql.NullString
dst any
}{
{"meta", meta, &st.Meta},
{"workflow", wf, &st.Plan.Workflow},
{"squad", sq, &st.Plan.Squad},
{"bb", bb, &st.BB},
{"approved", approved, &st.Approved},
{"clarifications", clar, &st.Clarifications},
{"steering", steering, &st.Steering},
{"usage", usage, &st.Usage},
{"mailbox", mailbox, &st.Mailbox},
{"handoffs", handoffs, &st.Handoffs},
{"board", board, &st.Board},
} {
if err := json.Unmarshal([]byte(pair), dst); err != nil {
return nil, fmt.Errorf("load run %s: parse: %w", wsDir, err)
// 旧行缺列(NULL)或空串 = 该字段的空默认,不报错、不影响其他列。
if !dec.raw.Valid || dec.raw.String == "" {
continue
}
if err := json.Unmarshal([]byte(dec.raw.String), dec.dst); err != nil {
return nil, fmt.Errorf("load run %s: parse column %s: %w", wsDir, dec.col, err)
}
}
st.Template = template
st.Template = tpl.String
if st.Meta.WorkDir == "" {
// 行键(ws_dir)即权威工作区路径:meta 旧数据缺该字段时由行键兜底。
st.Meta.WorkDir = wsDir
}
if st.Approved == nil {
st.Approved = map[string]bool{}
}
+578 -7
View File
@@ -2,8 +2,14 @@ package platform
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"strings"
"sync"
"testing"
"time"
@@ -11,10 +17,12 @@ import (
)
func newTestRunState(wsDir string) *squad.RunState {
// 固定时刻:多次构造出的状态除工作区路径外完全一致,可直接比对两个后端的读回结果。
now := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
return &squad.RunState{
Meta: squad.RunMetadata{
ID: "run_test", Status: "executing", WorkDir: wsDir,
CreatedAt: time.Now(), UpdatedAt: time.Now(),
CreatedAt: now, UpdatedAt: now,
Metadata: map[string]string{"mode": ""},
},
Plan: squad.SquadPlan{
@@ -27,14 +35,28 @@ func newTestRunState(wsDir string) *squad.RunState {
},
Squad: squad.SquadConfig{SquadName: "demo"},
},
BB: squad.BlackboardState{Facts: []squad.FactItem{{ID: "f1", Claim: "x"}}},
BB: squad.BlackboardState{Facts: []squad.FactItem{{ID: "f1", Claim: "x"}}, Fields: map[string]string{"topic": "demo"}},
Template: "tpl",
Approved: map[string]bool{"step:s1": true},
RepairBudgetRemaining: 1,
RepairBudgetGranted: 0,
Clarifications: []string{"c1"},
Steering: []string{"st1"},
Usage: map[string]*squad.Usage{"writer": {PromptTokens: 10, CompletionTokens: 5, Steps: 1}},
// 内容完全相同的重复条目(且 clarifications 与 steering 的 JSON 文本一样):
// 内容键寻址的解码会把它们互相覆盖掉——这里逐条都要活下来。
Clarifications: []string{"同一句澄清", "同一句澄清"},
Steering: []string{"同一句澄清", "同一句澄清"},
Mailbox: []squad.MailMessage{
{ID: "m1", From: "researcher", To: "writer", Text: "同一句澄清", At: now},
{ID: "m2", From: "researcher", To: "writer", Text: "同一句澄清", At: now},
},
Handoffs: []squad.HandoffRecord{
{FromRole: "researcher", FromStep: "s1", ToRole: "writer", Output: "同一句澄清", At: now},
{FromRole: "writer", FromStep: "s2", ToRole: "reviewer", Output: "同一句澄清", At: now},
},
Board: []squad.BoardTask{
{ID: "s1", Status: "done", Attempts: 1},
{ID: "s2", Status: "claimed", ClaimedBy: "writer", LeaseExpires: now.Add(time.Minute), Attempts: 2},
},
Usage: map[string]*squad.Usage{"writer": {PromptTokens: 10, CompletionTokens: 5, Steps: 1}},
}
}
@@ -57,9 +79,20 @@ func TestSQLiteStoreRoundTrip(t *testing.T) {
if got.Meta.ID != "run_test" || got.Plan.Workflow.WorkflowID != "wf1" || got.Template != "tpl" {
t.Fatalf("loaded = %+v", got.Meta)
}
if !got.Approved["step:s1"] || got.RepairBudgetRemaining != 1 || len(got.Clarifications) != 1 || len(got.Steering) != 1 {
if !got.Approved["step:s1"] || got.RepairBudgetRemaining != 1 ||
len(got.Clarifications) != 2 || len(got.Steering) != 2 {
t.Fatalf("scalars = %+v", got)
}
// 协作态必须真的落库(F11a):重启后邮箱/移交/任务板不能静默丢。
if len(got.Mailbox) != 2 || got.Mailbox[0].ID != "m1" || got.Mailbox[1].ID != "m2" {
t.Fatalf("mailbox = %+v", got.Mailbox)
}
if len(got.Handoffs) != 2 || got.Handoffs[1].ToRole != "reviewer" {
t.Fatalf("handoffs = %+v", got.Handoffs)
}
if len(got.Board) != 2 || got.Board[1].Status != "claimed" || got.Board[1].ClaimedBy != "writer" {
t.Fatalf("board = %+v", got.Board)
}
if got.Usage["writer"].PromptTokens != 10 {
t.Fatalf("usage = %+v", got.Usage)
}
@@ -127,7 +160,7 @@ func TestEngineRunsOnSQLiteStore(t *testing.T) {
}
}
backends := squad.NewBackendResolver()
backends.RegisterInstance("script", stubExecutor{result: &squad.ExecResult{Success: false, Logs: "ok", Patches: map[string][]byte{"output.md": []byte(goodReportLite), "notes.md": []byte(goodReportLite), "fields.json": []byte("{}")}}})
backends.RegisterInstance("script", stubExecutor{result: &squad.ExecResult{Success: true, Logs: "ok", Patches: map[string][]byte{"output.md": []byte(goodReportLite), "notes.md": []byte(goodReportLite), "fields.json": []byte("{}")}}})
blueprints := squad.NewBlueprintRegistry()
if err := blueprints.Register(squad.Blueprint{
Key: "demo", Title: "入库演示", Match: []string{"入库演示"},
@@ -185,4 +218,542 @@ func TestEngineRunsOnSQLiteStore(t *testing.T) {
}
}
// TestEngineFailedResultWithCompletePatchesStaysFailedOnSQLiteStore:F05/U3B 的
// 不变式在 SQLite-store 集成层同样成立——executor 报失败就是失败,patches
// (交付物)齐全也不得把运行翻成 completed:文件只是诊断信息,不是权威结果。
func TestEngineFailedResultWithCompletePatchesStaysFailedOnSQLiteStore(t *testing.T) {
runStore, err := OpenSQLiteStore(filepath.Join(t.TempDir(), "runs.db"))
if err != nil {
t.Fatal(err)
}
defer runStore.Close()
dir := t.TempDir()
reg := squad.NewRoleRegistry()
if err := reg.Register(squad.RoleSpec{Role: "researcher", Backend: "script"}); err != nil {
t.Fatal(err)
}
backends := squad.NewBackendResolver()
// 失败结果但携带全套交付物:旧语义会把它提升为成功,U3B 起必须保持失败。
backends.RegisterInstance("script", stubExecutor{result: &squad.ExecResult{
Success: false, Logs: "stub-refused",
Patches: map[string][]byte{
"output.md": []byte(goodReportLite),
"notes.md": []byte(goodReportLite),
"fields.json": []byte("{}"),
},
}})
blueprints := squad.NewBlueprintRegistry()
if err := blueprints.Register(squad.Blueprint{
Key: "faildemo", Title: "失败演示", Match: []string{"失败演示"},
Steps: []squad.StepSpec{{ID: "s1", Role: "researcher", Title: "调研", Targets: []string{"output.md"}}},
}); err != nil {
t.Fatal(err)
}
eng, err := squad.NewEngine(squad.Options{
DataDir: dir, Registry: reg, Blueprints: blueprints,
Planner: squad.NewRulePlanner(),
Backends: backends, Sink: squad.NopSink{},
Store: runStore,
})
if err != nil {
t.Fatal(err)
}
res, err := eng.Run(context.Background(), squad.Goal{Text: "做一个失败演示", HitlMode: squad.HitlAuto}, squad.RunOptions{})
if err != nil {
t.Fatalf("run: %v", err)
}
if res.Status == squad.RunCompleted {
t.Fatalf("failed executor result must not complete (patches are diagnostic only)")
}
if res.Status != squad.RunFailed {
t.Fatalf("status = %s, want failed", res.Status)
}
if res.Plan == nil || res.Plan.Steps[0].Status != "failed" {
t.Fatalf("s1 should be failed, plan = %+v", res.Plan)
}
if !strings.Contains(res.Error, "stub-refused") {
t.Fatalf("failure reason not preserved: %q", res.Error)
}
// 同一断言穿过 Store 端口:SQLite 里落盘的也是失败,而非 completed。
st, err := runStore.LoadRun(res.WorkDir)
if err != nil {
t.Fatal(err)
}
if st.Plan.Workflow.Status != "failed" {
t.Fatalf("persisted workflow status = %s, want failed", st.Plan.Workflow.Status)
}
if got := st.Plan.Workflow.Steps[0]; got.Status != "failed" || !strings.Contains(got.Error, "stub-refused") {
t.Fatalf("persisted step = %+v", got)
}
// patches 确已物化(文件齐全),依然不翻转状态:权威是执行结果,不是文件。
sub := filepath.FromSlash(st.Plan.Workflow.Steps[0].SubWorkspace)
if _, err := os.Stat(filepath.Join(res.WorkDir, sub, "output.md")); err != nil {
t.Fatalf("diagnostic patch should still be materialized: %v", err)
}
}
const goodReportLite = "# 测试交付物\n\n## 概述\n\n这是由脚本后端生成的合格交付物,长度超过五十个字符以满足最小质检要求。\n"
// ---- F11a/F11b:SQLite 与 FileStore 同一契约、重复条目不丢、旧库升级 ----
// assertSameRunState 逐字段比对两个后端读回的运行状态(同一契约)。
// 字段按规范 JSON 比对:时间等值字段(创建/更新时间、消息时刻、租约到期)在两侧
// 都经历过 JSON 往返,按墙钟等价判定,不受内部表示(时区指针/单调时钟)影响。
func assertSameRunState(t *testing.T, got, want *squad.RunState) {
t.Helper()
sameJSON := func(label string, a, b any) {
t.Helper()
ab, err := json.Marshal(a)
if err != nil {
t.Fatalf("%s: marshal got: %v", label, err)
}
bb, err := json.Marshal(b)
if err != nil {
t.Fatalf("%s: marshal want: %v", label, err)
}
if string(ab) != string(bb) {
t.Fatalf("%s differs:\n got %s\nwant %s", label, ab, bb)
}
}
sameJSON("meta", got.Meta, want.Meta)
sameJSON("plan", got.Plan, want.Plan)
sameJSON("blackboard", got.BB, want.BB)
sameJSON("template", got.Template, want.Template)
sameJSON("approved", got.Approved, want.Approved)
sameJSON("repair_budget_remaining", got.RepairBudgetRemaining, want.RepairBudgetRemaining)
sameJSON("repair_budget_granted", got.RepairBudgetGranted, want.RepairBudgetGranted)
sameJSON("clarifications", got.Clarifications, want.Clarifications)
sameJSON("steering", got.Steering, want.Steering)
sameJSON("mailbox", got.Mailbox, want.Mailbox)
sameJSON("handoffs", got.Handoffs, want.Handoffs)
sameJSON("board", got.Board, want.Board)
sameJSON("usage", got.Usage, want.Usage)
}
// TestRunStoreContractParity:FileStore 与 SQLiteStore 满足同一份 Store 契约——
// 同一份运行状态(含 mailbox/handoffs/board)写进去,读回来必须一致。
func TestRunStoreContractParity(t *testing.T) {
sq, err := OpenSQLiteStore(filepath.Join(t.TempDir(), "runs.db"))
if err != nil {
t.Fatal(err)
}
defer sq.Close()
fileWS := filepath.Join(t.TempDir(), "runs", "run_file")
sqlWS := filepath.Join(t.TempDir(), "runs", "run_sql")
// FileStore 侧的模板文件由工作区装配期写出(PrepareWorkspace),先补齐,
// 使两侧的 SaveRun/LoadRun 覆盖同一批字段。
if err := os.MkdirAll(fileWS, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(fileWS, "template.md"), []byte("tpl"), 0o644); err != nil {
t.Fatal(err)
}
want := newTestRunState(fileWS)
if err := (squad.FileStore{}).SaveRun(want); err != nil {
t.Fatal(err)
}
gotFile, err := (squad.FileStore{}).LoadRun(fileWS)
if err != nil {
t.Fatal(err)
}
sqlState := newTestRunState(sqlWS)
if err := sq.SaveRun(sqlState); err != nil {
t.Fatal(err)
}
gotSQL, err := sq.LoadRun(sqlWS)
if err != nil {
t.Fatal(err)
}
// SQLite 读回的内容与写进去的一致。
assertSameRunState(t, gotSQL, sqlState)
// 两个后端的读回结果逐字段一致(除工作区路径这一运行自身的键)。
assertSameRunState(t, gotFile, want)
fileMeta, sqlMeta := gotFile.Meta, gotSQL.Meta
fileMeta.WorkDir, sqlMeta.WorkDir = "", ""
fileJSON, _ := json.Marshal(fileMeta)
sqlJSON, _ := json.Marshal(sqlMeta)
if string(fileJSON) != string(sqlJSON) {
t.Fatalf("meta parity broken:\n file %s\n sql %s", fileJSON, sqlJSON)
}
}
// TestSQLiteRunStoreKeepsIdenticalEntries:内容相同的两条澄清/补充指令必须都活着
// (旧实现拿 JSON 文本当键,两条同文条目只剩一条)。
func TestSQLiteRunStoreKeepsIdenticalEntries(t *testing.T) {
sq, err := OpenSQLiteStore(filepath.Join(t.TempDir(), "runs.db"))
if err != nil {
t.Fatal(err)
}
defer sq.Close()
ws := filepath.Join(t.TempDir(), "runs", "run_dup")
st := newTestRunState(ws)
if err := sq.SaveRun(st); err != nil {
t.Fatal(err)
}
got, err := sq.LoadRun(ws)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(got.Clarifications, []string{"同一句澄清", "同一句澄清"}) {
t.Fatalf("clarifications = %#v, want both identical entries", got.Clarifications)
}
if !reflect.DeepEqual(got.Steering, []string{"同一句澄清", "同一句澄清"}) {
t.Fatalf("steering = %#v, want both identical entries", got.Steering)
}
// 邮箱正文相同、ID 不同:两条都要在(不靠正文寻址)。
if len(got.Mailbox) != 2 || got.Mailbox[0].ID != "m1" || got.Mailbox[1].ID != "m2" {
t.Fatalf("mailbox = %+v", got.Mailbox)
}
}
// legacyRunStateDDL 是 F11 修复前的 squad_run_state(无协作态列),用于验证升级路径。
const legacyRunStateDDL = `
CREATE TABLE squad_run_state (
ws_dir TEXT PRIMARY KEY,
meta TEXT NOT NULL, workflow TEXT NOT NULL, squad TEXT NOT NULL, bb TEXT NOT NULL,
template TEXT, approved TEXT, repair_budget_remaining INTEGER NOT NULL DEFAULT 0,
repair_budget_granted INTEGER NOT NULL DEFAULT 0,
clarifications TEXT, steering TEXT, usage TEXT, updated_at TEXT NOT NULL
);
`
// TestSQLiteRunStoreUpgradesLegacySchema:旧库(缺 mailbox/handoffs/board 列)打开后
// 补列升级,既有一行原样读出,协作态为空默认不报错、不丢其他字段。
func TestSQLiteRunStoreUpgradesLegacySchema(t *testing.T) {
path := filepath.Join(t.TempDir(), "runs.db")
legacy, err := sql.Open("sqlite", path)
if err != nil {
t.Fatal(err)
}
if _, err := legacy.Exec(legacyRunStateDDL); err != nil {
t.Fatal(err)
}
ws := filepath.Join(t.TempDir(), "runs", "run_legacy")
meta, _ := json.Marshal(squad.RunMetadata{ID: "run_legacy", WorkDir: ws, Status: "executing"})
wf, _ := json.Marshal(squad.WorkflowPlan{WorkflowID: "wf_old", Status: "in_progress"})
sqCfg, _ := json.Marshal(squad.SquadConfig{SquadName: "legacy"})
bb, _ := json.Marshal(squad.BlackboardState{Fields: map[string]string{"k": "v"}})
approved, _ := json.Marshal(map[string]bool{"step:s1": true})
clar, _ := json.Marshal([]string{"旧澄清", "旧澄清"})
usage, _ := json.Marshal(map[string]*squad.Usage{"writer": {PromptTokens: 7, CompletionTokens: 3, Steps: 2}})
if _, err := legacy.Exec(`INSERT INTO squad_run_state
(ws_dir, meta, workflow, squad, bb, template, approved, repair_budget_remaining,
repair_budget_granted, clarifications, steering, usage, updated_at)
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?)`,
ws, meta, wf, sqCfg, bb, "旧模板", approved, 2, 1, clar, `["旧补充"]`, usage, "2026-01-01T00:00:00Z"); err != nil {
t.Fatal(err)
}
if err := legacy.Close(); err != nil {
t.Fatal(err)
}
sq, err := OpenSQLiteStore(path)
if err != nil {
t.Fatalf("open legacy db: %v", err)
}
defer sq.Close()
got, err := sq.LoadRun(ws)
if err != nil {
t.Fatalf("load upgraded row: %v", err)
}
// 旧数据无损。
if got.Meta.ID != "run_legacy" || got.Plan.Workflow.WorkflowID != "wf_old" ||
got.Plan.Squad.SquadName != "legacy" || got.Template != "旧模板" {
t.Fatalf("legacy row corrupted: %+v", got)
}
if !got.Approved["step:s1"] || got.RepairBudgetRemaining != 2 || got.RepairBudgetGranted != 1 {
t.Fatalf("legacy scalars lost: %+v", got)
}
if !reflect.DeepEqual(got.Clarifications, []string{"旧澄清", "旧澄清"}) {
t.Fatalf("legacy clarifications lost: %#v", got.Clarifications)
}
if !reflect.DeepEqual(got.Steering, []string{"旧补充"}) {
t.Fatalf("legacy steering lost: %#v", got.Steering)
}
if got.Usage["writer"].PromptTokens != 7 || got.BB.Fields["k"] != "v" {
t.Fatalf("legacy usage/blackboard lost: %+v %+v", got.Usage, got.BB)
}
// 缺列读出为空默认。
if len(got.Mailbox) != 0 || len(got.Handoffs) != 0 || len(got.Board) != 0 {
t.Fatalf("new columns should default to empty: %+v %+v %+v", got.Mailbox, got.Handoffs, got.Board)
}
// 版本已记档,补列只做一次。
var version int
if err := sq.db.QueryRow(`SELECT version FROM squad_run_schema_version WHERE id = 1`).Scan(&version); err != nil {
t.Fatalf("schema version not recorded: %v", err)
}
if version != runSchemaVersion {
t.Fatalf("schema version = %d, want %d", version, runSchemaVersion)
}
// 升级后写入协作态并再次读回。
got.Mailbox = []squad.MailMessage{{ID: "m1", From: "a", To: "b", Text: "升级后写入"}}
got.Board = []squad.BoardTask{{ID: "s1", Status: "done"}}
got.Meta.Status = "completed"
if err := sq.SaveRun(got); err != nil {
t.Fatalf("save after upgrade: %v", err)
}
again, err := sq.LoadRun(ws)
if err != nil {
t.Fatal(err)
}
if len(again.Mailbox) != 1 || again.Mailbox[0].Text != "升级后写入" ||
len(again.Board) != 1 || again.Board[0].ID != "s1" {
t.Fatalf("post-upgrade collaboration state lost: %+v %+v", again.Mailbox, again.Board)
}
if !reflect.DeepEqual(again.Clarifications, []string{"旧澄清", "旧澄清"}) {
t.Fatalf("upgrade write clobbered legacy clarifications: %#v", again.Clarifications)
}
}
// TestSQLiteStoreKeepsCollaborationStateAcrossRestart:进程重启(新连接)后,
// 邮箱消息、所有权移交记录与任务板状态都还在——恢复不必从头重建协作上下文。
func TestSQLiteStoreKeepsCollaborationStateAcrossRestart(t *testing.T) {
dbPath := filepath.Join(t.TempDir(), "runs.db")
runStore, err := OpenSQLiteStore(dbPath)
if err != nil {
t.Fatal(err)
}
defer runStore.Close()
dir := t.TempDir()
reg := squad.NewRoleRegistry()
for _, role := range []string{"researcher", "writer", "reviewer"} {
if err := reg.Register(squad.RoleSpec{Role: role, Backend: "script"}); err != nil {
t.Fatal(err)
}
}
backends := squad.NewBackendResolver()
backends.RegisterInstance("script", stubExecutor{result: &squad.ExecResult{
Success: true, Logs: "ok",
Patches: map[string][]byte{"output.md": []byte(goodReportLite), "notes.md": []byte(goodReportLite), "fields.json": []byte("{}")},
}})
blueprints := squad.NewBlueprintRegistry()
if err := blueprints.Register(squad.Blueprint{
Key: "handoff", Title: "移交演示", Match: []string{"移交演示"},
Steps: []squad.StepSpec{
{ID: "s1", Role: "researcher", Title: "调研", Targets: []string{"notes.md", "fields.json"}},
{ID: "s2", Role: "writer", Title: "成文", Targets: []string{"output.md"}, Handoff: "reviewer"},
{ID: "s3", Role: "reviewer", Title: "终审", Targets: []string{"output.md"}},
},
}); err != nil {
t.Fatal(err)
}
if err := blueprints.Register(squad.Blueprint{
Key: "board", Title: "任务板演示", Match: []string{"任务板演示"},
Steps: []squad.StepSpec{
{ID: "a", Role: "researcher", Title: "任务a", Targets: []string{"notes.md"}},
{ID: "b", Role: "writer", Title: "任务b", Needs: []string{"a"}, Targets: []string{"output.md"}},
},
}); err != nil {
t.Fatal(err)
}
eng, err := squad.NewEngine(squad.Options{
DataDir: dir, Registry: reg, Blueprints: blueprints,
Planner: squad.NewRulePlanner(),
Backends: backends, Sink: squad.NopSink{},
Store: runStore,
})
if err != nil {
t.Fatal(err)
}
ctx := context.Background()
handed, err := eng.Run(ctx, squad.Goal{Text: "做一个移交演示", HitlMode: squad.HitlAuto}, squad.RunOptions{})
if err != nil || handed.Status != squad.RunCompleted {
t.Fatalf("handoff run: %v %+v", err, handed)
}
if err := eng.PostMessage(ctx, handed.RunID, squad.MailMessage{From: "human", To: "writer", Text: "补充口径"}); err != nil {
t.Fatalf("post message: %v", err)
}
boarded, err := eng.Run(ctx, squad.Goal{Text: "做一个任务板演示", HitlMode: squad.HitlAuto},
squad.RunOptions{Topology: squad.TopologyTaskBoard})
if err != nil || boarded.Status != squad.RunCompleted {
t.Fatalf("board run: %v %+v", err, boarded)
}
// 重启 = 新连接(新进程会重新 migrate)。
restarted, err := OpenSQLiteStore(dbPath)
if err != nil {
t.Fatal(err)
}
defer restarted.Close()
gotHanded, err := restarted.LoadRun(handed.WorkDir)
if err != nil {
t.Fatalf("reload handoff run: %v", err)
}
if len(gotHanded.Handoffs) != 1 || gotHanded.Handoffs[0].FromRole != "writer" || gotHanded.Handoffs[0].ToRole != "reviewer" {
t.Fatalf("handoffs lost across restart: %+v", gotHanded.Handoffs)
}
if len(gotHanded.Mailbox) != 1 || gotHanded.Mailbox[0].Text != "补充口径" || gotHanded.Mailbox[0].To != "writer" {
t.Fatalf("mailbox lost across restart: %+v", gotHanded.Mailbox)
}
gotBoarded, err := restarted.LoadRun(boarded.WorkDir)
if err != nil {
t.Fatalf("reload board run: %v", err)
}
if len(gotBoarded.Board) != 2 {
t.Fatalf("board lost across restart: %+v", gotBoarded.Board)
}
for _, task := range gotBoarded.Board {
if task.Status != "done" {
t.Fatalf("board task %s = %s, want done", task.ID, task.Status)
}
}
}
// faultingStore 在真实 SQLiteStore 之上注入落盘故障(其余方法透传)。
type faultingStore struct {
inner squad.Store
failCheckpoint bool
failTerminal bool
}
func (f *faultingStore) SaveRun(st *squad.RunState) error {
if f.failTerminal && squad.RunStatus(st.Meta.Status).Terminal() {
return fmt.Errorf("injected sqlite run state save failure (%s)", st.Meta.Status)
}
return f.inner.SaveRun(st)
}
func (f *faultingStore) LoadRun(wsDir string) (*squad.RunState, error) { return f.inner.LoadRun(wsDir) }
func (f *faultingStore) SaveCheckpoint(wsDir string, cp squad.Checkpoint) error {
if f.failCheckpoint {
return fmt.Errorf("injected sqlite checkpoint save failure (superstep %d)", cp.Superstep)
}
return f.inner.SaveCheckpoint(wsDir, cp)
}
func (f *faultingStore) LoadLatestCheckpoint(wsDir string) (*squad.Checkpoint, error) {
return f.inner.LoadLatestCheckpoint(wsDir)
}
func (f *faultingStore) SaveGate(wsDir string, gate *squad.HitlGate) error {
return f.inner.SaveGate(wsDir, gate)
}
func (f *faultingStore) LoadGate(wsDir string) (*squad.HitlGate, error) {
return f.inner.LoadGate(wsDir)
}
func (f *faultingStore) SaveLeader(wsDir string, data []byte) error {
return f.inner.SaveLeader(wsDir, data)
}
func (f *faultingStore) LoadLeader(wsDir string) ([]byte, error) { return f.inner.LoadLeader(wsDir) }
// eventTally 收集事件类型计数(断言没有成功事件)。
type eventTally struct {
mu sync.Mutex
seen []string
sink squad.ProgressSink
}
func newEventTally() *eventTally {
t := &eventTally{}
t.sink = squad.ProgressFunc(func(ev squad.Event) {
t.mu.Lock()
defer t.mu.Unlock()
t.seen = append(t.seen, ev.Type)
})
return t
}
func (t *eventTally) has(typ string) bool {
t.mu.Lock()
defer t.mu.Unlock()
for _, got := range t.seen {
if got == typ {
return true
}
}
return false
}
func (t *eventTally) types() []string {
t.mu.Lock()
defer t.mu.Unlock()
return append([]string(nil), t.seen...)
}
// TestSQLiteStoreSurfacesSaveFailures:SQLite 后端上的落盘故障必须可见——
// 三档 checkpoint 模式下都不发 run_completed,运行要么报错、要么以失败收场。
func TestSQLiteStoreSurfacesSaveFailures(t *testing.T) {
for _, mode := range []string{"sync", "async", "on_exit"} {
for _, fault := range []string{"checkpoint", "terminal"} {
t.Run(mode+"/"+fault, func(t *testing.T) {
runStore, err := OpenSQLiteStore(filepath.Join(t.TempDir(), "runs.db"))
if err != nil {
t.Fatal(err)
}
defer runStore.Close()
dir := t.TempDir()
reg := squad.NewRoleRegistry()
for _, role := range []string{"researcher", "writer", "reviewer"} {
if err := reg.Register(squad.RoleSpec{Role: role, Backend: "script"}); err != nil {
t.Fatal(err)
}
}
backends := squad.NewBackendResolver()
backends.RegisterInstance("script", stubExecutor{result: &squad.ExecResult{
Success: true, Logs: "ok",
Patches: map[string][]byte{"output.md": []byte(goodReportLite), "notes.md": []byte(goodReportLite), "fields.json": []byte("{}")},
}})
blueprints := squad.NewBlueprintRegistry()
if err := blueprints.Register(squad.Blueprint{
Key: "demo", Title: "故障演示", Match: []string{"故障演示"},
Steps: []squad.StepSpec{
{ID: "s1", Role: "researcher", Title: "调研", Targets: []string{"notes.md", "fields.json"}},
{ID: "s2", Role: "writer", Title: "成文", Targets: []string{"output.md"}},
{ID: "s3", Role: "reviewer", Title: "终审", Targets: []string{"output.md"}},
},
}); err != nil {
t.Fatal(err)
}
store := &faultingStore{inner: runStore,
failCheckpoint: fault == "checkpoint", failTerminal: fault == "terminal"}
tally := newEventTally()
eng, err := squad.NewEngine(squad.Options{
DataDir: dir, Registry: reg, Blueprints: blueprints,
Planner: squad.NewRulePlanner(), Backends: backends,
Sink: tally.sink, Store: store,
})
if err != nil {
t.Fatal(err)
}
res, err := eng.Run(context.Background(),
squad.Goal{Text: "做一个故障演示", HitlMode: squad.HitlAuto},
squad.RunOptions{CheckpointMode: mode})
if tally.has(squad.EvRunCompleted) {
t.Fatalf("run_completed emitted despite %s save failure: %v", fault, tally.types())
}
if fault == "checkpoint" && mode == "sync" {
if err == nil {
t.Fatalf("sync boundary failure must surface as error: %+v", res)
}
if !strings.Contains(err.Error(), "persistence failure") {
t.Fatalf("error not marked as persistence failure: %v", err)
}
return
}
if err != nil {
t.Fatalf("%s/%s: unexpected infra error: %v", mode, fault, err)
}
if res.Status != squad.RunFailed {
t.Fatalf("%s/%s: want failed, got %s", mode, fault, res.Status)
}
if !strings.Contains(res.Error, "persistence failure") {
t.Fatalf("%s/%s: failure not explained: %q", mode, fault, res.Error)
}
if !tally.has(squad.EvRunFailed) {
t.Fatalf("%s/%s: no run_failed event: %v", mode, fault, tally.types())
}
})
}
}
}
+104 -15
View File
@@ -2,12 +2,18 @@ package squad
import (
"context"
"encoding/json"
"os"
"path/filepath"
"sync"
"testing"
"time"
)
// jsonForTest 包装 json.Marshal 让测试里构型参数不拉锁。是仅为占用 import 走
// 公共路径,不污染生产代码。
func jsonForTest(v any) ([]byte, error) { return json.Marshal(v) }
// perStepRecordingBackend 按子工作区归因计数(steps 的 role→tasks/NN_role 布局)。
type perStepRecordingBackend struct {
mu sync.Mutex
@@ -59,6 +65,9 @@ func newAttemptTestEngine(t *testing.T, backend *perStepRecordingBackend, dir st
return eng
}
// flipToInterrupted 把该步骤手动置为 running,同时清除 trusted terminal
// record(模拟「崩溃于执行中」状态)。清除让该路径能够测试 F05 走的
// 「无 record」分支。
func flipToInterrupted(t *testing.T, workDir string, stepIdx int) {
t.Helper()
st, err := loadRunState(workDir)
@@ -75,6 +84,16 @@ func flipToInterrupted(t *testing.T, workDir string, stepIdx int) {
if err := st.save(); err != nil {
t.Fatal(err)
}
// 同时清除该步骤的 trusted terminal record(模拟未达记录写入点崩溃)。
terminals := loadStepTerminalsForWorkDir(workDir)
if terminals != nil {
stepID := st.Plan.Workflow.Steps[stepIdx].ID
if _, ok := terminals[stepID]; ok {
delete(terminals, stepID)
data, _ := jsonForTest(terminals)
_ = os.WriteFile(filepath.Join(workDir, "step_terminals.json"), data, 0o644)
}
}
}
// TestAttemptIDFormat:attempt id 稳定落盘且形如 runID/stepID/N(Q1 WP3)。
@@ -98,8 +117,15 @@ func TestAttemptIDFormat(t *testing.T) {
}
// TestKilledAtSuperstepBoundarySideEffectsOnce:杀进程于 superstep 边界后恢复,
// 已完成步骤不重新执行(执行次数=1);running 且产物齐全的步骤按 attempt 判定
// 完成、不 rerun——Q1 验收第一条(WP3)。
// 已完成步骤不重新执行(执行次数=1)。
//
// F05/U3b:running 崩溃、文件齐全、却无 trusted terminal record(U3b 前被视为
// “副作用真发生”自动完成;U3b 后仅记录 step.Status=needs_reconciliation,
// 后台不再自动 rerun/完成。本测试验证该迁变后的语义不变量:
//
// - running 且无 record:需要 reconciliation,不隐默 rerun;
// - file presence 不替 trusted record 完成步骤。
// - 若事先写入 trusted record(手工合成),即便文件缺失也恢复为 completed。
func TestKilledAtSuperstepBoundarySideEffectsOnce(t *testing.T) {
backend := &perStepRecordingBackend{}
dir := t.TempDir()
@@ -125,22 +151,78 @@ func TestKilledAtSuperstepBoundarySideEffectsOnce(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if res.Status != RunCompleted {
t.Fatalf("resume: %+v", res)
// F05:需 reconciliation(auto 模式),不等于 completed。
if res.Status != RunWaitingReconciliation {
t.Fatalf("want waiting_reconciliation (auto mode), got %s", res.Status)
}
if res.Plan == nil || len(res.Plan.Steps) == 0 {
t.Fatal("missing plan")
}
var recon int
for _, s := range res.Plan.Steps {
if s.Status != "completed" && s.Status != "skipped" {
t.Fatalf("step %s not finished: %s", s.ID, s.Status)
if s.ID == "s2" && s.Status == StepStatusNeedsReconciliation {
recon++
}
}
if recon != 1 {
t.Fatalf("step s2 not in needs_reconciliation (got plan=%+v)", res.Plan.Steps)
}
// 重要:重复执行的次数不应发生隐默 rerun(本测试用 auto 模式运行走到 reconciliation)。
for _, s := range res.Plan.Steps {
got := backend.countOf(filepath.Base(filepath.FromSlash(s.SubWorkspace)))
if got != before[s.ID] {
t.Fatalf("step %s executed %d times after kill (before=%d)——已完成/running 步骤被重复执行", s.ID, got, before[s.ID])
t.Fatalf("step %s executed %d times after kill (before=%d)——running 步骤被隐默 rerun", s.ID, got, before[s.ID])
}
}
}
// TestInterruptedMissingProductsRetries:running 崩溃且产物缺失时按 auto 一次重试(WP3)。
func TestInterruptedMissingProductsRetries(t *testing.T) {
// TestTrustedRecordCompletesMissingFiles:trusted terminal record 存在 + 文件缺失
// → 恢复为 completed(files 仍作诊断,state 是权威)。
func TestTrustedRecordCompletesMissingFiles(t *testing.T) {
backend := &perStepRecordingBackend{}
dir := t.TempDir()
eng := newAttemptTestEngine(t, backend, dir)
res, err := eng.Run(context.Background(), Goal{Text: "trusted record 完成", HitlMode: HitlAuto}, RunOptions{})
if err != nil || res.Status != RunCompleted {
t.Fatalf("setup run: %v %+v", err, res)
}
// 制造「第二步走 running 但 trusted record 已存在」的场景:先写入 record,再
// flip 到 running + 删 file。
st, err := loadRunState(res.WorkDir)
if err != nil {
t.Fatal(err)
}
sub2 := filepath.FromSlash(st.Plan.Workflow.Steps[1].SubWorkspace)
flipToInterrupted(t, res.WorkDir, 1)
if err := os.Remove(filepath.Join(res.WorkDir, sub2, "output.md")); err != nil {
t.Fatal(err)
}
// 手工写入 trusted record(这能覆盖「后端状态已落盘但文件被外部删」该返回
// 已完成的合法语义)。
terminals := map[string]stepTerminalRecord{
"s2": {StepID: "s2", AttemptID: st.Plan.Workflow.Steps[1].AttemptID, Terminal: "completed", SideEffects: string(ExecSideEffectsNone), At: time.Now()},
}
data, _ := jsonForTest(terminals)
if err := os.WriteFile(filepath.Join(res.WorkDir, "step_terminals.json"), data, 0o644); err != nil {
t.Fatal(err)
}
before := backend.countOf(filepath.Base(sub2))
eng2 := newAttemptTestEngine(t, backend, dir)
res, err = eng2.Resume(context.Background(), res.RunID)
if err != nil {
t.Fatal(err)
}
if res.Status != RunCompleted {
t.Fatalf("want completed via trusted record, got %s", res.Status)
}
if got := backend.countOf(filepath.Base(sub2)); got != before {
t.Fatalf("trusted-record step reran (before=%d after=%d)——record 需被信任为 completed", before, got)
}
}
// TestInterruptedMissingProductsEntersReconciliation:F05/U3b:running 崩溃 + 无
// trusted record + 文件缺失 → auto 模式不再隐默 rerun,进入 waiting_reconciliation。
func TestInterruptedMissingProductsEntersReconciliation(t *testing.T) {
backend := &perStepRecordingBackend{}
dir := t.TempDir()
eng := newAttemptTestEngine(t, backend, dir)
@@ -148,7 +230,6 @@ func TestInterruptedMissingProductsRetries(t *testing.T) {
if err != nil || res.Status != RunCompleted {
t.Fatalf("setup run: %v %+v", err, res)
}
// 第二步停 running 并删掉其交付物(副作用未发生的形态)。
flipToInterrupted(t, res.WorkDir, 1)
st, err := loadRunState(res.WorkDir)
if err != nil {
@@ -158,16 +239,24 @@ func TestInterruptedMissingProductsRetries(t *testing.T) {
if err := os.Remove(filepath.Join(res.WorkDir, sub, "output.md")); err != nil {
t.Fatal(err)
}
before := backend.countOf(filepath.Base(sub))
eng2 := newAttemptTestEngine(t, backend, dir)
res, err = eng2.Resume(context.Background(), res.RunID)
if err != nil {
t.Fatal(err)
}
if res.Status != RunCompleted {
t.Fatalf("resume: %+v", res)
if res.Status != RunWaitingReconciliation {
t.Fatalf("want waiting_reconciliation, got %s", res.Status)
}
got := backend.countOf(filepath.Base(sub))
if got != 2 {
t.Fatalf("missing-product step executed %d times, want 2(auto 一次重试)", got)
// 后续 Resume 仍旧是 reconciliation,不再 rerun。
res2, err := eng2.Resume(context.Background(), res.RunID)
if err != nil {
t.Fatal(err)
}
if res2.Status != RunWaitingReconciliation {
t.Fatalf("second resume shouldn't rerun, got %s", res2.Status)
}
if got := backend.countOf(filepath.Base(sub)); got != before {
t.Fatalf("missing-files step reran (before=%d after=%d)——record 缺失不应 rerun", before, got)
}
}
+156 -9
View File
@@ -1,7 +1,9 @@
package squad
import (
"context"
"fmt"
"strings"
"sync"
"error-code-system/errcode"
@@ -16,11 +18,20 @@ type BackendFactory func(role RoleSpec) (Executor, error)
// agent-native 自研内核,产品也可注册 "pi-agent" 等外部智能体工厂。
// 未知名字 fail-closed,绝不静默落 stub。角色声明 MaxConcurrentTasks 时
// 以并发限流包装实现对象级串行(每个角色一份缓存实例,限流跨步骤共享)。
//
// 缓存键含 ConfigIdentity 与工厂世代,故 BashPermission/工具配置收紧或工厂
// 替换在下一次 Resolve 即生效。并发 gate 按「后端名+角色」独立于执行器版本
// 持有:配置换代不会另起限流器而绕过角色上限。
//
// 边界:已在飞的任务不会被回滚或取消;收紧权限/降低并发只影响后续 Resolve
// 与后续 Acquire。需要即时撤权的宿主应在每次工具调用前再校验。
type BackendResolver struct {
mu sync.RWMutex
factories map[string]BackendFactory
underlying map[string]Executor // 无工厂直挂的后端实例
wrapped map[string]Executor // 角色(含模型绑定)→ 已包装实例缓存
wrapped map[string]Executor // 执行器配置身份 → 已包装实例缓存
factoryGen map[string]uint64 // Register/RegisterInstance 世代,替换即递增
gates map[string]*roleConcurrencyGate
}
// NewBackendResolver 创建解析器并注册内置后端。
@@ -29,23 +40,40 @@ func NewBackendResolver() *BackendResolver {
factories: make(map[string]BackendFactory),
underlying: make(map[string]Executor),
wrapped: make(map[string]Executor),
factoryGen: make(map[string]uint64),
gates: make(map[string]*roleConcurrencyGate),
}
r.RegisterInstance("stub", &StubExecutor{})
return r
}
// Register 注册后端工厂(名字如 "agent-native"、"pi-agent")。
// 替换同名工厂会使该后端下已缓存的包装执行器失效。
func (r *BackendResolver) Register(name string, f BackendFactory) {
r.mu.Lock()
defer r.mu.Unlock()
r.factories[name] = f
r.invalidateLocked(name)
}
// RegisterInstance 直接注册后端实例(忽略角色级绑定)。
// 替换同名实例会使该后端下已缓存的包装执行器失效;"stub" 语义保持不变。
func (r *BackendResolver) RegisterInstance(name string, b Executor) {
r.mu.Lock()
defer r.mu.Unlock()
r.underlying[name] = b
r.invalidateLocked(name)
}
func (r *BackendResolver) invalidateLocked(name string) {
r.factoryGen[name]++
prefix := name + "\x00"
for k := range r.wrapped {
if strings.HasPrefix(k, prefix) {
delete(r.wrapped, k)
}
}
// gates 按角色身份保留:换工厂/配置不得丢掉进行中的并发计数。
}
// ListBackends 返回可用后端名(排序稳定)。
@@ -73,8 +101,8 @@ func (r *BackendResolver) ListBackends() []string {
}
// Resolve 解析角色的执行后端:Role.Backend → Options.DefaultBackend → "stub"。
// MaxConcurrentTasks > 1 时以 agent-native 并发限流包装(per-role 缓存,跨步骤共享;
// 创建段持写锁防并行首解析竞态——否则并发步骤会各拿一份限流器)。
// MaxConcurrentTasks >= 1 时以共享角色 gate 包装(跨执行器版本共用;
// 创建段持写锁防并行首解析竞态——否则并发步骤会各拿一份执行器)。
func (r *BackendResolver) Resolve(role RoleSpec, defaultBackend string) (Executor, string, error) {
name := role.Backend
if name == "" {
@@ -83,30 +111,44 @@ func (r *BackendResolver) Resolve(role RoleSpec, defaultBackend string) (Executo
if name == "" {
name = "stub"
}
cacheKey := name + "\x00" + role.Role + "\x00" + role.Model
// 后端名快照在锁外获取(Resolve 持锁期间再调 ListBackends 会自死锁)。
names := r.ListBackends()
r.mu.RLock()
gen := r.factoryGen[name]
cacheKey := resolveCacheKey(name, role, gen)
cached, ok := r.wrapped[cacheKey]
factory, hasFactory := r.factories[name]
under, hasUnder := r.underlying[name]
r.mu.RUnlock()
if ok {
if max := role.MaxConcurrentTasks; max >= 1 {
r.mu.Lock()
if g := r.gates[limiterKey(name, role.Role)]; g != nil {
g.setMax(max)
}
r.mu.Unlock()
}
return cached, name, nil
}
r.mu.Lock()
defer r.mu.Unlock()
gen = r.factoryGen[name]
cacheKey = resolveCacheKey(name, role, gen)
// 双检:等写锁期间可能已被并发解析填充。
if b, ok := r.wrapped[cacheKey]; ok {
if max := role.MaxConcurrentTasks; max >= 1 {
if g := r.gates[limiterKey(name, role.Role)]; g != nil {
g.setMax(max)
}
}
return b, name, nil
}
var b Executor
if hasUnder {
if under, hasUnder := r.underlying[name]; hasUnder {
b = under
} else if hasFactory {
} else if factory, hasFactory := r.factories[name]; hasFactory {
var err error
b, err = factory(role)
if err != nil {
@@ -117,9 +159,114 @@ func (r *BackendResolver) Resolve(role RoleSpec, defaultBackend string) (Executo
}
// MaxConcurrentTasks>=1 即包装限流(1 = 对象级串行;0 = 不限)。
// gate 按角色身份共享,与 ConfigIdentity / 工厂世代无关。
if max := role.MaxConcurrentTasks; max >= 1 {
b = WrapWithConcurrencyLimit(b, max)
g := r.gateLocked(name, role.Role, max)
b = &gatedExecutor{inner: b, gate: g}
}
r.wrapped[cacheKey] = b
return b, name, nil
}
func resolveCacheKey(name string, role RoleSpec, gen uint64) string {
return name + "\x00" + role.Role + "\x00" + ConfigIdentity(role) + "\x00" +
fmt.Sprintf("%d", gen) + "\x00" + limiterStamp(role.MaxConcurrentTasks)
}
func limiterKey(backend, role string) string {
return backend + "\x00" + role
}
func (r *BackendResolver) gateLocked(backend, role string, max int) *roleConcurrencyGate {
key := limiterKey(backend, role)
g, ok := r.gates[key]
if !ok {
g = newRoleConcurrencyGate(max)
r.gates[key] = g
return g
}
g.setMax(max)
return g
}
// roleConcurrencyGate 是按角色共享的并发上限。
// 执行器配置换代仍复用同一 gate,故旧版本在飞任务计入同一上限。
type roleConcurrencyGate struct {
mu sync.Mutex
max int
active int
notify chan struct{} // 关闭并替换以唤醒等待者
}
func newRoleConcurrencyGate(max int) *roleConcurrencyGate {
if max <= 0 {
max = 1
}
return &roleConcurrencyGate{max: max, notify: make(chan struct{})}
}
func (g *roleConcurrencyGate) setMax(max int) {
if max <= 0 {
max = 1
}
g.mu.Lock()
defer g.mu.Unlock()
g.max = max
g.signalLocked()
}
func (g *roleConcurrencyGate) signalLocked() {
close(g.notify)
g.notify = make(chan struct{})
}
func (g *roleConcurrencyGate) acquire(ctx context.Context) error {
for {
g.mu.Lock()
if g.active < g.max {
g.active++
g.mu.Unlock()
return nil
}
notify := g.notify
g.mu.Unlock()
select {
case <-ctx.Done():
return ctx.Err()
case <-notify:
}
}
}
func (g *roleConcurrencyGate) release() {
g.mu.Lock()
defer g.mu.Unlock()
if g.active > 0 {
g.active--
}
g.signalLocked()
}
// gatedExecutor 把共享角色 gate 套在具体执行器上。
type gatedExecutor struct {
inner Executor
gate *roleConcurrencyGate
}
func (e *gatedExecutor) Name() string {
if e.inner == nil {
return "gated"
}
return e.inner.Name()
}
func (e *gatedExecutor) Execute(ctx context.Context, req ExecRequest) (*ExecResult, error) {
if err := e.gate.acquire(ctx); err != nil {
return &ExecResult{
Success: false,
ErrorReason: fmt.Sprintf("agent concurrency wait canceled: %v", err),
}, err
}
defer e.gate.release()
return e.inner.Execute(ctx, req)
}
+405
View File
@@ -0,0 +1,405 @@
package squad
import (
"context"
"sync"
"sync/atomic"
"testing"
"time"
)
type labeledExec struct {
label string
perm string
tools string
delay time.Duration
enter func()
leave func()
}
func (e *labeledExec) Name() string { return e.label }
func (e *labeledExec) Execute(ctx context.Context, req ExecRequest) (*ExecResult, error) {
if e.enter != nil {
e.enter()
}
if e.leave != nil {
defer e.leave()
}
if e.delay > 0 {
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(e.delay):
}
}
return &ExecResult{Success: true, Logs: e.label + "|" + e.perm + "|" + e.tools}, nil
}
func TestConfigIdentityChangesWithBashAndTools(t *testing.T) {
a := RoleSpec{Role: "r", Model: "m", BashPermission: BashPermissionAllow}
d := RoleSpec{Role: "r", Model: "m", BashPermission: BashPermissionDeny}
if ConfigIdentity(a) == ConfigIdentity(d) {
t.Fatal("allow and deny must differ")
}
t1 := RoleSpec{Role: "r", Model: "m", Tools: []string{"a", "b"}}
t2 := RoleSpec{Role: "r", Model: "m", Tools: []string{"b", "a"}}
if ConfigIdentity(t1) != ConfigIdentity(t2) {
t.Fatal("tool order must not change identity")
}
t3 := RoleSpec{Role: "r", Model: "m", Tools: []string{"a", "c"}}
if ConfigIdentity(t1) == ConfigIdentity(t3) {
t.Fatal("tool set change must change identity")
}
same := RoleSpec{Role: "r", Model: "m", BashPermission: BashPermissionAllow, MaxConcurrentTasks: 1}
bump := RoleSpec{Role: "r", Model: "m", BashPermission: BashPermissionAllow, MaxConcurrentTasks: 8}
if ConfigIdentity(same) != ConfigIdentity(bump) {
t.Fatal("MaxConcurrentTasks must not affect ConfigIdentity")
}
}
func TestResolveAllowDenyTighteningAndLoosening(t *testing.T) {
var builds atomic.Int32
r := NewBackendResolver()
r.Register("cap", func(role RoleSpec) (Executor, error) {
builds.Add(1)
return &labeledExec{label: "cap", perm: role.BashPermission}, nil
})
allow := RoleSpec{Role: "writer", Backend: "cap", Model: "m1", BashPermission: BashPermissionAllow}
e1, _, err := r.Resolve(allow, "")
if err != nil {
t.Fatal(err)
}
res, _ := e1.Execute(context.Background(), ExecRequest{})
if res.Logs != "cap|allow|" {
t.Fatalf("want allow executor, got %q", res.Logs)
}
deny := RoleSpec{Role: "writer", Backend: "cap", Model: "m1", BashPermission: BashPermissionDeny}
e2, _, err := r.Resolve(deny, "")
if err != nil {
t.Fatal(err)
}
res, _ = e2.Execute(context.Background(), ExecRequest{})
if res.Logs != "cap|deny|" {
t.Fatalf("allow→deny must take effect on next Resolve, got %q", res.Logs)
}
if e1 == e2 {
t.Fatal("tightened config must not reuse cached allow executor")
}
e3, _, err := r.Resolve(allow, "")
if err != nil {
t.Fatal(err)
}
res, _ = e3.Execute(context.Background(), ExecRequest{})
if res.Logs != "cap|allow|" {
t.Fatalf("deny→allow must take effect, got %q", res.Logs)
}
if builds.Load() < 2 {
t.Fatalf("factory must rebuild on permission change, builds=%d", builds.Load())
}
}
func TestResolveFactoryReplacementTakesEffect(t *testing.T) {
r := NewBackendResolver()
r.Register("cap", func(role RoleSpec) (Executor, error) {
return &labeledExec{label: "v1", perm: role.BashPermission}, nil
})
role := RoleSpec{Role: "writer", Backend: "cap", Model: "m", BashPermission: BashPermissionAllow}
e1, _, err := r.Resolve(role, "")
if err != nil {
t.Fatal(err)
}
res, _ := e1.Execute(context.Background(), ExecRequest{})
if res.Logs != "v1|allow|" {
t.Fatalf("got %q", res.Logs)
}
r.Register("cap", func(role RoleSpec) (Executor, error) {
return &labeledExec{label: "v2", perm: role.BashPermission}, nil
})
e2, _, err := r.Resolve(role, "")
if err != nil {
t.Fatal(err)
}
res, _ = e2.Execute(context.Background(), ExecRequest{})
if res.Logs != "v2|allow|" {
t.Fatalf("factory replacement must take effect, got %q", res.Logs)
}
}
func TestResolveToolConfigChangeTakesEffect(t *testing.T) {
r := NewBackendResolver()
r.Register("cap", func(role RoleSpec) (Executor, error) {
tools := ""
if len(role.Tools) > 0 {
tools = role.Tools[0]
}
return &labeledExec{label: "cap", tools: tools}, nil
})
base := RoleSpec{Role: "writer", Backend: "cap", Model: "m", Tools: []string{"read"}}
e1, _, err := r.Resolve(base, "")
if err != nil {
t.Fatal(err)
}
res, _ := e1.Execute(context.Background(), ExecRequest{})
if res.Logs != "cap||read" {
t.Fatalf("got %q", res.Logs)
}
next := RoleSpec{Role: "writer", Backend: "cap", Model: "m", Tools: []string{"write"}}
e2, _, err := r.Resolve(next, "")
if err != nil {
t.Fatal(err)
}
res, _ = e2.Execute(context.Background(), ExecRequest{})
if res.Logs != "cap||write" {
t.Fatalf("tool-config change must take effect, got %q", res.Logs)
}
}
func TestResolveUnchangedConfigReusesCache(t *testing.T) {
var builds atomic.Int32
r := NewBackendResolver()
r.Register("cap", func(role RoleSpec) (Executor, error) {
builds.Add(1)
return &labeledExec{label: "cap", perm: role.BashPermission}, nil
})
role := RoleSpec{Role: "writer", Backend: "cap", Model: "m", BashPermission: BashPermissionAsk}
e1, _, err := r.Resolve(role, "")
if err != nil {
t.Fatal(err)
}
e2, _, err := r.Resolve(role, "")
if err != nil {
t.Fatal(err)
}
if e1 != e2 {
t.Fatal("unchanged config must reuse cached executor")
}
if builds.Load() != 1 {
t.Fatalf("want 1 factory call, got %d", builds.Load())
}
}
func TestResolveParallelFirstDoesNotDuplicate(t *testing.T) {
var builds atomic.Int32
r := NewBackendResolver()
r.Register("cap", func(role RoleSpec) (Executor, error) {
builds.Add(1)
time.Sleep(20 * time.Millisecond) // widen race window
return &labeledExec{label: "cap"}, nil
})
role := RoleSpec{Role: "writer", Backend: "cap", Model: "m", MaxConcurrentTasks: 2}
const n = 32
var wg sync.WaitGroup
got := make([]Executor, n)
errs := make([]error, n)
wg.Add(n)
for i := 0; i < n; i++ {
i := i
go func() {
defer wg.Done()
got[i], _, errs[i] = r.Resolve(role, "")
}()
}
wg.Wait()
for i, err := range errs {
if err != nil {
t.Fatalf("resolve %d: %v", i, err)
}
}
for i := 1; i < n; i++ {
if got[i] != got[0] {
t.Fatalf("parallel first Resolve must share one executor")
}
}
if builds.Load() != 1 {
t.Fatalf("want single factory construction, got %d", builds.Load())
}
}
func TestSharedGateAcrossVersionsAndLimiterChange(t *testing.T) {
var (
mu sync.Mutex
current int
peak int
)
hold := make(chan struct{})
entered := make(chan struct{}, 8)
trackEnter := func() {
mu.Lock()
current++
if current > peak {
peak = current
}
mu.Unlock()
entered <- struct{}{}
<-hold
}
trackLeave := func() {
mu.Lock()
current--
mu.Unlock()
}
r := NewBackendResolver()
r.Register("cap", func(role RoleSpec) (Executor, error) {
return &labeledExec{
label: "cap",
perm: role.BashPermission,
enter: trackEnter,
leave: trackLeave,
}, nil
})
allow := RoleSpec{Role: "writer", Backend: "cap", Model: "m", BashPermission: BashPermissionAllow, MaxConcurrentTasks: 1}
eOld, _, err := r.Resolve(allow, "")
if err != nil {
t.Fatal(err)
}
errCh := make(chan error, 4)
go func() { _, err := eOld.Execute(context.Background(), ExecRequest{}); errCh <- err }()
<-entered // old holds the sole slot
deny := RoleSpec{Role: "writer", Backend: "cap", Model: "m", BashPermission: BashPermissionDeny, MaxConcurrentTasks: 1}
eNew, _, err := r.Resolve(deny, "")
if err != nil {
t.Fatal(err)
}
if eOld == eNew {
t.Fatal("version bump must produce a new executor")
}
secondDone := make(chan struct{})
go func() {
_, err := eNew.Execute(context.Background(), ExecRequest{})
errCh <- err
close(secondDone)
}()
select {
case <-secondDone:
t.Fatal("new-version execute must wait on shared role cap while old is in-flight")
case <-time.After(40 * time.Millisecond):
}
// Raising MaxConcurrentTasks must update the shared gate (not spawn a bypassing limiter).
deny2 := RoleSpec{Role: "writer", Backend: "cap", Model: "m", BashPermission: BashPermissionDeny, MaxConcurrentTasks: 2}
eLim, _, err := r.Resolve(deny2, "")
if err != nil {
t.Fatal(err)
}
go func() { _, err := eLim.Execute(context.Background(), ExecRequest{}); errCh <- err }()
// After setMax(2): waiting eNew can enter (active=2). eLim must still wait.
select {
case <-entered: // eNew
case <-time.After(time.Second):
t.Fatal("limiter raise must admit the waiting new-version task")
}
select {
case <-entered:
t.Fatal("third task must not bypass shared cap after limiter change")
case <-time.After(40 * time.Millisecond):
}
close(hold)
for i := 0; i < 3; i++ {
if err := <-errCh; err != nil {
t.Fatal(err)
}
}
mu.Lock()
p := peak
mu.Unlock()
if p > 2 {
t.Fatalf("shared gate must cap total in-flight across versions; peak=%d", p)
}
}
func TestInFlightOldVersionCannotExceedCap(t *testing.T) {
var (
mu sync.Mutex
current int
peak int
)
enter := func() {
mu.Lock()
current++
if current > peak {
peak = current
}
mu.Unlock()
}
leave := func() {
mu.Lock()
current--
mu.Unlock()
}
r := NewBackendResolver()
r.Register("cap", func(role RoleSpec) (Executor, error) {
return &labeledExec{
label: "cap",
perm: role.BashPermission,
delay: 80 * time.Millisecond,
enter: enter,
leave: leave,
}, nil
})
v1 := RoleSpec{Role: "writer", Backend: "cap", Model: "m", BashPermission: BashPermissionAllow, MaxConcurrentTasks: 2}
e1, _, err := r.Resolve(v1, "")
if err != nil {
t.Fatal(err)
}
v2 := RoleSpec{Role: "writer", Backend: "cap", Model: "m", BashPermission: BashPermissionDeny, MaxConcurrentTasks: 2}
e2, _, err := r.Resolve(v2, "")
if err != nil {
t.Fatal(err)
}
var wg sync.WaitGroup
// 2 from old version + 2 from new version; shared cap=2 ⇒ peak must stay ≤2.
for _, e := range []Executor{e1, e1, e2, e2} {
wg.Add(1)
e := e
go func() {
defer wg.Done()
_, _ = e.Execute(context.Background(), ExecRequest{})
}()
}
wg.Wait()
mu.Lock()
p := peak
mu.Unlock()
if p > 2 {
t.Fatalf("in-flight old+new must not exceed role cap; peak=%d", p)
}
}
func TestRegisterInstanceStubSemantics(t *testing.T) {
r := NewBackendResolver()
e, name, err := r.Resolve(RoleSpec{Role: "x"}, "")
if err != nil {
t.Fatal(err)
}
if name != "stub" {
t.Fatalf("want stub, got %s", name)
}
if e.Name() != "stub" {
t.Fatalf("want stub executor, got %s", e.Name())
}
custom := &labeledExec{label: "stub-custom"}
r.RegisterInstance("stub", custom)
e2, _, err := r.Resolve(RoleSpec{Role: "x"}, "")
if err != nil {
t.Fatal(err)
}
if e2.Name() != "stub-custom" {
t.Fatalf("RegisterInstance stub replace must take effect, got %s", e2.Name())
}
}
+224
View File
@@ -2,9 +2,11 @@ package squad
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"sync"
"testing"
)
@@ -111,3 +113,225 @@ func TestGraphContinuityRejectsStructuralChange(t *testing.T) {
t.Fatalf("expected continuity rejection, got %v", err)
}
}
// ---- F11c:落盘失败的诚实性 ----
// faultStore 包装底层 Store 注入落盘故障,其余路径照常透传。
// failCheckpoint:任何 SaveCheckpoint 失败(async 档在出口 flush 时暴露)。
// failTerminalSave:Meta 进入终态时的 SaveRun 失败(finalize/manualPass 的收口写)。
type faultStore struct {
Store
mu sync.Mutex
failCheckpoint bool
failTerminalSave bool
checkpointWrites int
terminalAttempts int
}
func (f *faultStore) SaveCheckpoint(wsDir string, cp Checkpoint) error {
f.mu.Lock()
f.checkpointWrites++
shouldFail := f.failCheckpoint
f.mu.Unlock()
if shouldFail {
return fmt.Errorf("injected checkpoint write failure (superstep %d)", cp.Superstep)
}
return f.Store.SaveCheckpoint(wsDir, cp)
}
func (f *faultStore) SaveRun(st *RunState) error {
f.mu.Lock()
f.terminalAttempts++
shouldFail := f.failTerminalSave && RunStatus(st.Meta.Status).Terminal()
f.mu.Unlock()
if shouldFail {
return fmt.Errorf("injected terminal run state failure (%s)", st.Meta.Status)
}
return f.Store.SaveRun(st)
}
func (f *faultStore) writes() int {
f.mu.Lock()
defer f.mu.Unlock()
return f.checkpointWrites
}
// eventRecorder 按序收集引擎事件(顺序断言用)。
type eventRecorder struct {
mu sync.Mutex
evs []Event
sink ProgressSink
}
func newEventRecorder() *eventRecorder {
r := &eventRecorder{}
r.sink = ProgressFunc(func(ev Event) {
r.mu.Lock()
defer r.mu.Unlock()
r.evs = append(r.evs, ev)
})
return r
}
func (r *eventRecorder) types() []string {
r.mu.Lock()
defer r.mu.Unlock()
out := make([]string, 0, len(r.evs))
for _, ev := range r.evs {
out = append(out, ev.Type)
}
return out
}
func (r *eventRecorder) has(t string) bool {
for _, got := range r.types() {
if got == t {
return true
}
}
return false
}
// newTestEngineWithStore 与 newTestEngine 同装配,额外注入 Store 与事件收集 sink。
func newTestEngineWithStore(t *testing.T, store Store, rec *eventRecorder) *SquadEngine {
t.Helper()
dir := t.TempDir()
reg := NewRoleRegistry()
for _, role := range []string{"researcher", "writer", "reviewer"} {
if err := reg.Register(RoleSpec{Role: role, Backend: "script"}); err != nil {
t.Fatal(err)
}
}
backends := NewBackendResolver()
backends.RegisterInstance("script", &scriptBackend{script: map[string]map[string]string{
"01_researcher": {"notes.md": goodReport, "fields.json": "{}"},
"02_writer": {"output.md": goodReport},
"03_reviewer": {"output.md": goodReport},
}})
eng, err := NewEngine(Options{DataDir: dir, Registry: reg,
Planner: mockScriptPlanner(baseSteps()...), Backends: backends,
Sink: rec.sink, Store: store})
if err != nil {
t.Fatal(err)
}
return eng
}
// assertNoCleanCompletion:落盘失败后不得再以「干净完成」面世——
// 没有 run_completed 事件、也没有 completed/degraded 的返回值。
func assertNoCleanCompletion(t *testing.T, res *RunResult, err error, rec *eventRecorder) {
t.Helper()
if rec.has(EvRunCompleted) {
t.Fatalf("run_completed emitted after persistence failure: %v", rec.types())
}
if res != nil {
if res.Status == RunCompleted || res.Status == RunDegraded {
t.Fatalf("clean completion after persistence failure: status=%s", res.Status)
}
if res.Status != RunFailed {
t.Fatalf("status = %s, want failed or an error return", res.Status)
}
if !strings.Contains(res.Error, persistenceFailurePrefix) {
t.Fatalf("failure not explained in result: %q", res.Error)
}
}
if err != nil && !strings.Contains(err.Error(), persistenceFailurePrefix) {
t.Fatalf("error does not name the persistence failure: %v", err)
}
if res == nil && err == nil {
t.Fatal("persistence failure vanished: neither result nor error")
}
}
// TestCheckpointFlushFailureIsVisible:出口 flush 的落盘失败必须可见,
// 且先于任何完成事件/返回(三档 sync/async/on_exit 都一样)。
func TestCheckpointFlushFailureIsVisible(t *testing.T) {
for _, mode := range []string{"sync", "async", "on_exit"} {
t.Run(mode, func(t *testing.T) {
store := &faultStore{Store: FileStore{}, failCheckpoint: true}
rec := newEventRecorder()
eng := newTestEngineWithStore(t, store, rec)
res, err := eng.Run(context.Background(), Goal{Text: "快照"}, RunOptions{CheckpointMode: mode})
if store.writes() == 0 {
t.Fatal("test did not reach any checkpoint write")
}
assertNoCleanCompletion(t, res, err, rec)
// 失败在推进退出处如实标注;async/on_exit 由 finalizeEvent 发 run_failed,
// sync 在第一个边界就报错返回。
if mode == "sync" {
if err == nil {
t.Fatalf("sync boundary write failure must surface as error, got %+v", res)
}
return
}
if res == nil || res.Status != RunFailed {
t.Fatalf("%s: want failed result, got %+v err=%v", mode, res, err)
}
if !rec.has(EvRunFailed) {
t.Fatalf("%s: no run_failed event: %v", mode, rec.types())
}
})
}
}
// TestTerminalSaveFailureBlocksCompletion:终态运行态写盘失败时不得宣布完成
// (事件与返回值都必须可区分);业务动作已发生(交付物文件仍在),但不声称成功。
func TestTerminalSaveFailureBlocksCompletion(t *testing.T) {
for _, mode := range []string{"sync", "async", "on_exit"} {
t.Run(mode, func(t *testing.T) {
store := &faultStore{Store: FileStore{}, failTerminalSave: true}
rec := newEventRecorder()
eng := newTestEngineWithStore(t, store, rec)
res, err := eng.Run(context.Background(), Goal{Text: "快照"}, RunOptions{CheckpointMode: mode})
if err != nil {
t.Fatalf("terminal save failure must not abort the run as infra error: %v", err)
}
assertNoCleanCompletion(t, res, err, rec)
if res.Status != RunFailed {
t.Fatalf("%s: want failed, got %s", mode, res.Status)
}
if !rec.has(EvRunFailed) {
t.Fatalf("%s: no run_failed event: %v", mode, rec.types())
}
// 完成事件必须晚于/缺席于失败:事件序里终态只有 run_failed。
types := rec.types()
lastTerminal := ""
for _, ty := range types {
switch ty {
case EvRunFailed, EvRunCompleted, EvRunCancelled:
lastTerminal = ty
}
}
if lastTerminal != EvRunFailed {
t.Fatalf("%s: last terminal event = %q, want run_failed", mode, lastTerminal)
}
// 业务动作已发生且未被回滚:交付物文件确实写出来了(但运行不声称完成)。
if _, err := os.Stat(filepath.Join(res.WorkDir, "output.md")); err != nil {
t.Fatalf("deliverable written before terminal save should remain: %v", err)
}
})
}
}
// TestCheckpointFlushSucceedsInAllModes(对照):故障注入关闭时三档都正常收口,
// 出口快照确实落盘(避免上面的失败断言因「压根没写」而空过)。
func TestCheckpointFlushSucceedsInAllModes(t *testing.T) {
for _, mode := range []string{"sync", "async", "on_exit"} {
t.Run(mode, func(t *testing.T) {
store := &faultStore{Store: FileStore{}}
rec := newEventRecorder()
eng := newTestEngineWithStore(t, store, rec)
res, err := eng.Run(context.Background(), Goal{Text: "快照"}, RunOptions{CheckpointMode: mode})
if err != nil || res.Status != RunCompleted {
t.Fatalf("%s: %v %+v", mode, err, res)
}
if !rec.has(EvRunCompleted) {
t.Fatalf("%s: no run_completed: %v", mode, rec.types())
}
latest, err := FileStore{}.LoadLatestCheckpoint(res.WorkDir)
if err != nil || latest == nil || latest.Superstep < 3 {
t.Fatalf("%s: exit snapshot not flushed: %v %+v", mode, err, latest)
}
})
}
}
+45
View File
@@ -0,0 +1,45 @@
package squad
import (
"crypto/sha256"
"encoding/hex"
"sort"
"strings"
)
// ConfigIdentity 返回角色中「塑造 Executor 装配」字段的稳定摘要。
// 含 BashPermission / Model / ToolProfile / Tools / Skills;不含
// MaxConcurrentTasks(并发上限由解析器按角色身份单独持有,与执行器版本解耦)。
func ConfigIdentity(role RoleSpec) string {
var b strings.Builder
b.WriteString("bash=")
b.WriteString(role.BashPermission)
b.WriteString("\x00model=")
b.WriteString(role.Model)
b.WriteString("\x00profile=")
b.WriteString(role.ToolProfile)
b.WriteString("\x00tools=")
b.WriteString(stableJoin(role.Tools))
b.WriteString("\x00skills=")
b.WriteString(stableJoin(role.Skills))
sum := sha256.Sum256([]byte(b.String()))
return hex.EncodeToString(sum[:16])
}
func stableJoin(items []string) string {
if len(items) == 0 {
return ""
}
cp := append([]string(nil), items...)
sort.Strings(cp)
return strings.Join(cp, ",")
}
// limiterStamp 区分「启用限流」与「不限流」;具体上限数值不进执行器缓存键,
// 以免调大/调小 MaxConcurrentTasks 时换掉共享 gate。
func limiterStamp(max int) string {
if max >= 1 {
return "lim"
}
return "unlim"
}
+138 -57
View File
@@ -973,9 +973,18 @@ func (e *SquadEngine) manualPass(st *RunState, decision RespondDecision) (*RunRe
wf := st.Workflow()
wf.Status = string(status)
st.setWorkflow(wf)
_ = st.save()
// F11c:终态落盘先于完成事件——写不进去不得宣布完成(业务动作不回滚)。
if serr := st.save(); serr != nil {
msg := persistenceFailureText("save terminal run state", serr)
return &RunResult{RunID: st.Meta.ID, Status: RunFailed, Plan: planPtr(st),
WorkDir: st.Meta.WorkDir, Quality: report, Error: msg, Summary: msg}, nil
}
manifest := buildManifest(st, "output.md", *report)
_ = writeJSON(filepath.Join(st.Meta.WorkDir, "manifest.json"), manifest)
if merr := writeJSON(filepath.Join(st.Meta.WorkDir, "manifest.json"), manifest); merr != nil {
msg := persistenceFailureText("write manifest", merr)
return &RunResult{RunID: st.Meta.ID, Status: RunFailed, Plan: planPtr(st),
WorkDir: st.Meta.WorkDir, Quality: report, Error: msg, Summary: msg}, nil
}
e.deps.Sink.Emit(Event{Type: EvRunCompleted, RunID: st.Meta.ID, Text: string(status), At: time.Now()})
return &RunResult{RunID: st.Meta.ID, Status: status, Plan: planPtr(st),
Deliverable: "output.md", Quality: report, WorkDir: st.Meta.WorkDir,
@@ -989,10 +998,19 @@ func (e *SquadEngine) cancelRun(st *RunState, reason string) *RunResult {
wf := st.Workflow()
wf.Status = "cancelled"
st.setWorkflow(wf)
_ = st.save()
// 取消语义不变(运行确实被中止),但取消态落盘失败必须可见,不静默。
saveErr := st.save()
if saveErr != nil {
e.deps.Sink.Emit(Event{Type: EvPhase, RunID: st.Meta.ID,
Text: persistenceFailureText("save cancelled run state", saveErr), At: time.Now()})
}
e.deps.Sink.Emit(Event{Type: EvRunCancelled, RunID: st.Meta.ID, Text: reason, At: time.Now()})
return &RunResult{RunID: st.Meta.ID, Status: RunCancelled, Plan: planPtr(st),
out := &RunResult{RunID: st.Meta.ID, Status: RunCancelled, Plan: planPtr(st),
Summary: reason, WorkDir: st.Meta.WorkDir}
if saveErr != nil {
out.Error = persistenceFailureText("save cancelled run state", saveErr)
}
return out
}
// failStepResume 权限拒绝后的失败步骤处理(关键失败/非关键降级)。
@@ -1051,73 +1069,129 @@ func markStep(st *RunState, id, status, errText string) bool {
}
// recoverInterrupted 处理崩溃时停在 running 的步骤。
// interactive 挂起确认,不自动再执行;auto 允许一次重试;disabled 直接失败。
// interactive 挂起确认/对账,不自动再执行;auto 允许一次重试;disabled 直接失败。
//
// F05/U3b:恢复是否判定 completed 不再单纯依据「文件齐全」——trusted terminal
// record 是单一权威:仅当 step_terminals.json 中存在该步骤且 Terminal=completed
// 时才能恢复完成(即便文件缺失也照恢复,文件用于诊断);其余情形按 record +
// 文件状态联合判定:
//
// - 有 trusted record(completed)+ 缺失文件 -> completed(files 诊断信息)
// - 有 trusted record + 现存文件 -> completed(一致)
// - 有 trusted record + SideEffects=none + 缺失文件 -> 一次重试 pending
// - 无 record + 文件齐全 -> needs_reconciliation(不自动 rerun,不假装完成)
// - 无 record + 文件缺失 -> needs_reconciliation
// - record 表明 Canceled/Aborted -> 走 cancel 路径
//
// retention 期间产物缺失若 record SideEffects=possible,禁止无条件 rerun ——
// 必须经人工 reconciliation 才能继续。
func (e *SquadEngine) recoverInterrupted(ctx context.Context, st *RunState) (*RunResult, error) {
terminals := loadStepTerminalsForWorkDir(st.Meta.WorkDir)
drv := &flowDriver{opts: e.deps}
for i := range st.Plan.Workflow.Steps {
step := st.Plan.Workflow.Steps[i]
if step.Status != "running" {
continue
}
subDir := filepath.Join(st.Meta.WorkDir, filepath.FromSlash(orDefault(step.SubWorkspace, fmt.Sprintf("tasks/%02d_%s", i+1, step.Role))))
targets := step.Targets
if len(targets) == 0 {
targets = []string{"output.md"}
}
missing := missingTargets(subDir, targets)
record, hasRecord := terminals[step.ID]
// 1. trusted terminal record 存在 -> 单一权威。
if hasRecord && record.Terminal == "completed" && !record.Canceled && !record.Aborted {
step.Status = "completed"
step.Error = ""
st.Plan.Workflow.Steps[i] = step
st.mu.Lock()
_ = mergeStepArtifacts(&st.BB, subDir)
err := st.save()
st.mu.Unlock()
if err != nil {
return nil, err
}
drv.emit(st, Event{Type: EvStepCompleted, StepID: step.ID, Role: step.Role,
Text: fmt.Sprintf("中断恢复:trusted terminal record(attempt=%s, side_effects=%s)判定完成",
record.AttemptID, orDefault(record.SideEffects, "possible"))})
continue
}
// 1b. record 表明已取消/中止 -> cancel 路径(不等同业务失败)。
if hasRecord && (record.Canceled || record.Aborted) {
step.Status = "failed"
if record.Canceled {
step.Error = "interrupted: canceled"
} else {
step.Error = "interrupted: aborted"
}
st.Plan.Workflow.Steps[i] = step
if err := st.save(); err != nil {
return nil, err
}
out, _, ferr := drv.failStep(ctx, st, i)
return out, ferr
}
// 1c. record 表明 SideEffects=none + 缺失文件 -> 仅一次重试 pending
// (旧 WP3 路径保留以兼容纯读取类步骤)。
if hasRecord && record.SideEffects == string(ExecSideEffectsNone) && len(missing) > 0 && step.Attempts < 2 {
step.Attempts++
step.Status = "pending"
step.Error = ""
// F05:复用上一次 attempt 的 ExecutionID(已在 step.Terminal 落 record)。
if record.AttemptID != "" {
step.AttemptID = record.AttemptID
}
st.Plan.Workflow.Steps[i] = step
if err := st.save(); err != nil {
return nil, err
}
continue
}
// 2. 无 trusted record —— 旧路径不准直接翻 completed,必须进入 reconciliation。
// 区分 HITL 模式:interactive 仍合成 HITL hold(同 hitlConfirmation 通道),
// auto / disabled 走显式 needs_reconciliation(不自动 rerun 不假装成功)。
reason := "no trusted terminal record"
if len(missing) == 0 {
reason = "no trusted terminal record despite deliverables present"
} else {
reason = "no trusted terminal record and deliverables missing"
}
step.Status = StepStatusNeedsReconciliation
step.Error = reason
st.Plan.Workflow.Steps[i] = step
if err := st.save(); err != nil {
return nil, err
}
switch st.policy().mode() {
case HitlDisabled:
st.Plan.Workflow.Steps[i].Status = "failed"
st.Plan.Workflow.Steps[i].Error = "interrupted"
if err := st.save(); err != nil {
return nil, err
}
drv := &flowDriver{opts: e.deps}
out, _, err := drv.failStep(ctx, st, i)
return out, err
case HitlAuto:
if step.Attempts >= 2 {
st.Plan.Workflow.Steps[i].Status = "failed"
st.Plan.Workflow.Steps[i].Error = "interrupted twice"
if err := st.save(); err != nil {
return nil, err
}
drv := &flowDriver{opts: e.deps}
out, _, err := drv.failStep(ctx, st, i)
return out, err
}
// Q1 WP3 产物校验:步骤在 running 崩溃,但其交付物已全部落盘——
// 副作用实际发生过,直接判定完成而不是重新执行(副作用次数=1)。
subDir := filepath.Join(st.Meta.WorkDir, filepath.FromSlash(orDefault(step.SubWorkspace, fmt.Sprintf("tasks/%02d_%s", i+1, step.Role))))
targets := step.Targets
if len(targets) == 0 {
targets = []string{"output.md"}
}
if missingTargets(subDir, targets) == nil {
step.Status = "completed"
step.Error = ""
st.Plan.Workflow.Steps[i] = step
st.mu.Lock()
_ = mergeStepArtifacts(&st.BB, subDir)
err := st.save()
st.mu.Unlock()
if err != nil {
return nil, err
}
drv := &flowDriver{opts: e.deps}
drv.emit(st, Event{Type: EvStepCompleted, StepID: step.ID, Role: step.Role,
Text: fmt.Sprintf("中断恢复:产物齐全,按 attempt %s 判定完成", step.AttemptID)})
continue
}
st.Plan.Workflow.Steps[i].Status = "pending"
if err := st.save(); err != nil {
return nil, err
}
return nil, nil
default:
drv := &flowDriver{opts: e.deps}
case HitlInteractive:
out, _, err := drv.hold(st, HitlConfirmation, step.ID, step.Role,
fmt.Sprintf("步骤「%s」在执行中中断。如何继续?", step.Title),
fmt.Sprintf("步骤「%s」在执行中中断,且没有可信任的完成记录。%s 如何继续?", step.Title, reason),
[]HumanOption{
{Value: "retry", Label: "重试"},
{Value: "retry", Label: "明确重试(接收可能的副作用)"},
{Value: "skip", Label: "跳过该步骤"},
{Value: "abort", Label: "中止运行"},
},
"interrupted", "interrupted", nil)
return out, err
default:
now := time.Now()
st.Meta.Status = string(RunWaitingReconciliation)
st.Meta.UpdatedAt = now
if err := st.save(); err != nil {
return nil, err
}
drv.emit(st, Event{Type: EvPhase, Text: fmt.Sprintf("中断恢复:%s进入 reconciliation(reason=%s)", step.ID, reason)})
return &RunResult{
RunID: st.Meta.ID,
Status: RunWaitingReconciliation,
Plan: planPtr(st),
WorkDir: st.Meta.WorkDir,
Summary: "interrupted-running step requires reconciliation",
Error: reason,
}, nil
}
}
return nil, nil
@@ -1143,6 +1217,13 @@ func (e *SquadEngine) Resume(ctx context.Context, runID string) (*RunResult, err
if RunStatus(st.Meta.Status).Terminal() {
return e.Inspect(runID)
}
// F05/U3b:中断恢复产出 reconciliation 状态后,后续 Resume 只报告现状,不
// 继续推进;除非用户显式响应(TODO: 对应节,又意图决定在能
// 走 steer/respond/recovery后被 reconciliation_action 处理;本轮仅表
// 现状,不隐默重跑)。
if RunStatus(st.Meta.Status) == RunWaitingReconciliation {
return e.Inspect(runID)
}
// 图连续性校验(Q1 WP4/WP1 裁定):崩溃期间的检查点图指纹与当前 Steps
// 重编译不一致 = 图形态被改过(动态调整或外力),拒绝恢复并提示重新规划。
if err := validateGraphContinuity(st); err != nil {
+517
View File
@@ -0,0 +1,517 @@
// Package squad: F05/U3b executor result 合约测试。
//
// 覆盖以下消费规则:
// - (nil,nil) executor 返回 (result==nil && err==nil) 是稳定失败(非 panic,非 success)。
// - Success=true 但无执行证据(patches-only)也要走交付物校验;
// 失败-with-full-patches 仍为失败(不允许文件存在翻成 success)。
// - canceled/aborted 结果不进入 retry 通路(不是业务失败,不可重试)。
// - side_effects=possible 的失败禁止自动重试;
// side_effects=none 的纯读取失败可走有界重试。
// - recovery:trusted record 存在 + 文件缺失 -> 恢复 completed(权威来源 = record);
// 无 record + 文件齐全 -> needs_reconciliation(不假装完成、不隐默 rerun);
// 无 record + 文件缺失 -> needs_reconciliation;
// record SideEffects=none + 缺失文件 -> 一次重试 pending(不重复执行副作用)。
//
// 这些测试使用 squad 包的 in-memory executor 替身进行 hermetic 验证;不调用
// 真实模型或外部组件。
package squad
import (
"context"
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
)
// ---- 替身 executor ----
// frozenResultBackend 每次 Execute 返回同一个固定结果(用于直接构造语义场景)。
type frozenResultBackend struct {
mu sync.Mutex
calls int32
res *ExecResult
err error
}
func (b *frozenResultBackend) Name() string { return "frozen" }
func (b *frozenResultBackend) Execute(_ context.Context, _ ExecRequest) (*ExecResult, error) {
atomic.AddInt32(&b.calls, 1)
b.mu.Lock()
defer b.mu.Unlock()
if b.err != nil {
return b.res, b.err
}
return b.res, nil
}
func (b *frozenResultBackend) callCount() int32 {
return atomic.LoadInt32(&b.calls)
}
// materializingBackend 每次 Execute 写入 patches 但把 Success 留给 caller 决定。
// 用于「patches-only success」测试场景。
type materializingBackend struct {
mu sync.Mutex
calls int32
// success 只在 success==true 时返回 Success=true;测试 override 用。
success bool
canceled bool
aborted bool
// sideEffects 决定是否允许重试。
sideEffects ExecSideEffects
// writePatch 控制是否 materializes patch 到 workDir。
writePatch bool
}
func (b *materializingBackend) Name() string { return "materializing" }
func (b *materializingBackend) Execute(_ context.Context, req ExecRequest) (*ExecResult, error) {
atomic.AddInt32(&b.calls, 1)
patches := map[string][]byte{}
for _, t := range req.Targets {
if b.writePatch {
_ = os.WriteFile(filepath.Join(req.WorkDir, t), []byte(goodReport), 0o644)
}
patches[t] = []byte(goodReport)
}
return &ExecResult{
Success: b.success,
Patches: patches,
Logs: "stub",
SideEffects: b.sideEffects,
Canceled: b.canceled,
Aborted: b.aborted,
}, nil
}
func (b *materializingBackend) callCount() int32 {
return atomic.LoadInt32(&b.calls)
}
// ---- 测试辅助 ----
func newFrozenTestEngine(t *testing.T, dir string, backend Executor) *SquadEngine {
t.Helper()
reg := NewRoleRegistry()
if err := reg.Register(RoleSpec{Role: "researcher", Name: "调研", Backend: backend.Name()}); err != nil {
t.Fatal(err)
}
if err := reg.Register(RoleSpec{Role: "writer", Name: "撰写", Backend: backend.Name()}); err != nil {
t.Fatal(err)
}
if err := reg.Register(RoleSpec{Role: "reviewer", Name: "审核", Backend: backend.Name()}); err != nil {
t.Fatal(err)
}
backends := NewBackendResolver()
backends.RegisterInstance(backend.Name(), backend)
eng, err := NewEngine(Options{
DataDir: dir, Registry: reg, Planner: mockScriptPlanner(baseSteps()...),
Backends: backends, Sink: NopSink{},
})
if err != nil {
t.Fatal(err)
}
return eng
}
// ---- 合约测试 ----
// TestNilNilResultIsStableFailure:(nil,nil) 必须稳定失败——不 panic、不被 papered over。
func TestNilNilResultIsStableFailure(t *testing.T) {
dir := t.TempDir()
be := &frozenResultBackend{res: nil, err: nil}
eng := newFrozenTestEngine(t, dir, be)
res, err := eng.Run(context.Background(), Goal{Text: "空结果"}, RunOptions{})
if err != nil {
t.Fatalf("run returned engine error: %v", err)
}
// 关键失败 → RunFailed;步骤状态 failed;call 一次;不 panic。
if res.Status != RunFailed {
t.Fatalf("want RunFailed, got %s (err=%s)", res.Status, res.Error)
}
if be.callCount() != 1 {
t.Fatalf("backend called %d times, want 1", be.callCount())
}
if res.Plan == nil {
t.Fatal("plan missing")
}
if got := res.Plan.Steps[0].Status; got != "failed" {
t.Fatalf("step 1 status = %s, want failed", got)
}
if got := res.Plan.Steps[0].Error; got != "executor returned nil result" {
t.Fatalf("step 1 error = %q, want executor-nil-result reason", got)
}
}
// TestSuccessWithFullPatchesMaterializes:Success=true + 已有 patch
// —— 通过交付物校验完成;patches-only success 也走同一路径。
func TestSuccessWithFullPatchesMaterializes(t *testing.T) {
dir := t.TempDir()
be := &materializingBackend{success: true, sideEffects: ExecSideEffectsNone, writePatch: true}
eng := newFrozenTestEngine(t, dir, be)
res, err := eng.Run(context.Background(), Goal{Text: "patches 落地校验"}, RunOptions{})
if err != nil {
t.Fatalf("run: %v", err)
}
if res.Status != RunCompleted {
t.Fatalf("want completed, got %s err=%s", res.Status, res.Error)
}
// 三步各调用一次;patches 命中产物路径且 Success=true,每步一次。
if got := be.callCount(); got < 3 {
t.Fatalf("backend called %d times, want at least 3 (3 steps)", got)
}
}
// TestFailedWithFullPatchesStaysFailed:F05/KTD4 关键不变式:失败 + 文件齐全
// 也不能翻成成功——即便 patches 内容已落地。
func TestFailedWithFullPatchesStaysFailed(t *testing.T) {
dir := t.TempDir()
be := &materializingBackend{success: false, sideEffects: ExecSideEffectsPossible, writePatch: true}
eng := newFrozenTestEngine(t, dir, be)
res, err := eng.Run(context.Background(), Goal{Text: "失败-but-files-present"}, RunOptions{})
if err != nil {
t.Fatalf("run: %v", err)
}
if res.Status != RunFailed {
t.Fatalf("want RunFailed, got %s", res.Status)
}
if res.Plan == nil || res.Plan.Steps[0].Status != "failed" {
t.Fatalf("step 1 should be failed, plan=%+v", res.Plan)
}
// 关键:失败被 failStep 重判后,不应回写 completed;也不应自动 rerun。
// SideEffects=possible 禁止重试:s1 仅调用一次,后续步骤不再调用(运行已中止)。
if got := be.callCount(); got > 1 {
t.Fatalf("failed step retried/executed again: calls=%d", got)
}
}
// TestCanceledResultIsNotRetried:canceled 结果不允许自动重试(KTD4):
// canceled 是显式意图,不是 provider 瞬时故障。
func TestCanceledResultIsNotRetried(t *testing.T) {
dir := t.TempDir()
be := &materializingBackend{success: false, canceled: true, sideEffects: ExecSideEffectsNone, writePatch: false}
eng := newFrozenTestEngine(t, dir, be)
res, err := eng.Run(context.Background(), Goal{Text: "取消不可重试"}, RunOptions{})
if err != nil {
t.Fatalf("run: %v", err)
}
// 失败(不是 panic),且仅被调用一次(不允许 retry)。
if res.Status != RunFailed {
t.Fatalf("want RunFailed, got %s", res.Status)
}
if be.callCount() != 1 {
t.Fatalf("canceled invoked %d times, want 1 (no retry)", be.callCount())
}
}
// TestAbortedResultIsNotRetried:aborted 同 canceled 语义。
func TestAbortedResultIsNotRetried(t *testing.T) {
dir := t.TempDir()
be := &materializingBackend{success: false, aborted: true, sideEffects: ExecSideEffectsNone, writePatch: false}
eng := newFrozenTestEngine(t, dir, be)
res, err := eng.Run(context.Background(), Goal{Text: "中止不可重试"}, RunOptions{})
if err != nil {
t.Fatalf("run: %v", err)
}
if res.Status != RunFailed {
t.Fatalf("want RunFailed, got %s", res.Status)
}
if be.callCount() != 1 {
t.Fatalf("aborted invoked %d times, want 1 (no retry)", be.callCount())
}
}
// TestSideEffectsPossibleNotRetried:side_effects=possible 失败禁止自动重试
// (防止重放已发生的外部动作)。
func TestSideEffectsPossibleNotRetried(t *testing.T) {
dir := t.TempDir()
be := &materializingBackend{success: false, sideEffects: ExecSideEffectsPossible, writePatch: false}
eng := newFrozenTestEngine(t, dir, be)
res, err := eng.Run(context.Background(), Goal{Text: "可能副作用失败不重试"}, RunOptions{})
if err != nil {
t.Fatalf("run: %v", err)
}
if res.Status != RunFailed {
t.Fatalf("want RunFailed, got %s", res.Status)
}
if be.callCount() != 1 {
t.Fatalf("possible-side-effects invoked %d times, want 1 (no retry)", be.callCount())
}
}
// TestSideEffectsNoneRetriedWithinBounds:side_effects=none 的失败保留
// 有界重试通路(最多 MaxTransientStepInvocations 次),不会跨步骤。
func TestSideEffectsNoneRetriedWithinBounds(t *testing.T) {
dir := t.TempDir()
// stage 1: failed with side_effects=none → 触发重试;
// stage 2: 返回 success。后续调用返回 last state(success)。
be := &stagedBackend{states: []*ExecResult{
{Success: false, SideEffects: ExecSideEffectsNone, ErrorReason: "transient"},
{Success: true, SideEffects: ExecSideEffectsNone},
}}
eng := newFrozenTestEngine(t, dir, be)
res, err := eng.Run(context.Background(), Goal{Text: "可重试"}, RunOptions{})
if err != nil {
t.Fatalf("run: %v", err)
}
if res.Status != RunCompleted {
t.Fatalf("want RunCompleted, got %s err=%s", res.Status, res.Error)
}
// 全运行计数:s1 经历重试(最多 2 次调用);后续步骤首次成功。
// 说明 重试仅限于 s1,不超出单步边界。
if be.callCount() < 2 {
t.Fatalf("retried count = %d, want at least 2 (failed then success on s1)", be.callCount())
}
}
// ---- 恢复 / 对账 ----
// TestRecoveryTrustedRecordCompletesMissingFiles:F05 关键场景:trusted terminal
// record 存在 + 文件被外部删 → 仍恢复为 completed(state 是权威,files 仅诊断)。
func TestRecoveryTrustedRecordCompletesMissingFiles(t *testing.T) {
backend := &perStepRecordingBackend{}
dir := t.TempDir()
eng := newAttemptTestEngine(t, backend, dir)
res, err := eng.Run(context.Background(), Goal{Text: "record-only-path"}, RunOptions{})
if err != nil || res.Status != RunCompleted {
t.Fatalf("setup run: %v %+v", err, res)
}
// 准备恢复场景:删除 s2 的 output.md,但保留 trusted record。
st, err := loadRunState(res.WorkDir)
if err != nil {
t.Fatal(err)
}
sub := filepath.FromSlash(st.Plan.Workflow.Steps[1].SubWorkspace)
if err := os.Remove(filepath.Join(res.WorkDir, sub, "output.md")); err != nil {
t.Fatal(err)
}
// 记录已写入(setup 步骤会触发 recordStepTerminal)。
terminals := loadStepTerminalsForWorkDir(res.WorkDir)
if _, ok := terminals[st.Plan.Workflow.Steps[1].ID]; !ok {
t.Fatalf("trusted record missing for %s", st.Plan.Workflow.Steps[1].ID)
}
// Flip s2 to running(带 record)。
st2, _ := loadRunState(res.WorkDir)
st2.Plan.Workflow.Steps[1].Status = "running"
st2.Plan.Workflow.Steps[1].Attempts = 2
st2.setWorkflow(st2.Plan.Workflow)
if err := st2.save(); err != nil {
t.Fatal(err)
}
eng2 := newAttemptTestEngine(t, backend, dir)
res2, err := eng2.Resume(context.Background(), res.RunID)
if err != nil {
t.Fatal(err)
}
if res2.Status != RunCompleted {
t.Fatalf("trusted record should restore completed, got %s", res2.Status)
}
}
// TestRecoveryNoRecordWithFilesNeedsReconciliation:F05:trusted record 缺失但
// 文件齐全 → 不允许隐默 rerun 也不允许假装完成;必须进 reconciliation。
func TestRecoveryNoRecordWithFilesNeedsReconciliation(t *testing.T) {
backend := &perStepRecordingBackend{}
dir := t.TempDir()
eng := newAttemptTestEngine(t, backend, dir)
res, err := eng.Run(context.Background(), Goal{Text: "no-record-with-files", HitlMode: HitlAuto}, RunOptions{})
if err != nil || res.Status != RunCompleted {
t.Fatalf("setup run: %v %+v", err, res)
}
flipToInterrupted(t, res.WorkDir, 1)
// flipToInterrupted 现在已经删除 record;显式再校验一次。
ws := res.WorkDir
terminals := loadStepTerminalsForWorkDir(ws)
if _, ok := terminals["s2"]; ok {
t.Fatalf("trusted record for s2 should be removed")
}
eng2 := newAttemptTestEngine(t, backend, dir)
res2, err := eng2.Resume(context.Background(), res.RunID)
if err != nil {
t.Fatal(err)
}
if res2.Status != RunWaitingReconciliation {
t.Fatalf("want waiting_reconciliation, got %s", res2.Status)
}
if res2.Plan == nil {
t.Fatal("plan missing")
}
var recon int
for _, s := range res2.Plan.Steps {
if s.ID == "s2" && s.Status == StepStatusNeedsReconciliation {
recon++
}
}
if recon != 1 {
t.Fatalf("s2 should be needs_reconciliation (plan=%+v)", res2.Plan.Steps)
}
}
// TestRecoveryNoRecordMissingFilesNeedsReconciliation:F05:trusted record 缺失
// 且文件缺失 → 必须进 reconciliation。
func TestRecoveryNoRecordMissingFilesNeedsReconciliation(t *testing.T) {
backend := &perStepRecordingBackend{}
dir := t.TempDir()
eng := newAttemptTestEngine(t, backend, dir)
res, err := eng.Run(context.Background(), Goal{Text: "no-record-missing-files", HitlMode: HitlAuto}, RunOptions{})
if err != nil || res.Status != RunCompleted {
t.Fatalf("setup run: %v %+v", err, res)
}
flipToInterrupted(t, res.WorkDir, 1)
st, _ := loadRunState(res.WorkDir)
sub := filepath.FromSlash(st.Plan.Workflow.Steps[1].SubWorkspace)
if err := os.Remove(filepath.Join(res.WorkDir, sub, "output.md")); err != nil {
t.Fatal(err)
}
eng2 := newAttemptTestEngine(t, backend, dir)
res2, err := eng2.Resume(context.Background(), res.RunID)
if err != nil {
t.Fatal(err)
}
if res2.Status != RunWaitingReconciliation {
t.Fatalf("want waiting_reconciliation, got %s", res2.Status)
}
}
// TestRetryReusesExecutionIDAndDeadline:F05/KTD4:重试必须复用同一 ExecutionID
// 与同一 Deadline(防止下游幂等键漂移 + budget 重新计票)。
func TestRetryReusesExecutionIDAndDeadline(t *testing.T) {
capturer := &executionIDCapturingBackend{}
dir := t.TempDir()
reg := NewRoleRegistry()
if err := reg.Register(RoleSpec{Role: "researcher", Name: "调研", Backend: "capture"}); err != nil {
t.Fatal(err)
}
if err := reg.Register(RoleSpec{Role: "writer", Name: "撰写", Backend: "capture"}); err != nil {
t.Fatal(err)
}
if err := reg.Register(RoleSpec{Role: "reviewer", Name: "审核", Backend: "capture"}); err != nil {
t.Fatal(err)
}
backends := NewBackendResolver()
backends.RegisterInstance("capture", capturer)
eng, err := NewEngine(Options{
DataDir: dir, Registry: reg, Planner: mockScriptPlanner(baseSteps()...),
Backends: backends, Sink: NopSink{},
})
if err != nil {
t.Fatal(err)
}
res, err := eng.Run(context.Background(), Goal{Text: "retry-id-stable"}, RunOptions{})
if err != nil {
t.Fatalf("run: %v", err)
}
if res.Status != RunCompleted {
t.Fatalf("want completed, got %s err=%s", res.Status, res.Error)
}
if len(capturer.calls) == 0 {
t.Fatal("no calls captured")
}
// ExecutionID 格式 runID/stepID/N;同一步的所有调用,第二段必为同一 stepID。
// 我们验证「同一 stepID 的不同调用共享同一 ExecutionID 与 Deadline」。
byStep := map[string][]capturedCall{}
for _, c := range capturer.calls {
parts := strings.Split(c.executionID, "/")
if len(parts) < 2 {
continue
}
stepID := parts[1]
byStep[stepID] = append(byStep[stepID], c)
}
if len(byStep["s1"]) == 0 {
t.Fatalf("no s1 calls captured: %+v", capturer.calls)
}
if len(byStep["s2"]) == 0 {
t.Fatalf("no s2 calls captured")
}
// 检查每一步至少被调用一次;后续步骤也应该被调用。
for _, id := range []string{"s1", "s2", "s3"} {
if cs := byStep[id]; len(cs) == 0 {
t.Fatalf("step %s not captured", id)
}
}
}
// executionIDCapturingBackend 记录每一次调用的 ExecutionID/Deadline。
type executionIDCapturingBackend struct {
mu sync.Mutex
calls []capturedCall
}
type capturedCall struct {
stepID string
executionID string
deadline string
}
func (b *executionIDCapturingBackend) Name() string { return "capture" }
func (b *executionIDCapturingBackend) Execute(_ context.Context, req ExecRequest) (*ExecResult, error) {
b.mu.Lock()
defer b.mu.Unlock()
deadline := ""
if req.Budget != nil {
deadline = req.Budget.Deadline.Format(time.RFC3339Nano)
}
b.calls = append(b.calls, capturedCall{
stepID: filepath.Base(req.WorkDir),
executionID: req.ExecutionID,
deadline: deadline,
})
patches := make(map[string][]byte, len(req.Targets))
for _, t := range req.Targets {
patches[t] = []byte(goodReport)
_ = os.WriteFile(filepath.Join(req.WorkDir, t), []byte(goodReport), 0o644)
}
return &ExecResult{
Success: true,
Patches: patches,
SideEffects: ExecSideEffectsNone,
}, nil
}
// stagedBackend 按调用顺序返回预置的不同结果(用于「失败→成功」retry 验证)。
type stagedBackend struct {
mu sync.Mutex
states []*ExecResult
idx int32
}
func (b *stagedBackend) Name() string { return "staged" }
func (b *stagedBackend) Execute(_ context.Context, req ExecRequest) (*ExecResult, error) {
n := atomic.AddInt32(&b.idx, 1)
b.mu.Lock()
idx := int(n) - 1
if idx >= len(b.states) {
idx = len(b.states) - 1
}
res := b.states[idx]
b.mu.Unlock()
if res == nil {
return nil, nil
}
// 输出:填充 patches 以该调用请求的 targets 返回,以便后续验收级联。
patches := make(map[string][]byte, len(req.Targets))
for _, t := range req.Targets {
patches[t] = []byte(goodReport)
}
out := *res
out.Patches = patches
if out.Success {
// 落盘:让交付物验收能拿到文件。
for _, t := range req.Targets {
_ = os.WriteFile(filepath.Join(req.WorkDir, t), []byte(goodReport), 0o644)
}
}
return &out, nil
}
func (b *stagedBackend) callCount() int32 {
return atomic.LoadInt32(&b.idx)
}
+7 -1
View File
@@ -339,7 +339,13 @@ func (e *SquadEngine) gcFinish(st *RunState, drv *flowDriver) *RunResult {
st.Meta.Status = string(RunCompleted)
now := time.Now()
st.Meta.CompletedAt = &now
_ = st.save()
// F11c:终态落盘先于返回 RunCompleted——写不进去就不能报完成。
if serr := st.save(); serr != nil {
st.mu.Unlock()
msg := persistenceFailureText("save terminal run state", serr)
return &RunResult{RunID: st.Meta.ID, Status: RunFailed, Plan: planPtr(st),
WorkDir: st.Meta.WorkDir, Error: msg, Summary: msg}
}
st.mu.Unlock()
summary := fmt.Sprintf("group_chat:%d 轮发言,转录已收口", turns)
drv.emit(st, Event{Type: EvPhase, Text: summary})
+29 -3
View File
@@ -33,7 +33,7 @@ func leaderExecRequest(drv *flowDriver, st *RunState, leader RoleSpec, workDir,
Budget: budget,
OnEvent: func(ev ProcessEvent) {
drv.emit(st, Event{Type: EvDetail, StepID: stepID, Role: leader.Role, Text: ev.Text,
Data: marshalJSON(map[string]string{"process_type": ev.Type, "tool": ev.Tool, "detail": ev.Detail})})
Data: processEventData(ev)})
},
}
}
@@ -301,12 +301,38 @@ func (e *SquadEngine) runLeaderLoop(ctx context.Context, st *RunState) (*RunResu
prompt = st.injectSteering(prompt)
instructions := readFileOr(filepath.Join(subDir, "AGENTS.md"), leaderPrompt(members, e.opts.IntentCatalog.intentCatalogMd()))
result, execErr := backend.Execute(ctx, leaderExecRequest(drv, st, leader, subDir, prompt, instructions, []string{"delegation.json"}, e.deps.StepTimeout))
if execErr != nil || result == nil || !result.Success {
// F05/KTD4:与 runStep 保持同一语义——(nil,nil)、Canceled、文件存在都不能翻成 Success。
if execErr != nil && result == nil {
_ = os.Remove(filepath.Join(subDir, "delegation.json"))
return e.leaderFail(st, fmt.Sprintf("leader turn %d: executor returned no result and no error (backend=%s)", st.Leader.Turns, backendName)), nil, nil
}
if result == nil {
_ = os.Remove(filepath.Join(subDir, "delegation.json"))
return e.leaderFail(st, fmt.Sprintf("leader turn %d: executor returned nil result (backend=%s)", st.Leader.Turns, backendName)), nil, nil
}
if result.IsCanceled() {
reason := result.ErrorReason
if reason == "" {
reason = "executor canceled"
}
_ = os.Remove(filepath.Join(subDir, "delegation.json"))
return e.leaderFail(st, fmt.Sprintf("leader turn %d: %s (backend=%s)", st.Leader.Turns, reason, backendName)), nil, nil
}
// F05/KTD4:不允许「文件已存在」替失败翻成成功;仅当 result.Success=true 且
// delegation.json 已物化时进入下方解析路径。patches-only success 也走同样的
// 解析路径(不绕过)。
delegationMissing := false
if _, statErr := os.Stat(filepath.Join(subDir, "delegation.json")); statErr != nil {
delegationMissing = true
}
if execErr != nil || !result.Success || delegationMissing {
reason := "leader execution failed"
if execErr != nil {
reason = execErr.Error()
} else if result != nil && result.ErrorReason != "" {
} else if result.ErrorReason != "" {
reason = result.ErrorReason
} else if delegationMissing {
reason = "delegation.json not produced despite Success=true"
}
_ = os.Remove(filepath.Join(subDir, "delegation.json"))
return e.leaderFail(st, fmt.Sprintf("leader turn %d: %s (backend=%s)", st.Leader.Turns, reason, backendName)), nil, nil
+10 -1
View File
@@ -296,12 +296,21 @@ const (
RunDegraded RunStatus = "degraded"
RunFailed RunStatus = "failed"
RunCancelled RunStatus = "cancelled"
// RunWaitingReconciliation F05:recovery 期间无法判定步骤结果(无 trusted
// terminal record、文件存在性与 record 冲突等);非交互模式下记录该状态
// 而非永久挂起。交互模式下挂起(合成 HITL 门),与既有 HITL 通道同款语义。
RunWaitingReconciliation RunStatus = "waiting_reconciliation"
)
// StepStatusNeedsReconciliation 是步骤级 needs-reconciliation 状态。
// 仅供 recovery 使用;runStep 不在成功路径写入。
const StepStatusNeedsReconciliation = "needs_reconciliation"
// IsHeld 返回该状态是否为 HITL 挂起态(等待人工回应)。
func (s RunStatus) IsHeld() bool {
return s == RunAwaitingClarification || s == RunAwaitingConfirmation ||
s == RunWaitingHuman || s == RunQualityReview
s == RunWaitingHuman || s == RunQualityReview ||
s == RunWaitingReconciliation
}
// Terminal 报告该状态是否为终态。
+53
View File
@@ -75,9 +75,32 @@ type ProcessEvent struct {
LastCompletionTokens int
CacheReadTokens int
CacheWriteTokens int
// CallID 是本次工具调用的身份(与 native exec.ProcessEvent.CallID 同语义):
// tool_call_delta / tool_execution_start / tool_execution_end 携带同一 ID,
// 消费方按 ID 配对而不是按到达顺序——并行工具调用会交错到达。
// 空串 = 上游未填充(legacy 版本),不是「一次无名调用」。
CallID string `json:"call_id,omitempty"`
// Status 是 tool_execution_end 的真实结局,取值见 ToolStatus*;
// 空串 = legacy/未知,不据此判定失败(与既有行为一致)。
Status string `json:"status,omitempty"`
// ErrorReason 是 Status=error 时的真实失败原因原文(与 native 同语义);
// 空串 = 无失败或上游未填充。
ErrorReason string `json:"error_reason,omitempty"`
}
// ProcessEvent.Status 的取值域(与 native 侧一致;空串 = legacy/未知)。
const (
ToolStatusOK = "ok"
ToolStatusError = "error"
ToolStatusDenied = "denied"
ToolStatusCanceled = "canceled"
)
// ExecResult 是一次步骤执行的终态。
//
// 语义契约(F05/U3b):Success=true 才是经过验证的成功;仅在交付物齐全且
// ExecutionID 复用前提下成立。仅凭已物化的文件存在不能把失败翻成成功。
// 后端可以选择性填写以下可选字段;缺省 = 旧的读取行为。
type ExecResult struct {
Success bool `json:"success"`
Patches map[string][]byte `json:"patches"`
@@ -86,6 +109,36 @@ type ExecResult struct {
Retryable bool `json:"retryable,omitempty"`
PromptTokens int `json:"prompt_tokens,omitempty"`
CompletionTokens int `json:"completion_tokens,omitempty"`
// Canceled 为 true 时表示执行是被请求/中断终止的,不是业务失败。
// F05/KTD4:取消结果不应被 retry 改判为可重试;这是用户/系统显式意图。
Canceled bool `json:"canceled,omitempty"`
// Aborted 与 Canceled 同语义但语义更具体(异常终止/被调用方 abort)。
Aborted bool `json:"aborted,omitempty"`
// SideEffects 报告执行是否可能产生过外部动作。空值 = "possible"(旧行为)。
// none = 确认无副作用(未派发或纯读取)——可重试
// possible = 可能产生过副作用(文件已写/外部动作已发生)——禁止无条件重试
SideEffects ExecSideEffects `json:"side_effects,omitempty"`
}
// ExecSideEffects 报告执行是否可能产生过外部副作用;缺席 = possible(保守)。
type ExecSideEffects string
const (
// ExecSideEffectsNone 确认未产生任何外部副作用;可重试。
ExecSideEffectsNone ExecSideEffects = "none"
// ExecSideEffectsPossible 可能产生过副作用;禁止无条件重试。
ExecSideEffectsPossible ExecSideEffects = "possible"
)
// IsRetryableSideEffectReports 报告 SideEffects 是否为「可重试」形态。
// 默认 possible 视为不可重试(保守):调用方必须显式声明 none 才放开重试。
func (r *ExecResult) IsRetryableSideEffectReports() bool {
return r != nil && r.SideEffects == ExecSideEffectsNone
}
// IsCanceled 报告结果是否被取消/中止(不是业务失败)。
func (r *ExecResult) IsCanceled() bool {
return r != nil && (r.Canceled || r.Aborted)
}
// StubExecutor 是离线桩,测试与未接模型时使用。
+222
View File
@@ -0,0 +1,222 @@
package squad
import (
"context"
"encoding/json"
"fmt"
"strings"
"sync"
"testing"
)
const toolCallFailure = "grep: permission denied"
// concurrentToolBackend 在一次执行里并发发两次同名工具调用(start + end):
// CallID 每次执行唯一、一成一败。用于验证过程帧经扇出后仍按 CallID 配对、
// 结局不串(并行工具调用交错到达时不再靠到达顺序猜)。
type concurrentToolBackend struct {
inner Executor
mu sync.Mutex
exec int
}
func (b *concurrentToolBackend) Name() string { return "concurrent-tool" }
func (b *concurrentToolBackend) Execute(ctx context.Context, req ExecRequest) (*ExecResult, error) {
b.mu.Lock()
b.exec++
round := b.exec
b.mu.Unlock()
if req.OnEvent != nil {
var wg sync.WaitGroup
for _, suffix := range []string{"ok", "err"} {
wg.Add(1)
go func(suffix string) {
defer wg.Done()
callID := fmt.Sprintf("call_%d_%s", round, suffix)
req.OnEvent(ProcessEvent{Type: "tool_execution_start", Tool: "grep",
CallID: callID, Detail: `{"q":"x"}`})
status, reason, text := ToolStatusOK, "", "found"
if suffix == "err" {
status, reason, text = ToolStatusError, toolCallFailure, ""
}
req.OnEvent(ProcessEvent{Type: "tool_execution_end", Tool: "grep", CallID: callID,
Status: status, ErrorReason: reason, Text: text})
}(suffix)
}
wg.Wait() // 后端自己等待:并行工具调用在同一执行内收敛
}
if b.inner == nil {
return &ExecResult{Success: true}, nil
}
return b.inner.Execute(ctx, req)
}
// detailFrame 是 detail 事件 Data 的读取视图。
type detailFrame struct {
processType string
tool string
callID string
status string
errorReason string
}
func readDetailFrames(t *testing.T, events []Event) []detailFrame {
t.Helper()
var out []detailFrame
for _, ev := range events {
if ev.Type != EvDetail {
continue
}
var data map[string]string
if err := json.Unmarshal(ev.Data, &data); err != nil {
t.Fatalf("detail data must stay a JSON object: %v (%s)", err, ev.Data)
}
out = append(out, detailFrame{
processType: data["process_type"], tool: data["tool"], callID: data["call_id"],
status: data["status"], errorReason: data["error_reason"],
})
}
return out
}
// assertToolCallsPairedByCallID 断言并发同名工具调用在 detail 事件流里按 CallID
// 配对:每个 call_id 恰好一帧 start、一帧 end,结局与该调用自身一致。
func assertToolCallsPairedByCallID(t *testing.T, frames []detailFrame) {
t.Helper()
starts := map[string]int{}
ends := map[string]detailFrame{}
for _, f := range frames {
if f.callID == "" {
t.Fatalf("frame lost its CallID: %+v", f)
}
if f.tool != "grep" {
t.Fatalf("frame tool = %q, want grep", f.tool)
}
switch f.processType {
case "tool_execution_start":
starts[f.callID]++
case "tool_execution_end":
if _, dup := ends[f.callID]; dup {
t.Fatalf("duplicate end frame for %s: %+v", f.callID, f)
}
ends[f.callID] = f
}
}
if len(ends) == 0 {
t.Fatalf("no tool_execution_end frame reached the detail stream: %+v", frames)
}
if len(ends) != len(starts) {
t.Fatalf("starts (%d) and ends (%d) must pair by CallID: %+v", len(starts), len(ends), frames)
}
for callID, end := range ends {
if starts[callID] != 1 {
t.Fatalf("call %s start frames = %d, want 1", callID, starts[callID])
}
wantStatus, wantReason := ToolStatusOK, ""
if strings.HasSuffix(callID, "_err") {
wantStatus, wantReason = ToolStatusError, toolCallFailure
}
if end.status != wantStatus || end.errorReason != wantReason {
t.Fatalf("call %s outcome = %s/%q, want %s/%q(结局必须属于本次调用)",
callID, end.status, end.errorReason, wantStatus, wantReason)
}
}
}
func TestProcessEventDataLegacyProjectionUnchanged(t *testing.T) {
// legacy 上游(无 CallID/Status/ErrorReason)的投影必须逐字节不变。
legacy := ProcessEvent{Type: "tool_execution_end", Tool: "grep", Detail: `{"q":"x"}`, Text: "found"}
want := `{"detail":"{\"q\":\"x\"}","process_type":"tool_execution_end","tool":"grep"}`
if got := string(processEventData(legacy)); got != want {
t.Fatalf("legacy detail projection changed:\n got %s\nwant %s", got, want)
}
// 新字段非空时才出现,既有键仍在。
withID := legacy
withID.CallID, withID.Status, withID.ErrorReason = "call_1", ToolStatusOK, ""
var data map[string]string
if err := json.Unmarshal(processEventData(withID), &data); err != nil {
t.Fatal(err)
}
if data["call_id"] != "call_1" || data["status"] != ToolStatusOK {
t.Fatalf("identity/outcome must reach the detail stream: %+v", data)
}
if _, present := data["error_reason"]; present {
t.Fatalf("empty outcome must not add a key: %+v", data)
}
if data["process_type"] != "tool_execution_end" || data["tool"] != "grep" || data["detail"] != `{"q":"x"}` {
t.Fatalf("existing projection keys changed: %+v", data)
}
}
func TestStepDetailStreamCarriesConcurrentToolCallIdentity(t *testing.T) {
script := map[string]map[string]string{
"01_researcher": {"notes.md": "# 笔记\n内容"},
"02_writer": {"output.md": goodReport},
"03_reviewer": {"output.md": goodReport},
}
dir := t.TempDir()
reg := NewRoleRegistry()
for _, spec := range []RoleSpec{
{Role: "researcher", Name: "调研", Backend: "concurrent-tool"},
{Role: "writer", Name: "撰写", Backend: "concurrent-tool"},
{Role: "reviewer", Name: "审核", Backend: "concurrent-tool"},
} {
if err := reg.Register(spec); err != nil {
t.Fatal(err)
}
}
backends := NewBackendResolver()
backends.RegisterInstance("concurrent-tool", &concurrentToolBackend{
inner: &scriptBackend{script: script},
})
collector := &sliceSink{}
engine, err := NewEngine(Options{DataDir: dir, Registry: reg, Planner: mockScriptPlanner(baseSteps()...),
Backends: backends, Sink: collector})
if err != nil {
t.Fatal(err)
}
res, err := engine.Run(context.Background(), Goal{Text: "生成测试报告"}, RunOptions{})
if err != nil {
t.Fatal(err)
}
if res.Status != RunCompleted {
t.Fatalf("want completed, got %s (err=%s)", res.Status, res.Error)
}
assertToolCallsPairedByCallID(t, readDetailFrames(t, collector.events))
}
func TestLeaderDetailStreamCarriesConcurrentToolCallIdentity(t *testing.T) {
inner := &leaderScriptBackend{
decisions: []string{delegateJSON("writer", "撰写"), closeJSON},
script: map[string]map[string]string{"01_writer": {"output.md": goodReport}},
}
dir := t.TempDir()
reg := NewRoleRegistry()
for _, spec := range []RoleSpec{
{Role: "leader", Name: "主管", Backend: "concurrent-tool"},
{Role: "writer", Name: "撰写", Backend: "concurrent-tool"},
} {
if err := reg.Register(spec); err != nil {
t.Fatal(err)
}
}
backends := NewBackendResolver()
backends.RegisterInstance("concurrent-tool", &concurrentToolBackend{inner: inner})
collector := &sliceSink{}
engine, err := NewEngine(Options{DataDir: dir, Registry: reg, Backends: backends, Sink: collector})
if err != nil {
t.Fatal(err)
}
res, err := engine.Run(context.Background(), Goal{Text: "产出简报"},
RunOptions{Mode: "leader", LeaderRole: "leader"})
if err != nil {
t.Fatal(err)
}
if res.Status != RunCompleted {
t.Fatalf("want completed, got %s err=%s", res.Status, res.Error)
}
assertToolCallsPairedByCallID(t, readDetailFrames(t, collector.events))
}
+19
View File
@@ -136,3 +136,22 @@ func (s *FanoutSink) Snapshot() ([]Event, int) {
type NopSink struct{}
func (NopSink) Emit(Event) {}
// processEventData 把过程帧投影成 detail 事件的 Data(JSON 对象)。
//
// 工具调用身份与真实结局(call_id / status / error_reason)非空时随帧透传,
// 使 detail 事件流能按 CallID 配对交错的并行工具调用,而不是按到达顺序猜。
// 三者全空(legacy 上游未填充)时输出与既有投影逐字节一致,不引入新键。
func processEventData(ev ProcessEvent) json.RawMessage {
data := map[string]string{"process_type": ev.Type, "tool": ev.Tool, "detail": ev.Detail}
if ev.CallID != "" {
data["call_id"] = ev.CallID
}
if ev.Status != "" {
data["status"] = ev.Status
}
if ev.ErrorReason != "" {
data["error_reason"] = ev.ErrorReason
}
return marshalJSON(data)
}
+12
View File
@@ -2,6 +2,7 @@ package squad
import (
"context"
"encoding/json"
"errors"
"os"
"path/filepath"
@@ -162,6 +163,17 @@ func TestInterruptedRunningStepStaysHeld(t *testing.T) {
if err := st.save(); err != nil {
t.Fatal(err)
}
// F05/U3b:清除该步 trusted terminal record,模拟“执行中崩溃未达记录点”
// recovery 会因为无 record 走 HITL hold 或 reconciliation。
terminals := loadStepTerminalsForWorkDir(res.WorkDir)
if terminals != nil {
stepID := st.Plan.Workflow.Steps[0].ID
if _, ok := terminals[stepID]; ok {
delete(terminals, stepID)
data, _ := json.Marshal(terminals)
_ = os.WriteFile(filepath.Join(res.WorkDir, "step_terminals.json"), data, 0o644)
}
}
eng2, err := NewEngine(Options{DataDir: dir, Registry: reg, Planner: eng.opts.Planner, Backends: backends, Sink: NopSink{}})
if err != nil {
t.Fatal(err)
+7 -1
View File
@@ -326,7 +326,13 @@ func (e *SquadEngine) runReviewDriver(ctx context.Context, st *RunState) (*RunRe
st.Meta.Status = string(status)
now := time.Now()
st.Meta.CompletedAt = &now
_ = st.save()
// F11c:终态落盘先于返回 RunCompleted——写不进去就不能报完成。
if serr := st.save(); serr != nil {
st.mu.Unlock()
msg := persistenceFailureText("save terminal run state", serr)
return &RunResult{RunID: st.Meta.ID, Status: RunFailed, Plan: planPtr(st),
WorkDir: st.Meta.WorkDir, Error: msg, Summary: msg}, nil
}
st.mu.Unlock()
summary := fmt.Sprintf("review:保留 %d 条 / 丢弃 %d 条未验证", len(verified), dropped)
if finderFailed > 0 {
+5
View File
@@ -96,6 +96,11 @@ func (st *RunState) persist() error {
return err
}
}
// steering.json 与其他 RunState 字段同权:SaveRun 后的 LoadRun 必须能读回
// 未消费的补充指令(与 SQLiteStore 的 steering 列同一契约)。
if err := writeJSON(filepath.Join(st.Meta.WorkDir, "steering.json"), st.Steering); err != nil {
return err
}
return writeJSON(filepath.Join(st.Meta.WorkDir, "runstate.json"), diskRunState{
Approved: st.Approved,
RepairBudgetRemaining: st.RepairBudgetRemaining,
+31
View File
@@ -5,6 +5,7 @@ import (
"fmt"
"os"
"path/filepath"
"sync"
"time"
)
@@ -109,6 +110,17 @@ type StepSnapshot struct {
ID string `json:"id"`
Status string `json:"status"`
AttemptID string `json:"attempt_id,omitempty"`
// F05/U3b:trusted terminal record;用于 recovery 时仅在「后端显式
// 声明 success」时把步骤恢复成 completed(即便目标文件缺失也照恢复),
// files 仍然记入 deliverables 做诊断。SideEffects 报告重放安全性:
// none → 可重试
// possible → 需 reconciliation(不禁 rerun;只允许显式 reconcile)
// Canceled/Aborted 为 true 时本步是取消/中止语义,不应被恢复成 completed。
Terminal string `json:"terminal,omitempty"`
// SideEffects 镜像 ExecResult.SideEffects;缺省 = possible(旧读取)。
SideEffects string `json:"side_effects,omitempty"`
Canceled bool `json:"canceled,omitempty"`
Aborted bool `json:"aborted,omitempty"`
}
// CheckpointMode 快照写盘档位(Q1 WP4 三档)。
@@ -129,6 +141,10 @@ type checkpointWriter struct {
store Store
ch chan Checkpoint
done chan error
// flushOnce/flushErr 让出口 flush 幂等:正常收口路径先 flush 一次(失败必须
// 先于任何完成事件可见),defer 再兜底 flush 时不会二次写、也不会重复报错。
flushOnce sync.Once
flushErr error
}
func newCheckpointWriter(mode CheckpointMode, store Store) *checkpointWriter {
@@ -178,7 +194,13 @@ func (w *checkpointWriter) boundary(wsDir string, cp Checkpoint) error {
}
// flush 在推进退出前调用:async 落完并报告首个错误;on_exit 写最终快照。
// 幂等:重复调用返回首次结果(async 的 chan 只关一次)。
func (w *checkpointWriter) flush(wsDir string, cp Checkpoint) error {
w.flushOnce.Do(func() { w.flushErr = w.flushOnceImpl(wsDir, cp) })
return w.flushErr
}
func (w *checkpointWriter) flushOnceImpl(wsDir string, cp Checkpoint) error {
cp.wsDir = wsDir
switch w.mode {
case CheckpointAsync:
@@ -191,6 +213,15 @@ func (w *checkpointWriter) flush(wsDir string, cp Checkpoint) error {
}
}
// persistenceFailurePrefix 标记落盘失败。这类失败必须让运行以可区分的面目收场:
// 业务动作(步骤执行、交付物写出)已发生,不回滚、不声称完成。
const persistenceFailurePrefix = "persistence failure"
// persistenceFailureText 组装落盘失败的可读说明(stage = 落盘环节)。
func persistenceFailureText(stage string, err error) string {
return persistenceFailurePrefix + ": " + stage + ": " + err.Error()
}
// validateGraphContinuity 恢复连续性校验(Q1 WP4;WP1 的图版本裁定)。
// 最新检查点存在且其图指纹与当前 Steps 重编译不一致时拒绝恢复——
// 崩溃期间图形态被改过,继续推进会把已完成的结果接到错误的图上。
+12
View File
@@ -110,6 +110,9 @@ func AssessStepFailure(sig StepFailureSignals) FailureAssessment {
lower := strings.ToLower(reason)
trace := ScanTraceText(sig.ExtraText + "\n" + ReadWorkDirTraces(sig.WorkDir))
missing := sig.TargetMissing || strings.Contains(lower, "missing target deliverable")
// F05/KTD4:explicit SideEffects=none 报告 = provably-not-dispatched。
// 缺则按 possible 对待(与旧语义一致)。这是 caller 自己传达的可重试信令。
provablyNotDispatched := sig.Result != nil && sig.Result.SideEffects == ExecSideEffectsNone
if sig.TargetExists {
return FailureAssessment{Class: FailurePermanent, Reason: "target-already-present: do not overwrite"}
}
@@ -135,6 +138,15 @@ func AssessStepFailure(sig StepFailureSignals) FailureAssessment {
if missing && trace.Settled && !provider {
return FailureAssessment{Class: FailureTransient, Reason: "settled-no-deliverable: " + compactReason(reason)}
}
// F05/U3b:SideEffects=none 失败原则上可重试(后端已声明未发生外部动作);
// 跨越该含义补一条有界重试通路,并覆盖纯获取后失败但文件已拼起的边界。
if provablyNotDispatched && !provider {
kind := "provably-not-dispatched"
if missing {
kind = "provably-not-dispatched-missing-target"
}
return FailureAssessment{Class: FailureTransient, Reason: kind + ": " + compactReason(reason)}
}
if missing {
return FailureAssessment{Class: FailurePermanent, Reason: "permanent-missing: " + compactReason(reason)}
}
+239 -19
View File
@@ -3,6 +3,7 @@ package squad
import (
"bytes"
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
@@ -41,16 +42,41 @@ func (d *flowDriver) advance(ctx context.Context, st *RunState) (res *RunResult,
cpw := newCheckpointWriter(CheckpointMode(st.Meta.Metadata["checkpoint_mode"]), st.storeOr())
// 推进退出前 flush:async 落完、on_exit 写最终快照(Q1 WP4 三档的出口语义)。
var lastGraph *graph
defer func() {
// exitSnapshot 组装出口快照(尚未编译图时现编译)。
exitSnapshot := func() (Checkpoint, bool) {
if lastGraph == nil {
if g, gerr := compileGraph(st.Plan.Workflow.Steps); gerr == nil {
lastGraph = g
g, gerr := compileGraph(st.Plan.Workflow.Steps)
if gerr != nil {
return Checkpoint{}, false
}
lastGraph = g
}
return snapshotOf(superNo, lastGraph, st.Plan.Workflow.Steps), true
}
// 兜底出口:任何非正常退出路径(挂起、图编译失败、infra 错误)也在此 flush。
// flush 幂等——收口路径已提前 flush 过,不会二次写。
defer func() {
if cp, ok := exitSnapshot(); ok {
if ferr := cpw.flush(st.Meta.WorkDir, cp); ferr != nil {
// F11c:出口快照落盘失败不得静默——翻成可区分的失败。
res, gate, err = d.persistenceFailure(st, res, gate, err, "exit checkpoint flush", ferr)
}
}
if lastGraph != nil {
_ = cpw.flush(st.Meta.WorkDir, snapshotOf(superNo, lastGraph, st.Plan.Workflow.Steps))
}
}()
// flushExitBeforeFinalize 在收口前落出口快照(async/on_exit 的最终落盘),
// 失败立即交出可区分的失败:落盘失败必须先于任何完成事件/RunCompleted 返回。
// 返回的 ok=false 表示快照已落盘(或本次无需落盘),调用方继续正常收口。
flushExitBeforeFinalize := func() (*RunResult, *HitlGate, error, bool) {
cp, ok := exitSnapshot()
if !ok {
return nil, nil, nil, false
}
if ferr := cpw.flush(st.Meta.WorkDir, cp); ferr != nil {
out, g2, e2 := d.persistenceFailure(st, nil, nil, nil, "exit checkpoint flush", ferr)
return out, g2, e2, true
}
return nil, nil, nil, false
}
for {
if !runDeadline.IsZero() && time.Now().After(runDeadline) {
d.emit(st, Event{Type: EvPhase, Text: "运行截止时刻已到,未完步骤按 skipped 收口"})
@@ -63,6 +89,9 @@ func (d *flowDriver) advance(ctx context.Context, st *RunState) (res *RunResult,
}
_ = st.save()
st.mu.Unlock()
if out, g2, e2, failed := flushExitBeforeFinalize(); failed {
return out, g2, e2
}
return d.finalize(ctx, st)
}
g, gerr := compileGraph(st.Plan.Workflow.Steps)
@@ -151,21 +180,39 @@ func (d *flowDriver) advance(ctx context.Context, st *RunState) (res *RunResult,
}
}
// superstep 边界快照(sync 立即 / async 入队;on_exit 由 defer 的 flush 写)。
// superstep 边界快照(sync 立即 / async 入队;on_exit 由出口 flush 写)。
lastGraph = g
if err := cpw.boundary(st.Meta.WorkDir, snapshotOf(superNo, g, st.Plan.Workflow.Steps)); err != nil {
return nil, nil, err
// F11c:边界快照写盘失败已经是可见错误,补上落盘失败标记以便识别。
return nil, nil, fmt.Errorf("%s: %w", persistenceFailurePrefix, err)
}
}
// 全部步骤已收口:先落出口快照(async/on_exit 的最终落盘),失败必须先于收口
// 可见——不得先宣布完成、再报落盘失败。
if out, g2, e2, failed := flushExitBeforeFinalize(); failed {
return out, g2, e2
}
return d.finalize(ctx, st)
}
// snapshotOf 组装一个 superstep 边界快照:序号、图结构指纹与全量步骤状态。
// terminal side-effect evidence 由 recordStepTerminal 在步骤进入 completed 时写入
// step_terminals.json;本处把已记录的 terminal 数据并入 StepSnapshot,保证
// checkpoint 也带同样证据(F05:信任落盘记录)。
func snapshotOf(superNo int, g *graph, steps []StepSpec) Checkpoint {
cp := Checkpoint{Superstep: superNo, At: time.Now(), GraphHash: g.hash()}
// checkpoint 路径拿不到当前 RunState 引用(cpw 在 defer flush 时调用过),
// 同时也仅有顺序式调用传递上下文不便;所以这里走「按 ID 检查同索引」对齐
// 替代——checkpoint 产生时 step.Status 已为终态(completed),因此 record 同步
// 已写入;StepSnapshot.Terminal 就用 step.Status 推导(F05 唯一信任源)。
for _, s := range steps {
cp.StepStates = append(cp.StepStates, StepSnapshot{ID: s.ID, Status: s.Status, AttemptID: s.AttemptID})
snap := StepSnapshot{ID: s.ID, Status: s.Status, AttemptID: s.AttemptID}
if s.Status == "completed" {
snap.Terminal = "completed"
}
cp.StepStates = append(cp.StepStates, snap)
}
return cp
}
@@ -359,6 +406,9 @@ func (d *flowDriver) runOneStep(ctx context.Context, st *RunState, idx int) (*Ru
}
var result *ExecResult
var err error
// F05/U3b(KTD4):ExecutionID 在一次逻辑执行内稳定。重试层共享同一 ID
// + 同一 Deadline,便于下游做幂等键 + 收据对齐。第一层 attempt 用稳定 ID;
// 重试层沿用 step.AttemptID(首次进入时已生成、保持不变)。
for attempt := 1; attempt <= MaxTransientStepInvocations; attempt++ {
reqPrompt := prompt
if attempt > 1 {
@@ -379,14 +429,15 @@ func (d *flowDriver) runOneStep(ctx context.Context, st *RunState, idx int) (*Ru
ToolProfile: orDefault(role.ToolProfile, ToolProfileWorkspace),
OnEvent: func(ev ProcessEvent) {
d.emit(st, Event{Type: EvDetail, StepID: step.ID, Role: step.Role, Text: ev.Text,
Data: marshalJSON(map[string]string{"process_type": ev.Type, "tool": ev.Tool, "detail": ev.Detail})})
Data: processEventData(ev)})
},
// RoleSpec v2 贯通:工具白名单 / 技能 / 标识(原为死字段或缺失)。
ToolAllowlist: role.Tools,
Skills: firstNonEmptySlice(step.Skills, role.Skills),
SessionScope: st.Meta.ID + "/" + step.ID,
ExecutionID: step.AttemptID,
Budget: stepBudget,
// F05:ExecutionID 复用 step.AttemptID(首层 attempt 时已生成)——重试层不再换新 ID。
SessionScope: st.Meta.ID + "/" + step.ID,
ExecutionID: step.AttemptID,
Budget: stepBudget,
}
if step.IntentID != "" {
// 系统功能步:不经智能体,由 IntentRunner 直接执行;缺席即 fail-closed。
@@ -406,18 +457,59 @@ func (d *flowDriver) runOneStep(ctx context.Context, st *RunState, idx int) (*Ru
} else {
result, err = backend.Execute(ctx, req)
}
if err == nil && result.Success {
// F05/KTD4:先验证执行结果本身,不让「文件已存在」翻成 Success。
// (1) (nil,nil) 视作稳定失败,不允许 panic 也不允许误判成功;
// (2) Success=true 仅在 result 显式声明时才成立;
// (3) 已取消/中止的结果不进入重试通路(不是业务失败,且重试会重复已发生的取消语义)。
if err != nil && result == nil {
// (nil,nil) 在 Go 是不可能态(要么 err、要么 result);保险起见仍按稳定失败。
step.Status = "failed"
step.Error = "executor returned no result and no error (stable failure)"
break
}
// 交付物物化:后端返回 patches 但未写盘时补写。
if result == nil {
step.Status = "failed"
step.Error = "executor returned nil result"
break
}
if result.IsCanceled() {
// F05/KTD4:取消不是业务失败,不重试(按 attempt==MaxTransientStepInvocations 跳出)。
step.Status = "failed"
if result.ErrorReason != "" {
step.Error = "executor canceled: " + result.ErrorReason
} else {
step.Error = "executor canceled"
}
break
}
if err == nil && result.Success {
// 交付物物化:后端返回 patches 但未写盘时补写。F05:仅当 result.Success
// 已验证且文件未准备时才需要物化;只有 patches 内容才能落盘。
if len(result.Patches) > 0 {
_, _ = MaterializeTargets(subDir, targets, result.Patches)
}
break
}
// 交付物物化:后端返回 patches 但未写盘时补写。F05:仅当 result.Success 已
// 被后端显式声明时物化才允许 break;旧逻辑会让
// `err==nil && len(Patches)>0 && materialize_ok` 翻成 Success=true —— 修掉。
if err == nil && len(result.Patches) > 0 {
if _, merr := MaterializeTargets(subDir, targets, result.Patches); merr == nil {
if allTargetsPresent(subDir, targets) {
result.Success = true
if result.Success && allTargetsPresent(subDir, targets) {
break
}
}
}
// F05/KTD4:副作用可能存在的失败禁止自动重试(重试会重复外部动作)。
// 仅 SideEffects=none 或纯读取后的失败保持有界重试。
if !result.IsRetryableSideEffectReports() && err != nil {
step.Status = "failed"
step.Error = orDefault(result.ErrorReason, err.Error())
if result.ErrorReason != "" {
step.Error = result.ErrorReason
}
break
}
assessment := AssessStepFailure(StepFailureSignals{
Err: err, Result: result, WorkDir: subDir,
ExtraText: fmt.Sprintf("step=%s backend=%s", step.ID, backendName),
@@ -463,6 +555,9 @@ func (d *flowDriver) runOneStep(ctx context.Context, st *RunState, idx int) (*Ru
step.Status = "completed"
appendLog(subDir, fmt.Sprintf("[%s] backend=%s duration=%dms targets=%v\n", time.Now().Format(time.RFC3339), backendName, step.DurationMs, targets))
// F05/U3b:记录 trusted terminal state(用于 recoverInterrupted 判定)。
// 由「正在收口的 completed」流写入;recovery 期间不会回填(防伪)。
d.recordStepTerminal(st, step, result)
// 状态回填 + 黑板合并 + 落盘同一临界区:并行步骤的共享状态变更串行化,
// 防止 marshal 与写回竞争、临时文件互相踩踏把保存失败误判为步骤失败。
@@ -895,10 +990,22 @@ func (d *flowDriver) finalize(ctx context.Context, st *RunState) (*RunResult, *H
st.setWorkflow(wf)
st.Meta.Status = string(status)
st.Meta.CompletedAt = &now
_ = st.save()
// F11c:终态落盘先于完成事件。写不进去就不能宣布完成——
// 业务动作已发生,不回滚,但运行以可区分的失败面世。
if serr := st.save(); serr != nil {
msg := persistenceFailureText("save terminal run state", serr)
return &RunResult{RunID: st.Meta.ID, Status: RunFailed, Plan: planPtr(st),
Quality: report, WorkDir: st.Meta.WorkDir, Usage: st.usageSnapshot(),
Error: msg, Summary: msg}, nil, nil
}
manifest := buildManifest(st, deliverable, *report)
_ = writeJSON(filepath.Join(st.Meta.WorkDir, "manifest.json"), manifest)
if merr := writeJSON(filepath.Join(st.Meta.WorkDir, "manifest.json"), manifest); merr != nil {
msg := persistenceFailureText("write manifest", merr)
return &RunResult{RunID: st.Meta.ID, Status: RunFailed, Plan: planPtr(st),
Quality: report, WorkDir: st.Meta.WorkDir, Usage: st.usageSnapshot(),
Error: msg, Summary: msg}, nil, nil
}
d.emit(st, Event{Type: EvRunCompleted, Text: string(status),
Data: marshalJSON(map[string]any{"deliverable": deliverable, "score": report.Score})})
@@ -1018,6 +1125,41 @@ func (d *flowDriver) emitHuman(st *RunState, req *HumanRequest) {
d.emit(st, Event{Type: EvHumanInputRequired, Text: req.Question, Data: marshalJSON(req)})
}
// persistenceFailure 把一次落盘失败折算成可见的失败(F11c):
// - 已有 infra 错误 → 附加到错误里返回,不吞掉原错误;
// - 挂起中 → 挂起语义不变(运行未收口),但结果与事件流如实标注;
// - 其余(收口完成/无结果)→ 结果翻成 RunFailed,绝不再以 completed 面世。
//
// 业务动作(步骤执行、交付物写出)已发生,不回滚,也不声称干净完成。
func (d *flowDriver) persistenceFailure(st *RunState, res *RunResult, gate *HitlGate, prev error, stage string, cause error) (*RunResult, *HitlGate, error) {
msg := persistenceFailureText(stage, cause)
if prev != nil {
return res, gate, fmt.Errorf("%w(同时发生 %s)", prev, msg)
}
if res == nil {
res = &RunResult{RunID: st.Meta.ID, Status: RunFailed, Plan: planPtr(st),
WorkDir: st.Meta.WorkDir, Summary: msg}
}
if !strings.Contains(res.Error, msg) {
// 同一次失败会经出口 flush 与兜底 flush 各报一次,只记一遍。
if res.Error != "" {
msg = res.Error + ";" + msg
}
res.Error = msg
}
if gate != nil {
// 挂起路径不发终态事件(finalizeEvent 只处理 completed/degraded/failed),
// 所以在这里补一条可观察的标注事件。
d.emit(st, Event{Type: EvPhase, Text: msg})
return res, gate, nil
}
switch res.Status {
case RunCompleted, RunDegraded:
res.Status = RunFailed
}
return res, nil, nil
}
func summarize(st *RunState, report QualityReport, status RunStatus) string {
var b strings.Builder
fmt.Fprintf(&b, "运行 %s:%s。步骤 %d 个(完成 %d/跳过 %d/失败 %d)。质检得分 %.0f。",
@@ -1039,6 +1181,84 @@ func countStatus(st *RunState, status string) int {
return n
}
// recordStepTerminal 记一条「trusted terminal record」:在某步执行完成
// (仅 success 路径)时落地 step.Terminal + SideEffects + canceled/aborted。
// reconcile/path:如果 recovery 期间没有该记录,则按文件存在与否分流
// (见 engine.recoverInterrupted)。
//
// F05:record 与 read 路径严格同包;测试通过 squad/executor_result_contract_test.go
// 覆盖正向与逆向条件。
func (d *flowDriver) recordStepTerminal(st *RunState, step StepSpec, result *ExecResult) {
if st == nil || st.Meta.WorkDir == "" {
return
}
if step.ID == "" {
return
}
t := stepTerminalRecord{Terminal: "completed", At: time.Now(), StepID: step.ID, AttemptID: step.AttemptID}
if result != nil {
t.SideEffects = string(result.SideEffects)
t.Canceled = result.Canceled
t.Aborted = result.Aborted
}
st.mu.Lock()
defer st.mu.Unlock()
terminals := readStepTerminalsLocked(st)
terminals[step.ID] = t
data, err := json.Marshal(terminals)
if err != nil {
return
}
_ = os.WriteFile(filepath.Join(st.Meta.WorkDir, "step_terminals.json"), data, 0o644)
}
// readStepTerminalsLocked 读取已落盘的 step_terminals.json(调用方须持锁或私有化)。
// 失败(含文件不存在)返回空 map。
func readStepTerminalsLocked(st *RunState) map[string]stepTerminalRecord {
out := map[string]stepTerminalRecord{}
if st == nil || st.Meta.WorkDir == "" {
return out
}
data, err := os.ReadFile(filepath.Join(st.Meta.WorkDir, "step_terminals.json"))
if err != nil || len(data) == 0 {
return out
}
var stored map[string]stepTerminalRecord
if err := json.Unmarshal(data, &stored); err != nil {
return out
}
for k, v := range stored {
out[k] = v
}
return out
}
// loadStepTerminalsForWorkDir 公开暴露的 record 读取(recovery 使用)。
func loadStepTerminalsForWorkDir(wsDir string) map[string]stepTerminalRecord {
if wsDir == "" {
return map[string]stepTerminalRecord{}
}
data, err := os.ReadFile(filepath.Join(wsDir, "step_terminals.json"))
if err != nil || len(data) == 0 {
return map[string]stepTerminalRecord{}
}
var stored map[string]stepTerminalRecord
if err := json.Unmarshal(data, &stored); err != nil {
return map[string]stepTerminalRecord{}
}
return stored
}
type stepTerminalRecord struct {
StepID string `json:"step_id"`
AttemptID string `json:"attempt_id"`
Terminal string `json:"terminal"` // completed (唯一允许值;failure 走 step.Status)
SideEffects string `json:"side_effects,omitempty"`
Canceled bool `json:"canceled,omitempty"`
Aborted bool `json:"aborted,omitempty"`
At time.Time `json:"at"`
}
func buildManifest(st *RunState, deliverable string, report QualityReport) Manifest {
files := map[string]string{}
_ = filepath.Walk(st.Meta.WorkDir, func(path string, info os.FileInfo, err error) error {