Skip to content

Commit c6111e3

Browse files
author
SqlRush
committed
Replay clipboard image paste events
1 parent b8b9cd8 commit c6111e3

5 files changed

Lines changed: 133 additions & 1 deletion

File tree

‎docs/cc-100-roadmap.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -810,6 +810,8 @@ M7 补充：prompt history `LogEntry` 读取现在接受 `sessionID`/`session`/`
810810

811811
本轮补充：interaction script paste payload 现在接受 DOM `clipboardData`/`dataTransfer` 对象，可从 `text/plain`、`plainText` 和 `items[].text` 恢复 pasted text，减少 ClipboardEvent 录制 fixture 的手工改写。
812812

813+
本轮补充：interaction script paste payload 现在也可从 DOM `clipboardData.items` 和 `dataTransfer.files` 中恢复 `image/*` file item，映射为已有 image paste，并避免把 image file 的 `data`/`base64` 内容误当成普通 pasted text。
814+
813815
本轮补充：interaction script resize payload 现在接受 DOM/window 尺寸别名 `innerWidth`/`innerHeight`、`clientWidth`/`clientHeight`、`offsetWidth`/`offsetHeight` 和 ResizeObserver 风格 `contentRect`/`target` wrapper。
814816

815817
本轮补充：interaction script resize payload 现在接受 ResizeObserver `contentBoxSize`/`borderBoxSize` 数组里的 `inlineSize`/`blockSize` 字段，覆盖现代浏览器 box-size 事件形态。

‎docs/claude-code-go-rewrite-plan.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -265,6 +265,7 @@ test/parity/ # golden tests against TS/official behavior
265265
- 本轮补充：interaction script touch payload 现在可从 `touches`/`targetTouches`/`changedTouches` 的首个 touch point 恢复坐标，`touchmove` 映射为 SGR drag motion，`touchcancel` 映射为 release，减少 DOM touch 录制 fixture 的手工改写。
266266
- 本轮补充：REPL dialog 鼠标处理现在忽略 SGR motion/drag button，只响应实际 press/click，避免 pointer/touch move 回放关闭 permission/task dialog。
267267
- 本轮补充：interaction script paste payload 现在接受 DOM `clipboardData`/`dataTransfer` 对象，可从 `text/plain`、`plainText` 和 `items[].text` 恢复 pasted text，减少 ClipboardEvent 录制 fixture 的手工改写。
268+
- 本轮补充：interaction script paste payload 现在也可从 DOM `clipboardData.items` 与 `dataTransfer.files` 中恢复 `image/*` file item 为 image paste，并避免把 image file 的 `data`/`base64` 内容误当成普通 pasted text。
268269
- 本轮补充：interaction script resize payload 现在接受 DOM/window 尺寸别名 `innerWidth`/`innerHeight`、`clientWidth`/`clientHeight`、`offsetWidth`/`offsetHeight` 和 ResizeObserver 风格 `contentRect`/`target` wrapper。
269270
- 本轮补充：interaction script resize payload 现在接受 ResizeObserver `contentBoxSize`/`borderBoxSize` 数组里的 `inlineSize`/`blockSize` 字段，覆盖现代浏览器 box-size 事件形态。
270271
- 本轮补充：嵌套 contract message 接受 `messageId`/`messageID`/`message_id` 和 `messageUuid`/`messageUUID`/`message_uuid` 作为自身 ID/UUID 别名，indexed resume 会保留 payload 自带的 nested message id。

‎docs/first-second-parity-audit.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -293,6 +293,7 @@ M7 progress now includes:
293293
- `internal/tui`: interaction script touch payloads now recover coordinates from the first `touches`/`targetTouches`/`changedTouches` touch point, map `touchmove` to SGR drag motion, and map `touchcancel` to release events for DOM touch recording fixtures.
294294
- `internal/tui`: REPL dialogs now ignore SGR motion/drag mouse buttons and only act on real press/click events, preventing pointer or touch move replay from closing permission/task dialogs.
295295
- `internal/tui`: interaction script paste payloads now accept DOM `clipboardData`/`dataTransfer` objects and recover pasted text from `text/plain`, `plainText`, and `items[].text` fields for ClipboardEvent recording fixtures.
296+
- `internal/tui`: interaction script paste payloads now also recover `image/*` file items from DOM `clipboardData.items` and `dataTransfer.files` as image paste events while avoiding accidental base64-as-text insertion from image `data` payloads.
296297
- `internal/tui`: interaction script resize payloads now accept DOM/window size aliases such as `innerWidth`/`innerHeight`, `clientWidth`/`clientHeight`, `offsetWidth`/`offsetHeight`, plus ResizeObserver-style `contentRect` and `target` wrappers.
297298
- `internal/tui`: interaction script resize payloads now accept ResizeObserver `contentBoxSize`/`borderBoxSize` array entries using `inlineSize`/`blockSize`, covering modern browser box-size event shapes.
298299
- `internal/tui`: interaction script event expectations now normalize camel/dash/provider-style event names such as `promptSubmitted`, `dialogAction`, `messageSubmitted`, and `focusOut` to canonical screen event types, and accept value aliases such as `eventPayload`, `eventValue`, `actionValue`, and `result`.

‎internal/tui/script_aliases.go‎

Lines changed: 100 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -971,6 +971,9 @@ func (step *ScriptStep) UnmarshalJSON(data []byte) error {
971971
if image := scriptImageJSONField(rawFieldMap, "Image", "image"); image != nil {
972972
step.Image = image
973973
}
974+
if step.Image == nil {
975+
step.Image = scriptClipboardImageJSONField(rawFieldMap, "clipboard", "clipboard_data", "clipboardData", "data_transfer", "dataTransfer", "files", "items", "clipboard_items", "clipboardItems")
976+
}
974977
if specs, ok, err := scriptKeybindingsJSONField(rawFieldMap); err != nil {
975978
return err
976979
} else if ok {
@@ -1836,13 +1839,19 @@ func applyScriptStepActionAlias(step *ScriptStep, fields map[string]json.RawMess
18361839
step.Keys = append(step.Keys, keys...)
18371840
}
18381841
} else if scriptDOMInputEventIsPaste(fields) {
1842+
if step.Image == nil {
1843+
step.Image = scriptClipboardImageJSONField(fields, "paste", "clipboard", "clipboard_data", "clipboardData", "data_transfer", "dataTransfer", "files", "items", "payload", "data", "value")
1844+
}
18391845
if step.Paste == "" {
18401846
step.Paste = scriptActionStringField(fields, "data", "text", "input", "content", "body", "message", "payload", "value")
18411847
}
18421848
} else if step.Text == "" {
18431849
step.Text = scriptActionStringField(fields, "data", "text", "input", "content", "body", "message", "payload", "value")
18441850
}
18451851
case "paste", "pastetext", "pastedtext", "clipboard", "clipboardtext":
1852+
if step.Image == nil {
1853+
step.Image = scriptClipboardImageJSONField(fields, "paste", "clipboard", "clipboard_data", "clipboardData", "data_transfer", "dataTransfer", "files", "items", "payload", "data", "value")
1854+
}
18461855
if step.Paste == "" {
18471856
step.Paste = scriptActionStringField(fields, "paste", "clipboard", "clipboard_data", "clipboardData", "data_transfer", "dataTransfer", "text/plain", "textPlain", "plain_text", "plainText", "clipboard_text", "clipboardText", "items", "item", "clipboard_items", "clipboardItems", "text", "content", "body", "message", "data", "payload", "value")
18481857
}
@@ -3721,6 +3730,9 @@ func scriptActionStringFromJSON(raw json.RawMessage, objectNames []string, depth
37213730
if err := json.Unmarshal(raw, &nested); err != nil {
37223731
return ""
37233732
}
3733+
if scriptActionStringSkipsImageLikeObject(nested, objectNames) {
3734+
return ""
3735+
}
37243736
if value := stringJSONField(nested, objectNames...); value != "" {
37253737
return value
37263738
}
@@ -3736,6 +3748,23 @@ func scriptActionStringFromJSON(raw json.RawMessage, objectNames []string, depth
37363748
return ""
37373749
}
37383750

3751+
func scriptActionStringSkipsImageLikeObject(fields map[string]json.RawMessage, objectNames []string) bool {
3752+
if !scriptActionStringNamesIncludeClipboardPayload(objectNames) || !scriptImageLikeFields(fields) {
3753+
return false
3754+
}
3755+
return stringJSONField(fields, "text/plain", "textPlain", "plain_text", "plainText", "clipboard_text", "clipboardText", "text") == ""
3756+
}
3757+
3758+
func scriptActionStringNamesIncludeClipboardPayload(names []string) bool {
3759+
for _, name := range names {
3760+
switch canonicalScriptStepAction(name) {
3761+
case "paste", "paste-text", "pastetext", "pasted-text", "pastedtext", "clipboard", "clipboard-data", "clipboarddata", "data-transfer", "datatransfer", "clipboard-items", "clipboarditems":
3762+
return true
3763+
}
3764+
}
3765+
return false
3766+
}
3767+
37393768
func scriptScalarStringFromJSON(raw json.RawMessage) string {
37403769
if value := scriptStringFromJSON(raw); value != "" {
37413770
return value
@@ -3804,6 +3833,19 @@ func scriptImageJSONField(fields map[string]json.RawMessage, names ...string) *S
38043833
return nil
38053834
}
38063835

3836+
func scriptClipboardImageJSONField(fields map[string]json.RawMessage, names ...string) *ScriptImage {
3837+
for _, name := range names {
3838+
raw, ok := fields[name]
3839+
if !ok {
3840+
continue
3841+
}
3842+
if image, ok := scriptClipboardImageFromJSON(raw, 0); ok {
3843+
return image
3844+
}
3845+
}
3846+
return nil
3847+
}
3848+
38073849
func scriptMouseFromJSON(raw json.RawMessage, depth int) (*ScriptMouse, bool) {
38083850
raw = bytes.TrimSpace(raw)
38093851
if len(raw) == 0 || bytes.Equal(raw, []byte("null")) {
@@ -3847,6 +3889,50 @@ func scriptMouseFromJSON(raw json.RawMessage, depth int) (*ScriptMouse, bool) {
38473889
return nil, false
38483890
}
38493891

3892+
func scriptClipboardImageFromJSON(raw json.RawMessage, depth int) (*ScriptImage, bool) {
3893+
raw = bytes.TrimSpace(raw)
3894+
if len(raw) == 0 || bytes.Equal(raw, []byte("null")) {
3895+
return nil, false
3896+
}
3897+
if depth >= 8 {
3898+
return nil, false
3899+
}
3900+
if raw[0] == '[' {
3901+
var items []json.RawMessage
3902+
if err := json.Unmarshal(raw, &items); err != nil {
3903+
return nil, false
3904+
}
3905+
for _, item := range items {
3906+
if image, ok := scriptClipboardImageFromJSON(item, depth+1); ok {
3907+
return image, true
3908+
}
3909+
}
3910+
return nil, false
3911+
}
3912+
if raw[0] != '{' {
3913+
return nil, false
3914+
}
3915+
fields := map[string]json.RawMessage{}
3916+
if err := json.Unmarshal(raw, &fields); err != nil {
3917+
return nil, false
3918+
}
3919+
if scriptImageLikeFields(fields) {
3920+
if image, ok := scriptImageFromJSON(raw, depth+1); ok {
3921+
return image, true
3922+
}
3923+
}
3924+
for _, name := range scriptRuntimePayloadWrapperNames("clipboard", "clipboard_data", "clipboardData", "data_transfer", "dataTransfer", "files", "file", "items", "item", "clipboard_items", "clipboardItems") {
3925+
nested, ok := fields[name]
3926+
if !ok {
3927+
continue
3928+
}
3929+
if image, ok := scriptClipboardImageFromJSON(nested, depth+1); ok {
3930+
return image, true
3931+
}
3932+
}
3933+
return nil, false
3934+
}
3935+
38503936
func scriptImageFromJSON(raw json.RawMessage, depth int) (*ScriptImage, bool) {
38513937
raw = bytes.TrimSpace(raw)
38523938
if len(raw) == 0 || bytes.Equal(raw, []byte("null")) {
@@ -3878,7 +3964,7 @@ func scriptImageFromJSON(raw json.RawMessage, depth int) (*ScriptImage, bool) {
38783964
if err := json.Unmarshal(raw, &fields); err != nil {
38793965
return nil, false
38803966
}
3881-
for _, name := range scriptRuntimePayloadWrapperNames("image", "paste_image", "pasteImage", "image_paste", "imagePaste", "attachment", "media") {
3967+
for _, name := range scriptRuntimePayloadWrapperNames("image", "paste_image", "pasteImage", "image_paste", "imagePaste", "attachment", "media", "clipboard", "clipboard_data", "clipboardData", "data_transfer", "dataTransfer", "files", "file", "items", "item", "clipboard_items", "clipboardItems") {
38823968
nested, ok := fields[name]
38833969
if !ok {
38843970
continue
@@ -3898,6 +3984,14 @@ func scriptImageHasData(image ScriptImage) bool {
38983984
return image.Filename != "" || image.MediaType != "" || image.Content != "" || image.SourcePath != "" || image.Dimensions != nil
38993985
}
39003986

3987+
func scriptImageLikeFields(fields map[string]json.RawMessage) bool {
3988+
mediaType := stringJSONField(fields, "media_type", "mediaType", "mime_type", "mimeType", "content_type", "contentType")
3989+
if mediaType == "" {
3990+
mediaType = stringJSONField(fields, "type", "kind")
3991+
}
3992+
return strings.HasPrefix(strings.ToLower(strings.TrimSpace(mediaType)), "image/")
3993+
}
3994+
39013995
func scriptStepHasFocusKey(step *ScriptStep) bool {
39023996
for _, key := range step.Keys {
39033997
if key == "focus-in" || key == "focus-out" {
@@ -4353,6 +4447,11 @@ func (image *ScriptImage) UnmarshalJSON(data []byte) error {
43534447
if image.MediaType == "" {
43544448
image.MediaType = stringJSONField(fields, "media_type", "mime_type", "mimeType", "content_type", "contentType")
43554449
}
4450+
if image.MediaType == "" {
4451+
if mediaType := stringJSONField(fields, "type", "kind"); strings.HasPrefix(strings.ToLower(strings.TrimSpace(mediaType)), "image/") {
4452+
image.MediaType = mediaType
4453+
}
4454+
}
43564455
if image.Content == "" {
43574456
image.Content = stringJSONField(fields, "data", "base64", "base64Content", "contentBase64")
43584457
}

‎internal/tui/tui_test.go‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7100,6 +7100,35 @@ func TestRunInteractionScriptAcceptsClipboardDataPastePayloads(t *testing.T) {
71007100
"expandedText": "alpha\nbetagammadelta",
71017101
"pastedContentCount": 3
71027102
}
7103+
},
7104+
{
7105+
"clipboardData": {
7106+
"items": [
7107+
{"kind": "file", "type": "image/png", "name": "clip.png", "data": "AAAA"}
7108+
]
7109+
},
7110+
"expectPrompt": {
7111+
"text": "[Pasted text #1 +1 lines][Pasted text #2][Pasted text #3][Image #4]",
7112+
"expandedText": "alpha\nbetagammadelta[Image #4]",
7113+
"pastedContentCount": 4,
7114+
"pastedContents": {"id": 4, "type": "image", "mediaType": "image/png", "filename": "clip.png", "content": "AAAA"}
7115+
}
7116+
},
7117+
{
7118+
"action": "paste",
7119+
"payload": {
7120+
"dataTransfer": {
7121+
"files": [
7122+
{"type": "image/jpeg", "fileName": "drop.jpg", "base64": "BBBB", "sourcePath": "/tmp/drop.jpg"}
7123+
]
7124+
}
7125+
},
7126+
"expectPrompt": {
7127+
"text": "[Pasted text #1 +1 lines][Pasted text #2][Pasted text #3][Image #4] [Image #5]",
7128+
"expandedText": "alpha\nbetagammadelta[Image #4] [Image #5]",
7129+
"pastedContentCount": 5,
7130+
"pastedContents": {"id": 5, "type": "image", "mediaType": "image/jpeg", "filename": "drop.jpg", "content": "BBBB", "sourcePath": "/tmp/drop.jpg"}
7131+
}
71037132
}
71047133
]`))
71057134
if err != nil {

0 commit comments

Comments
 (0)