Catch-up is extremely slow on large chats because of message hiding

[alpha-omicron]

Hey, I've been trying to use Summaryception on an existing very long conversation. The chat has 27,219 messages (last message index/id). Initial catch-up requires thousands of batches.

From inspecting the browser logs after enabling TRACE, the slowdown does not appear to be the summarizer API itself. The API response is fast, around 2-3 seconds per summary call. The expensive part appears to be message ghosting/hiding after each batch, which takes multiple minutes.

Example from one batch:

• 16:54:08.218 summarizer call starts
• 16:54:10.467 summarizer result returns
• 16:54:10.468 summarizedUpTo updated to 1858
• 16:56:31.925 Ghosted messages from index 0 to 1858

At this scale, doing expensive message hiding after every catch-up batch makes initial backlog processing take days even though the summarization calls themselves are fast.

Firefox console logs from sillytavern with TRACE logs enabled on summaryception

16:53:51.983 [Summaryception] Ghosted messages from index 0 to 1838 index.js:195:42
16:54:07.437 [Summaryception] [TRACE] <<< EXITING SUMMARIZEONEBATCHFROMTURNS - SUCCESS index.js:204:17
16:54:07.437 [Summaryception] [TRACE]   >>> SUMMARIZEONEBATCHFROMTURNS RETURNED SUCCESS index.js:204:17
16:54:08.203 [Summaryception] [TRACE]   LOOP ITERATION - COMPLETED: 3, FAILED: 0 index.js:204:17
16:54:08.213 [Summaryception] [TRACE]   CURRENTVISIBLE TURNS: 12690, VERBATIMTURNS LIMIT: 35 index.js:204:17
16:54:08.214 [Summaryception] [TRACE]   ABOUT TO CALL SUMMARIZEONEBATCHFROMTURNS... index.js:204:17
16:54:08.214 [Summaryception] [TRACE] >>> ENTERING SUMMARIZEONEBATCHFROMTURNS index.js:204:17
16:54:08.214 [Summaryception] [TRACE]   VISIBLETURNS: 12690 index.js:204:17
16:54:08.215 [Summaryception] [TRACE]   ELIGIBLETURNS AFTER FILTERING: 12690 index.js:204:17
16:54:08.215 [Summaryception] [TRACE]   BATCHSIZE: 10 index.js:204:17
16:54:08.215 [Summaryception] [TRACE]   BATCH PREPARED: 10 index.js:204:17
16:54:08.215 [Summaryception] [TRACE]   STARTIDX: 1840 endIdx: 1858 index.js:204:17
16:54:08.215 [Summaryception] [TRACE]   STORE.SUMMARIZEDUPTO: 1838 index.js:204:17
16:54:08.215 [Summaryception] [TRACE]   PASSAGESTART: 1839 endIdx: 1858 index.js:204:17
16:54:08.215 [Summaryception] [TRACE]   ABOUT TO CALL BUILDPASSAGEFROMRANGE... index.js:204:17
16:54:08.215 [Summaryception] [TRACE]   BUILDPASSAGEFROMRANGE RETURNED, LENGTH: 11511 index.js:204:17
16:54:08.215 [Summaryception] [TRACE]   ABOUT TO CALL BUILDFULLCONTEXT... index.js:204:17
16:54:08.216 [Summaryception] [TRACE]   BUILDFULLCONTEXT RETURNED, LENGTH: 25023 index.js:204:17
16:54:08.216 [Summaryception] [TRACE]   ABOUT TO CALL CALLSUMMARIZER... index.js:204:17
16:54:08.216 [Summaryception] [TRACE] >>> ENTERING CALLSUMMARIZER index.js:204:17
16:54:08.216 [Summaryception] [TRACE]   STORYTXT LENGTH: 11511 index.js:204:17
16:54:08.216 [Summaryception] [TRACE]   CONTEXTSTR LENGTH: 25023 index.js:204:17
16:54:08.216 [Summaryception] [TRACE]   SETTINGS LOADED: 
Object { connectionSource: "default", enabled: true }
index.js:204:17
16:54:08.218 [Summaryception] ── Summarizer Call ── index.js:195:42
16:54:08.218 [Summaryception] Context str length: 25023 chars index.js:195:42
16:54:08.218 [Summaryception] Story txt length: 11511 chars index.js:195:42
16:54:08.218 [Summaryception] No prompt manager available, skipping toggle snapshot. index.js:195:42
16:54:08.218 [Summaryception] [TRACE]   ATTEMPT 0 STARTING... index.js:204:17
16:54:08.218 [Summaryception] [TRACE]   ABOUT TO CALL SENDSUMMARIZERREQUEST WITH: 
Object { connectionSource: "default", summarizerSystemPrompt: "Role: precise narrative-state tracker. Output only", promptLength: 37437 }
index.js:204:17
16:54:10.466 [Summaryception] [TRACE]   SENDSUMMARIZERREQUEST RETURNED: [redacted]
16:54:10.467 [Summaryception] Result: [redacted]
16:54:10.467 [Summaryception] [TRACE] <<< EXITING CALLSUMMARIZER WITH SUCCESS index.js:204:17
16:54:10.467 [Summaryception] [TRACE]   CALLSUMMARIZER RETURNED, LENGTH: 394 index.js:204:17
16:54:10.468 [Summaryception] [TRACE]   UPDATED STORE.SUMMARIZEDUPTO TO: 1858 index.js:204:17
16:56:31.925 [Summaryception] Ghosted messages from index 0 to 1858 index.js:195:42
16:56:41.529 [Summaryception] [TRACE] <<< EXITING SUMMARIZEONEBATCHFROMTURNS - SUCCESS index.js:204:17
16:56:41.529 [Summaryception] [TRACE]   >>> SUMMARIZEONEBATCHFROMTURNS RETURNED SUCCESS index.js:204:17
16:56:42.297 [Summaryception] [TRACE]   LOOP ITERATION - COMPLETED: 4, FAILED: 0 index.js:204:17
16:56:42.306 [Summaryception] [TRACE]   CURRENTVISIBLE TURNS: 12680, VERBATIMTURNS LIMIT: 35 index.js:204:17
16:56:42.306 [Summaryception] [TRACE]   ABOUT TO CALL SUMMARIZEONEBATCHFROMTURNS... index.js:204:17
16:56:42.307 [Summaryception] [TRACE] >>> ENTERING SUMMARIZEONEBATCHFROMTURNS index.js:204:17
16:56:42.307 [Summaryception] [TRACE]   VISIBLETURNS: 12680 index.js:204:17
16:56:42.307 [Summaryception] [TRACE]   ELIGIBLETURNS AFTER FILTERING: 12680 index.js:204:17
16:56:42.308 [Summaryception] [TRACE]   BATCHSIZE: 10 index.js:204:17
16:56:42.308 [Summaryception] [TRACE]   BATCH PREPARED: 10 index.js:204:17
16:56:42.308 [Summaryception] [TRACE]   STARTIDX: 1860 endIdx: 1878 index.js:204:17
16:56:42.308 [Summaryception] [TRACE]   STORE.SUMMARIZEDUPTO: 1858 index.js:204:17
16:56:42.308 [Summaryception] [TRACE]   PASSAGESTART: 1859 endIdx: 1878 index.js:204:17
16:56:42.308 [Summaryception] [TRACE]   ABOUT TO CALL BUILDPASSAGEFROMRANGE... index.js:204:17
16:56:42.308 [Summaryception] [TRACE]   BUILDPASSAGEFROMRANGE RETURNED, LENGTH: 11404 index.js:204:17
16:56:42.308 [Summaryception] [TRACE]   ABOUT TO CALL BUILDFULLCONTEXT... index.js:204:17
16:56:42.308 [Summaryception] [TRACE]   BUILDFULLCONTEXT RETURNED, LENGTH: 25418 index.js:204:17
16:56:42.309 [Summaryception] [TRACE]   ABOUT TO CALL CALLSUMMARIZER... index.js:204:17
16:56:42.309 [Summaryception] [TRACE] >>> ENTERING CALLSUMMARIZER index.js:204:17
16:56:42.309 [Summaryception] [TRACE]   STORYTXT LENGTH: 11404 index.js:204:17
16:56:42.309 [Summaryception] [TRACE]   CONTEXTSTR LENGTH: 25418 index.js:204:17
16:56:42.309 [Summaryception] [TRACE]   SETTINGS LOADED: 
Object { connectionSource: "default", enabled: true }
index.js:204:17
16:56:42.310 [Summaryception] ── Summarizer Call ── index.js:195:42
16:56:42.311 [Summaryception] Context str length: 25418 chars index.js:195:42
16:56:42.311 [Summaryception] Story txt length: 11404 chars index.js:195:42
16:56:42.311 [Summaryception] No prompt manager available, skipping toggle snapshot. index.js:195:42
16:56:42.311 [Summaryception] [TRACE]   ATTEMPT 0 STARTING... index.js:204:17
16:56:42.311 [Summaryception] [TRACE]   ABOUT TO CALL SENDSUMMARIZERREQUEST WITH: 
Object { connectionSource: "default", summarizerSystemPrompt: "Role: precise narrative-state tracker. Output only", promptLength: 37725 }
index.js:204:17
16:56:45.024 [Summaryception] [TRACE]   SENDSUMMARIZERREQUEST RETURNED: [redacted]
16:56:45.025 [Summaryception] Result: [redacted]
16:56:45.026 [Summaryception] [TRACE] <<< EXITING CALLSUMMARIZER WITH SUCCESS index.js:204:17
16:56:45.026 [Summaryception] [TRACE]   CALLSUMMARIZER RETURNED, LENGTH: 344 index.js:204:17
16:56:45.026 [Summaryception] [TRACE]   UPDATED STORE.SUMMARIZEDUPTO TO: 1878 index.js:204:17

Possible improvement

Could Summaryception support deferred/batched message hiding for catch-up? For example:

• During catch-up, mark messages as sc_ghosted but optionally skip visual /hide calls.
• After catch-up completes, provide an action to apply actual SillyTavern message hiding to all already-ghosted messages.

Having a button to perform this action explicitly would probably also be useful for being able to hide/unhide the messages after modifying Disable Message Hiding (e.g. to retroactively hide/unhide messages after they've been batched, after changing this toggle).

Also, I think the ghosting logic runs on every message id from 0 to the last msg id in the batch? If it must run every time, why not at least make it run only from batch start msg id -> batch end msg id? Though there might actually be some good reasons for that.

I think it would also be a good idea to consider making the catch-up logic not block, and run async from batching. I don't know how important it is to have it block further batching, but it doesn't seem like it should have to block further batching logic to me. However, this isn't quite a fix for it being slow, it would just let the batching itself run much faster, the slowness should still be addressed somehow.

[alpha-omicron]

I've actually moved to https://github.com/vadash/Extension-Summaryception that fork does many things in better ways, like running a single /hide with the whole range, and only doing startIdx->endIdx in the logic, instead of 0->endIdx. But I also really like it's other features. I'll keep this opened, as it remains an issue for this repo, but I'm no longer going to be available to test it, as I have the fork installed instead of this extension now.