Support caching of chat cogs

[ric2b]

While developing a new workflow it is common to have to re-run it multiple times for testing.

Chat cogs result in both higher costs and slower iteration.

I'm currently using this short monkeypatch which can serve as example, but a real implementation would need to support sessions (and maybe other parameters I haven't considered) and should ideally be configurable per chat cog, instead of globally:

require 'digest'

module CachedChat
  def execute(input)
    cache_dir = '/tmp/roast_chat_cache'

    chat_digest = Digest::SHA256.hexdigest(input.valid_prompt!)
    cache_key = "#{config.valid_provider!}:#{config.valid_model}:#{chat_digest}"
    cache_file = File.join(cache_dir, "chat_#{cache_key}.txt")

    if File.exist?(cache_file)
      cached_response = File.read(cache_file)
      empty_session = Roast::Cogs::Chat::Session.new([])
      return Roast::Cogs::Chat::Output.new(empty_session, cached_response)
    end

    result = super
    FileUtils.mkdir_p(cache_dir)
    File.write(cache_file, result.response)
    result
  end
end

Roast::Cogs::Chat.prepend(CachedChat)

[dersam]

Need some clarification on intent. Is the intention to make workflows with multiple chat cog steps that feed into each other, and allow the ability to designate a chat cog as "cacheable" so we only run it once rather than every time the workflow runs?

In that case - why is the chat cog necessary? It seems like this should be hard coded data in the workflow.

[juniper-shopify]

I think the intent is for ease of debugging / developing a workflow, where you have a chat cog that is a dependency of a later cog that you're working on and need to run again and again. You want the first chat cog to not have to re-run while you're doing this but still be present in the workflow so that the later cog can use its results as input. Not as a production capability, but as a development/debugging tool. Is that what you're asking for?

This is something I've been wanting to. It will make debugging a lot faster as well as cheaper. If that's what you want I think it's a very legit feature request, and I'll try to prioritize it but my intuition is that there's a bunch of design to get right so that it "just works" from a user perspective, which is why I haven't built it already