How to get the usage and cost if the response is streamed?

[scottmessinger]

I'm unclear how I can both stream the response and get the usage/cost.

To stream:

{:ok, stream} = ExLLM.stream_chat(:openai, messages)

for chunk <- stream do
  case chunk do
    %{content: content} when content != nil ->
      IO.write(content)
      
    %{finish_reason: reason} when reason != nil ->
      IO.puts("\nFinished: #{reason}")
      
    _ ->
      # Other chunk types (role, etc.)
      :ok
  end
end

and

{:ok, response} = ExLLM.chat(:openai, messages)
%ExLLM.Types.LLMResponse{
  content: "I'm doing well, thank you!",
  model: "gpt-4o",
  finish_reason: "stop",
  usage: %{
    input_tokens: 12,
    output_tokens: 8,
    total_tokens: 20
  },
  cost: %{
    input_cost: 0.00006,
    output_cost: 0.00016,
    total_cost: 0.00022,
    currency: "USD"
  }
}

I'm not sure how to get the response if I'm streaming. Is this possible?

[scottmessinger]

It looks like it might be possible but "metadata" is a bare map on the Types.StreamChunk and, more importantly, at least with Gemini, the metadata isn't being set. The problem appears here:

  defp convert_content_response_to_stream_chunk(%GenerateContentResponse{} = response) do
    case response.candidates do
      [candidate | _] ->
        text = extract_text_from_candidate(candidate)


        %Types.StreamChunk{
          content: text,
          finish_reason: candidate.finish_reason
        }


      [] ->
        %Types.StreamChunk{
          content: "",
          finish_reason: "stop"
        }
    end
  end

[azmaveth]

The metadata issue should be fixed now. Can you test again and let me know if it works for you?