Migrate Gemini 1.5-Pro models to Gemini 2.0 Flash models

[allisonmorgan]

Google recommends migrating Gemini 1.5 Pro models to Gemini 2.0 Flash. When I evaluated this branch (currently equivalent with main & dev), the summary performance under Gemini 1.5 Pro (current LLM) is very similar to Gemini 2.0 Flash (0.335 vs 0.333 using DeepEval's summarization score; 0.360 vs 0.386 ROUGE). These differences seem very slight to me, so this branch migrates our summarization functionality to Gemini 2.0 Flash.

Two things worth noting: (1) Gemini 2.0 Flash models will be deprecated February 26th, so we'll want to redo this analysis in seven months. (2) I also tested Gemini 2.0 Flash Lite, which is marketed as Google's cheapest model. Performance based on our DeepEval summarization score is slightly lower (0.335 vs 0.285 summarization score; 0.360 vs 0.365 ROUGE). Qualitatively, these summaries do feel really similar though so happy to re-run the Gemini 2.0 Flash Lite evaluation if you think we should?

Example Documents

	Gemini 1.5 Pro	Gemini 2.0 Flash	Gemini 2.0 Flash Lite
1	This presentation outlines the EPA Climate Pollution Reduction Grants program for the Salt Lake City metropolitan area. It details the program's timeline, including key dates for guidance release, applications, and grant awards, along with the required deliverables such as the Priority Climate Action Plan ("PCAP") and the Comprehensive Climate Action Plan ("CCAP"). The presentation also discusses the two types of grants: Planning Grants and Implementation Grants.	These slides provide an overview of the EPA's Climate Pollution Reduction Grants program. The presentation includes a timeline for metro areas, key deliverables, and information on planning and implementation grants. The slides are from a Salt Lake City Metropolitan Area Local Governments Meeting on April 24, 2023.	This PowerPoint presentation is about EPA Climate Pollution Reduction Grants. It covers topics such as an overview of the grant program, SLC Metropolitan Statistical Area process, and the grant timeline. The presentation also outlines key deliverables, including a priority climate action plan and comprehensive climate action plan.
2	The document illustrates a plan for Kensington Byway at the intersection of 1300 East, focused on implementing bike-friendly infrastructure. It details the placement of new signage, including "Neighborhood Byway" designations and pedestrian signals, as well as pavement markings like shared lane markings and sharrows. Specific sign types and dimensions are provided, along with references to separate sheets for detailed placement and custom sign information.	The diagram illustrates the proposed signing and striping plan for the Kensington East-West Neighborhood Byway. It details the placement of various signs, including custom neighborhood byway signs and pedestrian signal signs, along Kensington Avenue. The plan also includes pavement markings and messages to enhance safety and wayfinding.	This document is a diagram for the Kensington East-West Neighborhood Byway, specifically focusing on signing and striping details. The diagram includes the layout of lanes, bike lanes, pedestrian signals, and the locations of various signs and pavement markings along Kensington Avenue and 1300 E. It also provides a sign code legend and key to understand the markings used in the plan.
3	The Salt Lake City Planning Commission met on December 10, 2008. Several items were on the agenda, including discussion of the Pavlos and Leon Ordinance Text Amendment, zoning changes, and building code revisions. Public comment was heard on the matter.	This document is the agenda for the Salt Lake City Planning Commission meeting held on December 10, 2008. The agenda includes reports from the Chair and Vice Chair, a review of condition of previously approved planned development, public hearings, and discussions on zoning and building ordinances. Several petitions related to zoning amendments, master plan amendments, and surplus property were postponed.	This document is the agenda for the Salt Lake City Planning Commission meeting held on December 10, 2008. The agenda includes reports from the Chair and Vice Chair, public hearings, and zoning ordinance discussions. The meeting adjourned at 6:20 p.m.

• What additional steps are required to test this branch locally?

Run the application locally and confirm that the document summary functionality still works for you.

• Are there any areas you would like extra review? None.
• Are there any rake tasks to run on production? None.

[lkacenja]

I tested this out locally with Gemini 2.0 Flash. It worked well and seemed performant. My one comment is non-blocking. As far as cost analysis and consideration of the Gemini 2.0 Flash Lite model goes, I'd lean toward the slightly better accuracy of Gemini 2.0 Flash, while monitoring costs over time. Right now, it seems like our biggest expense is running the evaluation suite. The below expense chart has spikes on days that correlate with evaluation work.

Each peak represents ~$10-15 charge. Summary certainly contributes to that cost. Unless we think the cost we are incurring is problematic, I think it is worth using the more expensive model. I'll make a Slack reminder to evaluate our spend mid-August.

[lkacenja]

Non-blocking: I think we could go ahead and take 1.5-pro out of the list. It feels pretty unlikely to me that we'd revert.

[allisonmorgan]

Great call! Agreed. Will remove this option.