Code change for WV improve event related entities

[harshitasingh360]

When the scraper reads a West Virginia committee agenda, it now spots two kinds of things and links them to the agenda item. First, if the agenda mentions bills like HB5675/SB939, it records each one separately (HB5675 and SB939) as related bills. Second, the agenda also lists government rules like 191 CSR 03. Since these aren't bills, instead of tagging them as bills, it saves them as links to the rule text.

Output -WV_13Aug.zip

[jessemortenson]

The related bill entity and admin rule media link logic seems to be good here, but now there are a lot of agenda items that actually are not top-level agenda items. See requests for change below.

[jessemortenson]

I think we may have a misunderstanding about how to parse out agenda items and related entities. When I look in the wv.3.zip data, I am seeing that each related_entity (or media link) is getting its own agenda item. So for a made up example:

1. Roll call
2. Consider bills 
    - SB 1945
    - HB 349
3. Adjourn

The output data in wv.3.zip shows that would be parsed to the following agenda items:

1. Roll Call
2. Consider Bills
3. SB 1945 (related bill: SB 1945)
4. HB 349 (related bill: SB 349)
5. Adjourn

The problem is that text within an agenda item is being promoted up to the level of a new agenda item. The agenda output that we should see is:

1. Roll Call
2. Consider Bills (related bills: SB 1945, SB 349)
3. Adjourn

In other words, we should try to keep the top-level agenda items as described in the event text, and find-and-associate bill identifiers (and admin law rules). In the examples above, where I note "(related bill: )" that is to indicate that add_bill() has been called to associate those related bill(s), NOT that the text of agenda item has to include the text "(related bill: ...)".

I'm not sure exactly where to make this change in the code, hence the general comment up top here.

[harshitasingh360]

Sure Jesse, Thank you for the details, let me look into the same and get back to you with the code change

[jessemortenson]

I'm still seeing some issues with agenda item parsing. The structure of the HTML isn't very helpful, so I can see how this would be slippery. If there is a hard tradeoff or constraint here, let me know. But I think this can be refined further. Here are some examples:

Still seeing sub-agenda items split into their own top-level agenda items

Again, I get why this is challenging because there isn't a consistent and distinct HTML structure. But maybe there is a logic AFTER parsing from HTML that can detect whether there is a set of items that start with either numerals ("1.", "2.", "3."... etc) OR roman numerals ("I.", "II", "III" ... etc), and then remove-and-merge items in between them that do NOT start with that top-level notation (related bills/media would get attached to the top-level agenda item before the removed ones).

http://www.wvlegislature.gov/committees/interims/intcomsched.cfm?day1=01/12/2026
Joint Standing Committee on Finance 2026-01-12 14:00:00

['1.   Call to Order', '2.   Roll Call', '3.  Approval of Minutes', '4.  Review of FY 2026 Budget Year Matters', 'Eric Nelson, Cabinet Secretary, West Virginia Department of Revenue', 'Mark Muchow, Deputy Secretary, West Virginia Department of Revenue', 'Pete Shirley, Deputy Secretary, West Virginia Department of Revenue', 'Mike McKown, Director, West Virginia State Budget Office', '5.  Presentation: Reform Ideas for West Virginia’s Medicaid Program', 'Ben Beakes, Executive Director, West Virginia Association of Health Plans', '6.  Presentation: Managed Care Impacts on Long-Term Care', 'Marty Wright, Chief Executive Officer, West Virginia Health Care Association', '7.  Other Business', '8.    Adjournment', '*Agenda subject to change']	

Items like "Eric Nelson....", "Mark Muchow...", "Marty Wright..." etc are details nested underneath agenda items, and would ideally not be their own top-level agenda items. "*Agenda subject to change" is not an agenda item. Looking at the source page, I see 8 top level agenda items for that event (the numbered items).

http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=January 21, 2026&chart=jud
Senate Judiciary Committee 2026-01-21 20:00:00

['AGENDA', 'Date: Wednesday, January 21, 2026  Time: 3:00 p.m.  Location: Senate Judiciary Committee Room, 208 West', 'I.	Call to Order', 'II.	Roll Call', 'III.	Approval of Minutes', 'IV.	Bills to be Considered', '1.	Com. Sub. for S. B. 281 - Governor''s Committee on Crime, Delinquency, and Correction rule relating to law-enforcement training and certification standards (DG)  a.	149 CSR 02 - Law Enforcement Training and Certification  b.	87 CSR 01 - Fire Code  c.	55 CSR 01 - Emergency Planning and Community Right-To-Know  d.	81 CSR 03 - West Virginia State Police Career Progression System  e.	81 CSR 05 - Contracted Police or Security Services  f.	81 CSR 08 - West Virginia State Police Member Grievance Procedure', '2.	Com. Sub. for S. B. 309 - DMV rule relating to motor vehicle equipment (DG)  a.	91 CSR 11 - Motor Vehicle Equipment   b.	91 CSR 13 - Compulsory Motor Vehicle Liability Insurance  c.	91 CSR 25 – Special Purpose Vehicles', 'V.	Other Business', 'VI.	Adjournment', 'Note: Agenda and order are tentative and subject to change.']

"AGENDA" is not an agenda item. "Date: ...." is not an agenda item. The items starting "1." and "2." are details nested underneath the agenda item "IV. Bills to be Considered" not their own top-level items. So what I'm expecting to see is a set of agenda items corresponding to the items that have roman numerals (I, II, III, IV etc), including an item "IV. Bills to be considered" that should have two related entities ("SB 281" and "SB 309").

Sometimes the first agenda item is missing

"1. Call to Order" not found in the agenda items for http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=March 10, 2026&chart=edu

"I. Call to Order" not found in agenda items for http://www.wvlegislature.gov/committees/senate/senate_com_agendas.cfm?input=3/10/2026&Chart=hhr

These two examples are sort of trivial: "Call to order" does not convey anything substantive. HOWEVER it makes me nervous that perhaps a first agenda item that does have meaning could be missed.

[jessemortenson]

Hi Harshita, I'm still seeing some data issues with the agenda items. I responded to the prior comment, so please check that. Thank you