Categorize treats CSV survey_text value None as missing (nan)

[digitalWestie]

Input CSV rows with unquoted survey_text of None (e.g. participant_1374,None) causes categorize to crash with a Pydantic validation error (text must be a string; got nan).

2026-06-04 15:17:43,044 - ERROR - Process crashed with an error: 1 validation error for Statement
text
  Input should be a valid string [type=string_type, input_value=nan, input_type=float]
    For further information visit https://errors.pydantic.dev/2.13/v/string_type
Traceback (most recent call last):
  File "/home/digitalwestie/Code/consulsensemaking/sensemaking-tools/src/categorization_runner.py", line 521, in <module>
    log_dir_path = asyncio.run(main())
                   ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/digitalwestie/Code/consulsensemaking/sensemaking-tools/src/categorization_runner.py", line 382, in main
    statements_to_process = _convert_csv_rows_to_statements(original_csv_rows)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalwestie/Code/consulsensemaking/sensemaking-tools/src/categorization_runner.py", line 128, in _convert_csv_rows_to_statements
    statements_map[statement_id] = custom_types.Statement(
                                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalwestie/Code/consulsensemaking/sensemaking-tools/.venv/lib/python3.12/site-packages/pydantic/main.py", line 263, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for Statement
text
  Input should be a valid string [type=string_type, input_value=nan, input_type=float]

This is not an empty field; pandas read_csv interprets the token None as NA.

For example, this will fail:

participant_id,survey_text
1,None
2,I'm not aware of any active campaigns

Quotes will also fail:

participant_id,survey_text
1,"None"
2,"I'm not aware of any active campaigns"

This will work though:

participant_id,survey_text
1,None at all
2,I'm not aware of any active campaigns

Suggested fix is to use:

keep_default_na=False when calling read_csv e.g. in categorization_runner.py

def _read_csv_to_dicts(input_file_path: str) -> List[Dict[str, str]]:
  """Reads a CSV file into a list of dictionaries."""
  if not input_file_path:
    raise ValueError("Input file path is missing!")

  file_path = os.path.expanduser(input_file_path)
  if not os.path.exists(file_path):
    raise FileNotFoundError(f"Input file not found: {file_path}")

  df = pd.read_csv(file_path, dtype=str, keep_default_na=False)
  # Convert all header names to lowercase
  df.columns = [h.lower() for h in df.columns]
  return df.to_dict("records")