Refactor homogenized filtering and config policy#195
Conversation
luigilcsilva
commented
Jul 10, 2026
- reorganize param config into run, filters, output, preparation, and diagnostics sections
- add instrument_type_homogenized and object_type_homogenized input filters
- apply homogenized filters during preparation and fail only when all inputs are emptied
- always create homogenized columns internally and add configurable output retention
- remove tie_result=3 participation semantics from deduplication
- move diagnostics/preparation settings out of flags_translation.yaml
- keep scientific translation schema in flags_translation.yaml and require core science keys
- add fallback warnings for optional scientific settings missing from flags_translation.yaml
- preserve legacy flat config compatibility through normalization/migration
- support formatted JSON output from python config.py
- expand tests for filters, homogenized output policy, translation validation, and empty-input handling
- reorganize param config into run, filters, output, preparation, and diagnostics sections - add instrument_type_homogenized and object_type_homogenized input filters - apply homogenized filters during preparation and fail only when all inputs are emptied - always create homogenized columns internally and add configurable output retention - remove tie_result=3 participation semantics from deduplication - move diagnostics/preparation settings out of flags_translation.yaml - keep scientific translation schema in flags_translation.yaml and require core science keys - add fallback warnings for optional scientific settings missing from flags_translation.yaml - preserve legacy flat config compatibility through normalization/migration - support formatted JSON output from python config.py - expand tests for filters, homogenized output policy, translation validation, and empty-input handling
|
@gschwend Apliquei as ideias que discutimos hoje, sobre a eliminação do tie_result 3 e aplicação dos filtros sempre no início. Depois podemos ajustar os defaults, se necessário. Ainda terá a tarefa futura de ajuste do frontend também. @crisingulani Eu mexi bastante na configuração (config.py, config.template.yaml...). Não tenho certeza de como será a interação disso com o frontend, então o seu olhar sobre a configuração será importante. Nesse primeiro momento minha preocupação é só se o que já funcionava e já está implementado no frontend vai continuar funcionando. Não estou pensando em campos/opções novas. Ah, eu acho que consegui consertar a formatação do json gerado por |
- require object_type_homogenized translation only when object-type filters need it - let existing homogenized columns and compatible fast paths bypass YAML translation errors - fail on missing YAML sources or invalid conditions only when YAML is the required path - add tests for fast-path bypasses and required YAML translation failures
- keep flags_translation.yaml validation strict for external science settings - ignore runtime-only diagnostic/preparation overlays inside _homogenize validation - add regression coverage for runtime translation_config overlays
|
vou precisar de ajuda para testar |