Skip to content

Refactor homogenized filtering and config policy#195

Open
luigilcsilva wants to merge 8 commits into
mainfrom
object-type-column
Open

Refactor homogenized filtering and config policy#195
luigilcsilva wants to merge 8 commits into
mainfrom
object-type-column

Conversation

@luigilcsilva

Copy link
Copy Markdown
Collaborator
  • reorganize param config into run, filters, output, preparation, and diagnostics sections
  • add instrument_type_homogenized and object_type_homogenized input filters
  • apply homogenized filters during preparation and fail only when all inputs are emptied
  • always create homogenized columns internally and add configurable output retention
  • remove tie_result=3 participation semantics from deduplication
  • move diagnostics/preparation settings out of flags_translation.yaml
  • keep scientific translation schema in flags_translation.yaml and require core science keys
  • add fallback warnings for optional scientific settings missing from flags_translation.yaml
  • preserve legacy flat config compatibility through normalization/migration
  • support formatted JSON output from python config.py
  • expand tests for filters, homogenized output policy, translation validation, and empty-input handling

- reorganize param config into run, filters, output, preparation, and diagnostics sections
- add instrument_type_homogenized and object_type_homogenized input filters
- apply homogenized filters during preparation and fail only when all inputs are emptied
- always create homogenized columns internally and add configurable output retention
- remove tie_result=3 participation semantics from deduplication
- move diagnostics/preparation settings out of flags_translation.yaml
- keep scientific translation schema in flags_translation.yaml and require core science keys
- add fallback warnings for optional scientific settings missing from flags_translation.yaml
- preserve legacy flat config compatibility through normalization/migration
- support formatted JSON output from python config.py
- expand tests for filters, homogenized output policy, translation validation, and empty-input handling
@luigilcsilva

luigilcsilva commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator Author

@gschwend Apliquei as ideias que discutimos hoje, sobre a eliminação do tie_result 3 e aplicação dos filtros sempre no início. Depois podemos ajustar os defaults, se necessário. Ainda terá a tarefa futura de ajuste do frontend também.

@crisingulani Eu mexi bastante na configuração (config.py, config.template.yaml...). Não tenho certeza de como será a interação disso com o frontend, então o seu olhar sobre a configuração será importante. Nesse primeiro momento minha preocupação é só se o que já funcionava e já está implementado no frontend vai continuar funcionando. Não estou pensando em campos/opções novas. Ah, eu acho que consegui consertar a formatação do json gerado por python config.py, para evitar de a gente ter que ir no https://jsonformatter.curiousconcept.com/# formatar. Acho que agora já vai sair bem formatado no terminal.

- require object_type_homogenized translation only when object-type filters need it
- let existing homogenized columns and compatible fast paths bypass YAML translation errors
- fail on missing YAML sources or invalid conditions only when YAML is the required path
- add tests for fast-path bypasses and required YAML translation failures
- keep flags_translation.yaml validation strict for external science settings
- ignore runtime-only diagnostic/preparation overlays inside _homogenize validation
- add regression coverage for runtime translation_config overlays
@luigilcsilva luigilcsilva linked an issue Jul 14, 2026 that may be closed by this pull request
@gschwend

Copy link
Copy Markdown
Member

vou precisar de ajuda para testar

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

back-end da seleção por object type e instrumento

2 participants