Skip to content

metasurvey.core— Reproducible Survey Data Processing with Step Pipelines #796

Description

@mauroloprete

Editor:
Submitting Author Name: Mauro Loprete
Submitting Author Github Handle: @mauroloprete
Other Package Authors Github handles: @natydasilva, @fabriciomsv
Repository: https://github.com/metasurveyr/metasurvey.core
Version submitted: 0.3.0
Submission type: Standard
Editor: TBD
Reviewers: TBD

Archive: TBD
Version accepted: TBD
Language: en


  • Paste the full DESCRIPTION file inside a code block below:
Package: metasurvey.core
Title: Local Engine for Reproducible Survey Data Processing
Version: 0.3.0
URL: https://metasurveyr.github.io/metasurvey.core/,
    https://github.com/metasurveyr/metasurvey.core
BugReports: https://github.com/metasurveyr/metasurvey.core/issues
Authors@R:
    c(
        person(
            given = "Mauro",
            family = "Loprete",
            role = c("aut", "cre"),
            email = "mauro.loprete@fcea.edu.uy",
            comment = c(ORCID = "0000-0003-1560-0183")
        ),
        person(
            given = "Natalia",
            family = "da Silva",
            role = "aut",
            email = "natalia.dasilva@fcea.edu.uy",
            comment = c(ORCID = "0000-0002-6031-7451")
        ),
        person(
            given = "Fabricio",
            family = "Machado",
            role = "aut",
            email = "fabricio.mch.slv@gmail.com",
            comment = c(ORCID = "0009-0008-2653-9166")
        )
    )
Description: The processing engine of the 'metasurvey' ecosystem. Provides
    a step-based pipeline for reproducible survey data processing, building on
    the 'survey' package for complex sampling designs. Supports rotating panels
    with bootstrap replicate weights and a recipe system for reproducing data
    transformation workflows across survey editions.
License: GPL (>= 3)
Depends:
    R (>= 4.1)
Imports:
    data.table (>= 1.14.2),
    cli (>= 3.0.0),
    glue (>= 1.6.0),
    lifecycle (>= 1.0.0),
    jsonlite (>= 1.7.2),
    R6 (>= 2.5.0),
    survey (>= 4.2.1),
    methods,
    stats,
    utils
Suggests:
    tibble (>= 3.1.3),
    gt (>= 0.10.0),
    visNetwork (>= 2.0.9),
    htmltools (>= 0.5.0),
    foreign (>= 0.8-81),
    haven,
    eph,
    PNADcIBGE,
    ipumsr,
    openxlsx,
    rio (>= 0.5.27),
    here (>= 1.0.1),
    convey,
    digest,
    dplyr (>= 1.0.7),
    magrittr,
    parallel,
    testthat (>= 3.0.0),
    knitr (>= 1.33),
    rmarkdown (>= 2.11)
VignetteBuilder: knitr
Config/testthat/edition: 3
Encoding: UTF-8
Language: en-US
Roxygen: list(markdown = TRUE)
RoxygenNote: 8.0.0

Scope

  • Please indicate which category or categories from our package fit policies this package falls under:

    • workflow automation
  • Explain how and why the package falls under these categories (1-2 paragraphs):

    metasurvey.core provides the infrastructure to turn ad-hoc survey
    processing scripts into reproducible, shareable pipelines. Its lazy step
    pipeline (step_compute, step_recode, step_rename, step_remove,
    step_join, step_filter) operates within the context of a
    survey::svydesign/svrepdesign object, preserving sampling design
    metadata (weights, strata, PSUs, replicate weights) through every
    mutation, and is materialized via bake_steps(). The Recipe system
    bundles those pipelines into portable, versioned objects with metadata
    (survey type, edition, variable dependencies) that can be validated
    against a survey's schema before execution and applied to new editions
    of the same survey; the workflow system wraps
    svymean/svytotal/svyratio/svyby with provenance tracking, tying
    each estimate back to the recipe that produced its input data.

    Following the modularization discussed in this thread, metasurvey.core
    contains only the survey-agnostic engine: it runs fully offline. The
    registry REST client, the Shiny explorer, the ANDA download client, and
    the STATA transpiler now live in companion packages
    (metasurvey.explorer.backend, metasurvey.explorer.frontend,
    metasurvey.anda, metasurvey.fromstata), which reach core through an
    injectable backend-provider hook.

  • Who is the target audience and what are scientific applications of this package?

    Government statistical offices (INEs) and academic researchers in
    economics, demography, and the social sciences working with household
    survey microdata. The primary use case is reproducible processing of
    recurring national surveys — Uruguay's ECH, Argentina's EPH, Brazil's
    PNAD-C, Chile's CASEN, or the US CPS.

    metasurvey.core is not a package for a single survey. Packages like
    ech, eph, PNADcIBGE, or lodown are tied to one survey and
    hardcode its variable names and conventions. metasurvey.core is
    survey-agnostic infrastructure: the abstraction layer (steps, recipes,
    workflows, provenance) that works for any complex survey. What makes
    it useful for a specific survey is a Recipe contributed by the
    community, not code baked into the package — so the ecosystem scales
    through community contributions. The bundled vignettes demonstrate this
    with a ECH 2023 sample (shipped with the package, runs offline) and
    reproducible examples for seven international surveys.

  • Are there other R packages that accomplish the same thing? If so, how does yours differ?

    No single package covers the same ground; metasurvey.core integrates
    capabilities that individually overlap with existing tools:

    • survey (Lumley): the estimation engine. metasurvey.core wraps it
      and adds everything upstream — the pipeline that prepares microdata
      before svymean()/svytotal() can run.
    • srvyr: tidyverse syntax for survey objects, focused on the
      analysis phase. metasurvey.core focuses on the processing phase and is
      complementary.
    • recipes (tidymodels): general-purpose preprocessing for
      modeling. metasurvey.core's recipes are domain-specific: they carry
      survey metadata, validate against a survey's schema, and operate on
      design-aware survey objects, not bare data frames.
    • ech/eph/PNADcIBGE/lodown: survey-specific packages.
      metasurvey.core is the infrastructure layer underneath, usable with or
      without them.
    • targets: general workflow orchestration. metasurvey.core's
      workflow system is survey-specific, with design-aware estimation and
      provenance tied to recipes.
  • Any other questions or issues we should be aware of?

    This submission continues metasurvey — Reproducible Survey Data Processing with Step Pipelines #752: following the editors' concern about
    package size, the original metasurvey monolith (now archived as
    metasurvey-legacy)
    was split into a modular ecosystem, and metasurvey.core is its base.
    Compared with the monolith reviewed in the original checks, core has no
    network, Shiny, or STATA code and no httr2/xml2/shiny dependencies;
    8 Imports total. We chose R6 because survey objects are inherently stateful (accumulating steps,
    holding mutable design objects) — happy to discuss this design choice.

Technical checks

Confirm each of the following by checking the box.

Use of Generative AI

  • Generative AI tools were used to produce some of the material in this submission.

    As described in detail earlier in this thread
    (2026-02-21): the architecture and all design decisions are the
    authors'; Claude Code assisted with test boilerplate, documentation
    drafts, packaging mechanics, and the modularization grunt work. The
    wording and translation of the English vignettes were AI-assisted
    and each English vignette discloses this in a footnote; the Spanish
    vignettes, the code, the examples, and all technical content are by the
    authors. Everything AI-assisted was human-reviewed.

Publication options

  • Do you intend for this package to go on CRAN?
  • Do you intend for this package to go on Bioconductor?
  • Do you wish to submit an Applications Article about your package to Methods in Ecology and Evolution?

Code of conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions