Skip to content

Complete deidentify, tidy, and clean pipeline steps #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
aleortiz-uchicago wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Complete deidentify, tidy, and clean pipeline steps #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion code/2-deidentify.do
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
**# 1 Identify columns with PII
**------------------------------------------------------------------------------

local pii_vars
local pii_vars devicephonenum gps child_name_1 child_name_2 child_name_3

**------------------------------------------------------------------------------
**# 2 Save crosswalk
Expand Down
36 changes: 30 additions & 6 deletions code/3-tidy.do
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,13 @@
- unique IDs before and after
- expected observation counts
- documentation of what each table now is
One household (key uuid:11cda278-5ad1-4913-b0b6-957cdc40f500)
lists children in hh_children but has no valid child-level data
for them; it is excluded only from the expected-count check
below, not from the data itself.
hh_children (a household self-report) and the actual child
roster do NOT match: hh_children sums to 1631 children, but the
roster holds 1629. One household (key uuid:11cda278-5ad1-4913-
b0b6-957cdc40f500) reports 2 children in hh_children yet fills
no child slots. The child table's row count is driven by the
roster, never by hh_children; the discrepancy is a data-quality
issue preserved for the HFC stage, not reconciled here.

hh_children is household-level
information, so it is dropped before the reshape and lives only
Expand All @@ -38,13 +41,20 @@
**# 1 Identify units of observation present
**------------------------------------------------------------------------------

* Two units of observation are mixed in this wide file:
* - household level: one row per submission (key)
* - child level: child_age_*, diarrhea_2d_*, diarrhea_7d_* hold up to
* 3 children per household row
* Split them into one tidy table per unit of observation.

**------------------------------------------------------------------------------
**# 2 Household-level information
**------------------------------------------------------------------------------

preserve

* Drop the child-level columns; keep one row per household
drop child_age_* diarrhea_2d_* diarrhea_7d_*

iesave "${data_tidy}/household-tidy.dta", ///
idvars(key) ///
Expand All @@ -60,18 +70,32 @@

**## 3.1 Check expected number of obs
**------------------------------------------------------------------------------


* Expected child rows = filled roster slots (a non-missing child age),
* NOT the self-reported hh_children -- the two disagree and reconciling
* them is out of scope (see header Notes).
egen slot_filled = rownonmiss(child_age_1 child_age_2 child_age_3)
su slot_filled, meanonly
local n_children = r(sum)
drop slot_filled


**## 3.2 Keep relevant variables
**------------------------------------------------------------------------------

keep key child_age_* diarrhea_2d_* diarrhea_7d_*


**## 3.3 Change the relationship between rows and columns
**------------------------------------------------------------------------------

reshape long /*[ variables here ]*/, i(key) j(child_index)
reshape long child_age_ diarrhea_2d_ diarrhea_7d_, i(key) j(child_index)
rename *_ *

* reshape emits 3 child slots per household; keep only filled roster
* slots, identified by a non-missing child age
drop if missing(child_age)

**## 3.4 Confirm the ID and number of observations
**------------------------------------------------------------------------------

Expand Down
46 changes: 40 additions & 6 deletions code/4-clean-household.do
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
Inputs: data/tidy/household-tidy.dta

Outputs: data/clean/household-clean.dta (id: key)
data/raw/household.md (iesave report)
data/clean/household-clean.md (iesave report)
documentation/data-dictionaries/household-clean.xlsx
(iecodebook mini-codebook)

Expand All @@ -24,10 +24,12 @@

Remember the one rule: representation changes, values don't.

Notes: -666/-888/-999 are this survey's sentinel codes for "Don't
know"/"Refused"/"Other"; they are recoded to Stata's extended
missing values (.o/.r/.k) rather than dropped, so they stay
analyzable as missing without being mistaken for real answers
Notes: -666/-888/-999 are this survey's sentinel codes. Per the
SurveyCTO form's "choices" sheet: -666 = "Other", -888 =
"Declined to answer" (refused), -999 = "Don't know". They are
recoded to Stata's extended missing values (.o/.r/.d) rather
than dropped, so they stay analyzable as missing without being
mistaken for real answers

Section 6 variable labels use the question wording and section/
question codes (e.g. C6) from the SurveyCTO form in
Expand All @@ -54,6 +56,15 @@

local date_vars submissiondate starttime endtime

* SCTO exports these as strings like "7/12/26 4:40"; convert to Stata
* datetime (%tc). Two-digit year is read as 20YY.
foreach dvar of local date_vars {
gen double `dvar'_dt = clock(`dvar', "MD20Yhm")
format `dvar'_dt %tc
drop `dvar'
rename `dvar'_dt `dvar'
}

**## 1.4 GPS (requires identified data)
**------------------------------------------------------------------------------

Expand All @@ -76,9 +87,15 @@
**# 3 Replace missing codes
**------------------------------------------------------------------------------

local num_vars consent resp_age resp_sex resp_hh_head resp_educ hh_size ///
hh_children hh_watersource stored_yn stored_container stored_covered ///
stored_clean storage_time stored_chlorine treat_chlorine treat_boil ///
treat_notablets water_safety water_satisfaction

* Sentinel codes from the form: -666 Other, -888 Refused, -999 Don't know
recode `num_vars' (-666 = .o) ///
(-888 = .r) ///
(-999 = .k)
(-999 = .d)

**------------------------------------------------------------------------------
**# 4 Label categories (extracted from survey form)
Expand Down Expand Up @@ -125,6 +142,15 @@
add
}

* Attach value labels to their variables
lab val `dummy_vars' yesno
lab val resp_sex sex
lab val resp_educ educ
lab val hh_watersource source
lab val stored_container container
lab val water_safety safe
lab val water_satisfaction satisfied

**------------------------------------------------------------------------------
**# 5 Recategorize other values
**------------------------------------------------------------------------------
Expand Down Expand Up @@ -196,4 +222,12 @@
**# 7 Save and export codebook
**------------------------------------------------------------------------------

iesave "${data_clean}/household-clean.dta", ///
idvars(key) ///
version(14) ///
replace userinfo ///
report(path("${data_clean}/household-clean.md") replace)

iecodebook export using "${doc}/data-dictionaries/household-clean.xlsx", replace

********************************************************************************
8 changes: 6 additions & 2 deletions main.do
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,9 @@ FOR THIS TEMPLATE TO WORK CORRECTLY, EDIT THE FILE PATHS IN SECTION 2 TO MATCH Y
global box "C:/Users/luizaandrade/Box/project-folder"
global github "C:/Users/luizaandrade/GitHub/dil-template-repo"
}
else if c(username) == "username" {
else if c(username) == "Alejandro Ortiz" {
global box "C:/Users/username/Box/project-folder"
global github "C:/Users/username/GitHub/dil-template-repo"
global github "C:/Users/Alejandro Ortiz/Git Projects/Others/welcomeweek/a-ww-datatrack2026"
}
Comment on lines +29 to 32

global code "${github}/code"
Expand All @@ -37,6 +37,10 @@ FOR THIS TEMPLATE TO WORK CORRECTLY, EDIT THE FILE PATHS IN SECTION 2 TO MATCH Y
global doc_box "${box}/documentation"
global doc_git "${github}/documentation"
global output "${github}/output"
global data_raw "${github}/data/raw"
global data_tidy "${github}/data/tidy"
global data_clean "${github}/data/clean"
global doc "${github}/documentation"

/*------------------------------------------------------------------------------
3 Initial settings
Expand Down