Weights and Model Calculated in Different Samples

[akmontoya]

Hi! This is my first time using the WeightIt package, so I hope this question makes sense. I'm using two different samples for the analysis that we're doing. Sample 1 is a larger sample from which we're estimating the propensity score model, and Sample 2 is a smaller sample, where we're using propensity score weights estimated with the model fit on Sample 1 to estimate our model in Sample 2 (ATE). We have a secondary outcome that is dichotomous, so I wanted to use a logistic regression model . For the primary outcome (which is continuous) I had calculated the weights without the WeightIt package and used lm() with weights, but when I started to look at the dichotomous outcome and tried to use glm() with the weights argument, but got an error that made it clear that the weights in glm are not sampling weights.

I looked into the WeightIt package which seems great, however it seems like the only way to enter in weights is by generating a weightit object using the weightit function. But given that the weightit function is calculating weights for Sample 1 and the model I want to fit is for Sample 2, I don't really think this is going to work. Is there someway I can feed in the weights I've calculated directly into the glm_weightit() function? I understand this may be less than ideal because it will not account for the covariance structure, but this seems like my only option.

If you have any other suggestions I would be very open to it!

[ngreifer]

Hi Amanda,

This is an interesting problem. First, I'll say that the warning on the glm() help page that the weights argument is only for frequency weights (or any other specific purpose) is incorrect. glm() performs weighted maximum likelihood estimation (WMLE), which is completely agnostic to the origin or purpose of the weights. What is inaccurate are the standard errors produced by using summary() on a glm object fit with weights. This is also true of lm objects. You cannot use the standard summary() function to compute standard errors in weighted samples. To compute valid standard errors after weighting, you either need to manually bootstrap the whole process or use a robust standard error (the latter of which is only a conservative approximation). You can request robust standard errors using some other R packages. Another popular option is to pretend the PS weights are survey weights and use the survey package, which is what lots of old guides on PS weighting in R used to recommend.

If you want to use WeightIt, there is a kind of way to "hack" the functions to do what you want. Here's how I would do it:

1. Combine Sample 1 and Sample 2 into a single dataset, including a new variable (S) for sample membership. It's easiest if this variable is 1 for Sample 2 and 0 for Sample 1. If outcomes are not present in Sample 1, give them any arbitrary value (e.g., randomly generate 0s and 1s, but don't use all 0 or all 1).
2. Estimate weights in the combined dataset, ignoring S in the weighting model.
3. Fit an outcome model using glm_weightit() that is equivalent to a separate models for Sample 1 and Sample 2. The model formula should look like Y ~ 1 + S + S : (T * (X1 + X2)) where Y is the outcome, S is the sample ID variable, T is the treatment, and X1 and X2 are the covariates you're adjusting for. This fit's an intercept-only model for Sample 1 and the full outcome model for Sample 2.
4. Use g-computation to compute the treatment effect in Sample 2. You'll use something like avg_comparisons(fit, variables = "T", newdata = subset(S == 1)).

This is the same workflow you'd use if you were performing moderation analysis in a more traditional scenario. The only ways this differs is that the PS model is the same for both samples (whereas usually in moderation analysis you would allow the models to differ by subgroup) and that in the end, you are only performing g-computation for one of the subgroups instead of for both.

If you want to fully separate the two samples so that Sample 2 isn't used at all in the estimation of the PS (rather than what I described above, which pools information across both samples to estimate the PS), you could generate a new set of sampling weights which are equal to 1 for Sample 1 and 1e-8 for Sample 2 and then supply those to the s.weights argument of weightit(). You would not enter them anywhere else, though (i.e., not in the wts argument of avg_comparisons()).

To verify that this procedure works, you can generate different values to go in the Sample 1 outcomes, and you should find that the treatment effect estimate and SE for Sample 2 are invariant to this choice.

I would say that it's probably easier to just use your original method but make sure you are correctly estimating the SE after weighting. It's kind of ambiguous what sources of variability should be accounted for; you could treat the weights as fixed and Sample 2 as random (since Sample 2 wasn't used to estimate the weights), or you could consider Sample 2 as fixed and Sample 1 as random, in which case the variability would come from estimating the weights in the random Sample 1 but not from Sample 2. I don't know enough about that to make a good assessment.

[akmontoya]

Thank you! This is incredibly helpful! Much appreciated! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Want to learn more about Mediation, Moderation, and Conditional Process Analysis? Sign-up for one of my upcoming courses: Mediation, Moderation, and Conditional Process Analysis I - GSERM (June 8 - 12) <https://gserm.org/instructors/amanda-k-montoya/>, Introduction to Structural Equation Modeling - GSERM (June 15 - 19) <https://gserm.org/instructors/amanda-k-montoya/> Amanda Kay Montoya (hear name <https://namedrop.io/amandamontoya>) Associate Professor Department of Psychology, UCLA Pritzker Hall 502 Portola Plaza, Room 4530 Los Angeles, CA 90095 pronouns: she, her, hers email: ***@***.*** phone: 310-794-5069 website: akmontoya.com To request a meeting: https://doodle.com/mm/amandamontoya/schedule (M-F, 9am to 5pm PST) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

…

On Thu, Apr 23, 2026 at 11:49 AM Noah Greifer ***@***.***> wrote: *ngreifer* left a comment (ngreifer/WeightIt#90) <#90 (comment)> Hi Amanda, This is an interesting problem. First, I'll say that the warning on the glm() help page that the weights argument is only for frequency weights (or any other specific purpose) is incorrect. glm() performs weighted maximum likelihood estimation (WMLE), which is completely agnostic to the origin or purpose of the weights. What is inaccurate are the standard errors produced by using summary() on a glm object fit with weights. This is also true of lm objects. You cannot use the standard summary() function to compute standard errors in weighted samples. To compute valid standard errors after weighting, you either need to manually bootstrap the whole process or use a robust standard error (the latter of which is only a conservative approximation). You can request robust standard errors using some other R packages. Another popular option is to pretend the PS weights are survey weights and use the *survey* package, which is what lots of old guides on PS weighting in R used to recommend. If you want to use *WeightIt*, there is a kind of way to "hack" the functions to do what you want. Here's how I would do it: 1. Combine Sample 1 and Sample 2 into a single dataset, including a new variable (S) for sample membership. It's easiest if this variable is 1 for Sample 2 and 0 for Sample 1. If outcomes are not present in Sample 1, give them any arbitrary value (e.g., randomly generate 0s and 1s, but don't use all 0 or all 1). 2. Estimate weights in the combined dataset, ignoring S in the weighting model. 3. Fit an outcome model using glm_weightit() that is equivalent to a separate models for Sample 1 and Sample 2. The model formula should look like Y ~ 1 + S + S : (T * (X1 + X2)) where Y is the outcome, S is the sample ID variable, T is the treatment, and X1 and X2 are the covariates you're adjusting for. This fit's an intercept-only model for Sample 1 and the full outcome model for Sample 2. 4. Use g-computation to compute the treatment effect in Sample 2. You'll use something like avg_comparisons(fit, variables = "T", newdata = subset(S == 1)). This is the same workflow you'd use if you were performing moderation analysis in a more traditional scenario. The only ways this differs is that the PS model is the same for both samples (whereas usually in moderation analysis you would allow the models to differ by subgroup) and that in the end, you are only performing g-computation for one of the subgroups instead of for both. If you want to fully separate the two samples so that Sample 2 isn't used at all in the estimation of the PS (rather than what I described above, which pools information across both samples to estimate the PS), you could generate a new set of sampling weights which are equal to 1 for Sample 1 and 1e-8 for Sample 2 and then supply those to the s.weights argument of weightit(). You would not enter them anywhere else, though (i.e., not in the wts argument of avg_comparisons()). To verify that this procedure works, you can generate different values to go in the Sample 1 outcomes, and you should find that the treatment effect estimate and SE for Sample 2 are invariant to this choice. I would say that it's probably easier to just use your original method but make sure you are correctly estimating the SE after weighting. It's kind of ambiguous what sources of variability should be accounted for; you could treat the weights as fixed and Sample 2 as random (since Sample 2 wasn't used to estimate the weights), or you could consider Sample 2 as fixed and Sample 1 as random, in which case the variability would come from estimating the weights in the random Sample 1 but not from Sample 2. I don't know enough about that to make a good assessment. — Reply to this email directly, view it on GitHub <#90?email_source=notifications&email_token=ADBI57HTM6N7MSUW6KEH2AL4XJQTBA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMZQGY4TMOJSGAZ2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-4306969203>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADBI57CM3AOSWYVZLIMRTKT4XJQTBAVCNFSM6AAAAACYEFJ7HSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DGMBWHE3DSMRQGM> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/ADBI57BEVO4OABDDJW7DQPL4XJQTBA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMZQGY4TMOJSGAZ2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/ADBI57B6SHIMDQYJF5I32YD4XJQTBA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMZQGY4TMOJSGAZ2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>