Feature Request: Secure, blinded email handling for strict ethics compliance in longitudinal runs

[raphael303]

Problem
To my knowledge Longitudinal studies in formr currently require the run owner to manage email addresses within the R environment (via OpenCPU) to facilitate automated reminders. For researchers operating under strict ethical guidelines, this creates a trust boundary issue.

Even if a researcher uses an R script to delete or hash emails after a study concludes, they inherently possess the technical capability to link personally identifiable information (PII) to research data while the run is active. Under this circumstance it is not possible to honestly tell a participant "we CAN NOT match your data to your identity / email / phone". Thus, ethics boards often reject this setup because the run owner controls the execution environment where the data is temporarily linked. Currently, the only workaround is building an external “Data Trustee” server via webhooks, which adds significant overhead and defeats the all-in-one convenience of formr.

Feature Request
Introduce a native, “firewalled” email architecture. The goal is to allow formr to collect emails and send automated reminders based on session IDs, while making it technically impossible for the run owner (via the R sandbox) to access the plaintext email addresses or the cryptographic keys used to secure them.

Possible Solution
Implement a strict separation of concerns between the PHP backend (System Environment) and the OpenCPU sandbox (Run Environment).

1. System-Level Encryption: A master encryption key is generated and stored on the server, accessible only to the PHP backend (e.g., as an environment variable). This key must be strictly isolated from the OpenCPU R sandbox. This key is not available for the researcher downloading the data or managing the run.
2. Blind Collection and Storage: Introduce a specific “secure email” item type. When a participant submits their email, the PHP backend intercepts it, encrypts it using the master key, and stores the ciphertext in the MySQL database. If the run owner downloads the dataset or queries it in R, they only see the encrypted string.
3. Secure Mail API: Introduce a new R function for the formr package (e.g., formr_secure_mail()) to act as a secure trigger.
   • Usage: formr_secure_mail(session_id = current_session, template = “wave_2”)
   • Execution: The R environment passes this request to the PHP backend. The backend looks up the ciphertext for that session, decrypts the email in memory, hands it to the SMTP server, and immediately flushes the plaintext from memory.
   • Return: The PHP backend returns only a basic boolean (TRUE/FALSE) to the R script confirming if the email was dispatched.

This architecture ensures the R environment never handles plaintext emails, allowing researchers to credibly guarantee to participant and ethics boards that they have zero technical access to participant PII, while retaining the full power of formr’s automated run logic.

[rubenarslan]

are you aware of these settings? I see how your suggestions improve on it, just asking. The reason why I haven't gone further down this alley is of course that with the inherent customizability of formr, people could write some client-side code to intercept emails entered etc. so it's hard to make it zero-trust. the approach we have now allows for separation and prevention of accidental discovery of PII+research data links.

also, to clarify: emails are currently sent via PHP. the openCPU sandbox does not see the email address in the usual case (unless you write R code to determine which of multiple addresses to email). but it is possible for R to access emails, so I see that point.

beyond that, the new PWA functionality allows push messaging participants without collecting any identifiable information. this seems preferable to me, because email lacks immediacy anyway.

if you have an example of how an ethics board has rejected a formr study for these technical reasons, I'd be interested in the precise language.

[raphael303]

If doing an anonymous study and the researcher is at ANY point technically or theoretically capable of matching PII AND survey data, our ethics board rejects the survey as non-anonymous. We had this many times in the past. For longitudinal studies, the ONLY allowed way to do so was by collecting email addresses in a separate survey and using that to indiscriminately re-invite and re-re-invite participants (no matter if they already completed the second wave or not), and match survey responses with pseudonyms which the participants generated themselves by creating a "word" with the first letter of responses to personal questions which hopefully remain constant over the course of the study.

This is very strict here. I don't see what the "Survey Unlinking" helps here. Apart from removing a way to re-invite participants based on their progress (so I guess it only makes sense once a run is completed), it is a step which undoubtedly comes AFTER I COULD have matched their PID to their data. Furthermore, I don't see this option's point, as I could also just delete that survey's data, as without a session ID and no way to match it to anything elsewhere, it is unusable for anything anyway.

Also, IF I can at ANY point match PID with survey responses, I can never tell a participant "Researchers cannot match your responses to your email address," and if I cannot say this, it will already be a confounding variable on the answers. Even more so than normal, social desirability will play a higher role.

The Architecture: Asymmetric (Public/Private Key) Encryption
To achieve this securely without breaking formr's dynamic logic:

The server generates a system-wide Master Key-Pair (e.g., using libsodium which is native to PHP).
The Public Key is made available to the formr web application to encrypt data.
The Private Key is kept strictly secret on the server (e.g., in a .env file) and only the system's background email dispatcher (cron worker/daemon) is allowed to read it.
Required Codebase Changes
To implement this, you would need to modify roughly 4-5 core areas of the formr codebase:

1. A New Item Type: encrypted_email
   You would need to add a new survey item type (e.g., encrypted_email) to the spreadsheet definitions. When the frontend form is generated by formr, it looks like a normal email input, but it signals to the backend that this is highly sensitive.

2. Hooking the Save Mechanism (SurveyStudy.php / Item Saving logic)
   When the participant clicks "Next" and submits their survey, the PHP backend processes the variables before writing them to the database. You would add a hook intercepting the encrypted_email item: The system takes the plaintext email and encrypts it using the server's Public Key (e.g., sodium_crypto_box_seal).
   The system base64-encodes the ciphertext (e.g., ENC_xxxxBase64xxxx) and saves that string into survey_results.
   Result: When the run owner downloads the data, or looks at it in formr_results(), they exclusively see ENC_xxxxBase64xxxx.

3. Intercepting the Email Sender (application/Model/RunUnit/Email.php)
   This is the core mail logic in formr. When an Email RunUnit reaches a participant, it pulls their destination email using $this->recipient = ...->getRecipientEmail().
   At this point, the unit receives the ciphertext ENC_xxxxBase64xxxx.
   You would modify Email.php to detect the ENC_ prefix.
   Right before validating the email address and handing it off to SwiftMailer (sendNow() or queueNow()), the script uses the server's Private Key to decrypt the string back into a plaintext email address.
   The email is sent to the participant reliably.

4. Protecting the Logs
   formr tracks all sent emails in the survey_email_log database table (which owners can see in their run dashboards). You would need to edit Email.php's logging methods (logMail and queueNow functions) to ensure that if the recipient was an encrypted address, it only logs "Encrypted Address" or logs a cryptographic hash of the email address rather than exposing the decrypted plaintext in the dashboard.

[rubenarslan]

to clarify, you can enable unlinked results before collecting any data. after this is turned on, the study admin can no longer link the PII data to other data (it is randomly ordered, no user codes). however, the data can be used to send and personalize emails.

you can also disable display of results. after this is turned, data is still stored, but the study admin cannot view it. you can still use it to send emails.

did you previously understand that this is what these features do and describe it like this to your ethics board? I can't tell from your response.

[raphael303]

you can enable unlinked results before collecting any data. after this is turned on, the study admin can no longer link the PII data to other data (it is randomly ordered, no user codes). however, the data can be used to send and personalize emails.

you can also disable display of results. after this is turned, data is still stored, but the study admin cannot view it. you can still use it to send emails.

No, I was not aware that you can do it BEFORE you start the run! That changes the picture!

I have looked into how you did this now, and if I understand it correctly, the emails are still physically in the database including the session ID, but if I export it, it intentionally strips the session ID and scrambles it for me. Is that how it works? I believe that would work perfectly for our ethics department. As long as the researcher is structurally prevented from linking it at any point, it doesn't matter if the link exists underneath the hood where the researcher can't reach.

And if I understand correctly, if I use the "hide results" setting additionally, the study admin can't even download or see the unlinked emails/phones, but the automated background tasks can still use them perfectly fine?

Given that, I'd say great solution. Not how I had expected it to be done, but really smart! I guess an extremely tech-savvy board member might want to see encryption, but really, it wouldn't change anything. A system admin who compromises the server could read the plaintext database now, and under my proposed encryption solution, that same sysadmin would just steal the decryption keys from the server anyway.

[rubenarslan]

I think your proposed solution has some niceties that the current solution does not have, but I think it should prevent non-malicious linkage of PII and research data. to guard against malicious linkage, you need a third party, e.g. what we do with push messages.

I am interested in encrypting survey data for other reasons, so we might pursue something like your solution down the line, but maybe not now. and I could mask the emails in the email logs for these.

[rubenarslan]

oh and a further request for clarification: you say your ethics board rejects the survey as non-anonymous. does your ethics reject all studies that collect PII? because people can also be identified by age, sex, location etc. this is data we often want to collect and link to research data. so often time guaranteeing full anonymity is not possible anyway while preserving data usefulness. but you can get consent for it.

[raphael303]

oh and a further request for clarification: you say your ethics board rejects the survey as non-anonymous. does your ethics reject all studies that collect PII? because people can also be identified by age, sex, location etc. this is data we often want to collect and link to research data. so often time guaranteeing full anonymity is not possible anyway while preserving data usefulness. but you can get consent for it.

No, they are fine with gathering demographics if consent is given AND definitive identification is still impossible, but they do not allow for any definitive identification through an email, phone number, address, social security number, or any such things. Except of course if it is not an anonymous survey in the first place.

[rubenarslan]

I just think 'impossible to re-identify in theory' is a very high bar with any useful amount of demographic data. is this the same legal basis as GDPR or does switzerland have a substantially different law around this?

[rubenarslan]

btw your suggestion for an implementation is sound. it's not a priority for me or any paying users atm, but you could submit a PR?