Bech32 embeds#1078
Conversation
|
I like this. It could solve the |
|
So an eyeglasses payload from an electronic medical record looks like this: {
"resourceType": "VisionPrescription",
"status": "active",
"created": "2014-06-15",
"patient": {
"reference": "Patient/example"
},
"dateWritten": "2014-06-15",
"prescriber": {
"reference": "Practitioner/example"
},
"lensSpecification": [
{
"eye": "right",
"sphere": -2,
"prism": [
{
"amount": 0.5,
"base": "down"
}
],
"add": 2
},
{
"eye": "left",
"sphere": -1,
"cylinder": -0.5,
"axis": 180,
"prism": [
{
"amount": 0.5,
"base": "up"
}
],
"add": 2
}
]
}It is signed as a {
"id": "73a5ace1c95de239ea355a745abfafee39d4dc613081a5c3dc25158c9385620d",
"pubkey": "bcd4715cc34f98dce7b52fddaf1d826e5ce0263479b7e110a5bd3c3789486ca8",
"created_at": 1708705561,
"kind": 82,
"tags": [],
"content": "{\"resourceType\":\"VisionPrescription\",\"status\":\"active\",\"created\":\"2014-06-15\",\"patient\":{\"reference\":\"Patient/example\"},\"dateWritten\":\"2014-06-15\",\"prescriber\":{\"reference\":\"Practitioner/example\"},\"lensSpecification\":[{\"eye\":\"right\",\"sphere\":-2,\"prism\":[{\"amount\":0.5,\"base\":\"down\"}],\"add\":2},{\"eye\":\"left\",\"sphere\":-1,\"cylinder\":-0.5,\"axis\":180,\"prism\":[{\"amount\":0.5,\"base\":\"up\"}],\"add\":2}]}",
"sig": "e87f7c360a1b01fc436b081037eb11b5e0c23714907427c5e124611a3a004d3c13841d965842766dd85a0a3eaa6c14e053764f67848e892f3a4776ce5046b63f"
}which creates the following bech32:
Agree. But I didn't want to make a whole new serialization scheme for events. |
|
Funny thing... if we start using compression, we could gzip the JSON-stringified event and the final re-encoded bech32 becomes smaller than the original event. The full JSON of the signed event is 802 chars Compression of the signed event JSON string is 474 bytes As a comparison, a regular nevent with relays in it is this big: |
These are just ideas: |
|
I just want to call attention to this embed spec since we are starting to seriously use it to transfer health data inside NIP-17 DMs. |
# Conflicts: # 19.md
|
Why not just shove the JSON in instead of encoding it to bech32? We can envelope it in some wrapper to make the purpose clear, like |
|
It's just harder to parse given that the json can have multiple encodings, spaces and so on. Parsers would need to find/count the Plus, the nembed is GZIP compressed, which actually saves A LOT of bytes, given how verbose JSONs are. |
|
Plus 2: It allows us to build a nice URI for the event, which can be used directly inside QRs/NFC tags for instance. |
But then we make gzip a hard requirement? I don't like it. Also zstd is better.
What if we ensured the full event was all in one line and nothing else? |
We can separate to another NIP. nembeds should be optional to the folks that need it. It doesn't need to be mandatory.
We can use zstd. I just went for GZIP to make sure all languages have easy access to a stable lib. And since most relays do offer gzip compression for communications, it seemed to be a natural fit.
It doesn't make much sense. Our parsers already have to deal with large bech32 uris for ln, cashu, nevents, naddr... This to me would just be another line in that parser. |
I meant a hard requirement for implementing this NIP. It feels useful to have events mentioned in the context of texts, but first encoding them into a binary format, then compressing, then converting the resulting bytes into ascii feels like a very unnecessary sequence of steps.
LN and Cashu are not supported everywhere, and they don't really have to be. If you have an extra feature that turns LN invoices into a clickable widget in your client that is a nice extra, but LN invoices can be understood without that, they can be copy-pasted into an LN wallet and LN users know how to do that. Same for Cashu. The nevents and naddrs are more well-supported, but even when they're not they can still be copy-pasted into any other Nostr client. |
|
Having bech32 entities inside events was an error already, but it was bound to happen as people copy-pasted stuff so it was good that we embraced it. But this is unnecessary. It's better to follow the standard of kind 6 reposts. |
That's the goal here too. If clients don't support embedding, people should be able to copy and paste the URI into the client that does support it. Copy pasting a json will be a mess (because of the spaces and so on...)
Why? I actually think we got it right with bech32 URIs. The URI format is very helpful for both parsers and people. The only problem is that it should have been base64 instead. :) |
|
How about we extend |
Without compression, it would be 2-3 times bigger. Otherwise, it works as well. Doing I am not sure what reusing the nevent is adding. I think people prefer to not overload things. |
|
Why 3 times bigger? It would be smaller than the event JSON.
It provides an immediate fallback for clients that don't support it, they will try to load the event from a relay.
We are already overloading bech32 when we could just reuse the JSON format...
Just |
Tags and content don't change. You are still encoding strings in the TLV. And the strings are the biggest component of the event (especially for health payloads)
There won't be a relay to load it from. The event is embedded inside a GiftWrap, it's not available on a relay. The goal is exactly to avoid pinging a relay.
Yep, but why doing all this work if the JSON already does it for you? |
You're set on the need for compressing the event. I don't think that should be a requirement. Yes, compressing reduces the size of the event, doesn't mean we should do it all the time. It makes no sense to compress just these embedded events and just this time, sounds very ad-hoc. Also events are already compressed on the WebSocket layer if that is supported.
In the specific use case you have in mind. But should we standardize this feature just for that?
Unrelated question, but why not put it in a relay?
Exactly, let's just embed the JSON. I'm glad you finally agree. |
|
It seems to me that what Vitor is defining does not require standardization in a NIP. Because they are his health records inside of GiftWraps never on a relay. And there is no nostr client that cares about these prescriptions except for his. |
I started this NIP without compressions. I was convinced by people in this thread to reduce the size of the URI to facilitate view and the copy/paste of the event. Since I lived through both cases, I now do think there are significant savings in compressing the data before creating a URI. It is just better. We might not want to do it, but it is undeniably better IMO. All NIP-19 uris should've been compressed, IMO. But it's too late for that.
We can do embedding with everything. I don't think this is a niche feature. Quote posts, for instance, could embed the full event instead of just the IDs to make sure the inner events don't disappear when they inevitably get deleted from relays.
Certain events must stay so private relays don't even know about them. That's why we created gift wraps. |
Agree for now, but I am sure people will start to care when clients can't display DMs with prescriptions in them. So, I am making an effort to discuss this before that happens. |
|
For my project, I have implemented a nembed functions for creating and parsing that accept and return any json object. I like this format better than embedded stringified json and plan to use it in my implementation. Beow, is what is it looks like for an eye prescription, encoded and unencoded below. I don't really gain any space advantage (it looks bigger than the json). I don't think that really matters when I am putting into the .content field. The big advantage is that it is easily human copy-and-pastable and Qr-code friendly. This example is a bit too long for a QR code, but at least it doesn't choke - it just truncates.
|
|
Keep in mind that if the embed doesn't have the doctor signature, when the user copy and pastes the URI, the signature of the doctor is lost. Meaning that if they receive their prescription on a DM, they cannot forward it to an optician. That's why we putted the FHIR data inside the KIND 82 first, then compressed it with gzip and only then generated the nembed URI. In that way, if the user wants, they can copy-paste the URI and send it to anyone else with the doctor's signature in it. |
|
I didn't count, but it looks like that nembed is longer than the data it allegedly compresses. |
Yeah, it's about double the size of what it purportedly compresses. So not great at compression... |
I created a kind 82 event that is signed with the eye prescription embedded. Uncompressed it is 3018 characters in length; compress (using gzip) it is 1234 characters in length. Much better! |
|
My Python code for doing the compressed nembeds, using gzip. I pass in the signed event as a json dict def create_nembed_compressed(json_obj):
buffer = io.BytesIO()
encoded_data = []
if type(json_obj) != dict:
raise ValueError("not a json object")
json_obj_str = json.dumps(json_obj)
with gzip.GzipFile(fileobj=buffer, mode="wb") as gz:
gz.write(json_obj_str.encode())
json_bytes = buffer.getvalue()
encoded_data.extend(json_bytes) # Public key bytes
converted_data = convertbits(encoded_data, 8, 5, True)
return bech32_encode("nembed",converted_data )
def parse_nembed_compressed(encoded_string):
# Decode the Bech32 string
hrp, data = bech32_decode(encoded_string)
# print(f"hrp {hrp} data {data}")
if hrp not in {"nembed"} or data is None:
raise ValueError("Invalid Bech32 string or unsupported prefix")
# Convert 5-bit data to 8-bit for processing
decoded_data = bytes(convertbits(data, 5, 8, False))
# this is gzipped data
buffer = io.BytesIO(decoded_data)
with gzip.GzipFile(fileobj=buffer, mode="rb") as gz:
decompressed_data = gz.read()
try:
json_obj = json.loads(decompressed_data.decode())
except:
json_obj = {}
return json_obj |
Yeah, that sounds about right. JSONs are VERY sparse. They compress pretty well. Give me a nembed with this new method. I can add as a test case on Amethyst. :) |
Here we go. You can see my code in a previous comment
|
|
Working great here! :) |
Super. I am going to go with the compressed approach for nembeds. We should propose this for the Bech32 encoded NIP? |
|
Here is an example of an immunization record in a signed kind 82 event. 1500 characters in length. Very manageable.
|
6 participants
Creates a
nembedbech32 encoding to carry JSON-stringified events as Nostr URIs and facilitate the loading of those events by Clients without having to ping a relay.This is particularly helpful for exporting existing/pre-signed health information payloads from corporate/private relays and send to users who might not have direct access to them. Health providers are used to including these payloads inside private communications with patients and adding information around them for better context.
A use case would be the following encrypted DM:
PS: I am not attached to the
embedname. Suggestions are welcome.