http and gzip spec proposal

[shadders]

This is a basic standard proposal for encoding http element as a two data chunks, . The headers are JSON encoded but otherwise identical HTTP headers. This allows content-type, encoding and other meta data to be described so that OP_RETURN viewing tools can easily determine how to display the contained data. This enables html, js, css, json, images and other binary files to be easily represented.

An addition spec for defining data elements as GZIP encoded is also included. This can extend the HTTP protocol to indicate that headers or body (or both) are gzip encoded. HTTP can also specify this using HTTP headers but there is no way to do this for the header without making the HTTP protocol itself more complex. The GZIP protocol neatly overlays it and is easily applicable to any other protocol that uses framed data elements.

[shade]

Awesome!

[bakketun]

The headers could instead be encoded using pairs of pushdata items. End of headers is signalled by an empty key name (OP_0).

This is much simpler to parse than JSON. Implementing a JSON parser is not as simple as it might look, there are a lot of subtle details. Speaking of experience.

[rt121212121]

I am not sure worrying about implementing a JSON parser is necessary. Every modern/popular language or programming language has a parser already.

[ryanxcharles]

HTTP:

I love the idea of having a standard way to put HTTP on the blockchain. However, why make it any different at all from HTTP? Can't it just be one giant file with headers in text format followed by the html? If it were exactly the same as HTTP, that will probably make it easier to use existing libraries to parse it. By changing it even by a bit, we now have to write new code to parse it rather than just using a library.

A simpler way to do http:

The 4-byte protocol specifier is "http"

Followed by a single pushdata containing the headers in text format followed by the html in text format.

gzip:

I don't know enough about gzip to comment intelligently, but it seems like a great idea to have a standard way to do gzip in opreturn.

[rt121212121]

Followed by a single pushdata containing the headers in text format followed by the html in text format.

It's been a long time since I wrote a HTTP parser, but isn't the format of the data that follows the headers determined by the content type in the headers?

[shadders]

HTTP:

I love the idea of having a standard way to put HTTP on the blockchain. However, why make it any different at all from HTTP? Can't it just be one giant file with headers in text format followed by the html? If it were exactly the same as HTTP, that will probably make it easier to use existing libraries to parse it. By changing it even by a bit, we now have to write new code to parse it rather than just using a library.

The main reasoning for separating headers and body is that I presume given the storage cost space may be considered a premium. Perhaps for a single page it's negligible but if you want to host hundreds of thousands of small pages it can become non-trivial. HTTP headers are not compressed even if the body is. This gives a way to compress the headers as well. For debug purposes though you may want the option to leave headers uncompressed whilst compressing the body.

There will always some kind of intermediary between the blockchain and the webserver. You don't always have the option of setting headers in web server frameworks as a plain string in the format it appears in HTTP. Usually you set them like a map: response.setHeader("Content-Type", "text/html"). If you do want to reconstitute the full http response it's easy for the intermediary: String response = headers + body.

As to the use of JSON. I'm open to leaving it as line delimited to match the exact HTTP format. It adds an additional parsing burden to the intermediary though. Again in most web client frameworks I've worked with by the time you actually get to access the headers they've already been parsed into a map:
boolean isHtml = response.getHeader("Content-Type") == "text/html"

[bakketun]

I second @ryanxcharles suggestion to just include the headers in http syntax and suggesting to simplify it even more by including the status line and dropping the "http" pushdata prefix.

A single pushdata starting with 485454502f312e3120 ("HTTP/1.1" followed by a space) is a complete http response message.

[shadders]

As I think about it I think we might conflating concerns here. HTTP is actually a TRANSFER protocol not a storage protocol. The reason I was thinking about http headers was because I thought some meta data about the content type might be useful for viewer site like alice.bitdb.network and pagereturn.com

But if you think about it most of that information is generated in real time by the webserver based on the file extension it's serving. I'm starting to think all we need is a file storage mechanism. I like Ryan's proposal but I would add an optional <metadata> field between the <filename> and the <data> elements. If you don't want metadata just insert an 0x00 byte to indicate the metadata is null.

[bakketun]

Ok, I think I better understand what the goal is.

Object with metadata in OP_RETURN

Prefix: TBD (use Bitcom)

Format

OP_RETURN followed by three pushdata elements.

prefix metadata data

Optional gzip compression

The metadata and data elements can be optionally compress using gzip as defined rfc1952. If the element starts with 0x1f, 0x8b, try decompressing it. If decompression fails, assume it's uncompressed.

Note: the gzip prefix is invalid UTF-8

Metadata

The metadata object is a list of key, value pairs encoded using script pushdata format.

key-length key-data value-length value-data

The field names and values follows the semantics of HTTP.

Only header types that express metadata information of the object should be used, i.g. Content-Type and Last-Modified. HTTP protocol related header should not be used and ignored if present, e.g.

• Content-Transfer-Encoding
• Date
• Server
• Accept-Ranges
• Vary

[shadders]

Closing this PR in favour of this one: #5

The HTTP protocol doesn't make sense as it's a protocol for transfer not storage. It is replaced by the simpler 'file' and 'data' protocols.

'meta' and 'gzip' are provided as two different markup options for 'file' and 'data'.

For reference this is the longer explanation for why I'm closing this PR:

As I think about it I think we might conflating concerns here. HTTP is actually a TRANSFER protocol not a storage protocol. The reason I was thinking about http headers was because I thought some meta data about the content type might be useful for viewer site like alice.bitdb.network and pagereturn.com

But if you think about it most of that information is generated in real time by the webserver based on the file extension it's serving. I'm starting to think all we need is a file storage mechanism. I like Ryan's proposal but I would add an optional <metadata> field between the <filename> and the <data> elements. If you don't want metadata just insert an 0x00 byte to indicate the metadata is null.