Extensible primitives

[jpco]

I have been thinking a lot about the idea of extensible primitives lately. The idea, which to be clear isn't mine, is something like:

What if the set of primitives in es were extensible, like functions are today?

There are a lot of different ways this could look, and depending on how it's done a lot of different use cases it could enable. What I am about to describe here is mostly daydreaming with the hope of establishing a common notion of a long-term direction for es.

The simplest version of this idea would just be a change to the es build process, allowing people to easily specify additional primitive-defining files at compile time in order to include novel behaviors into the shell. This could be used for line-editing libraries like $&readline, additional convenience primitives like $&getcwd or $&getpid, or math primitives.

Dynamically loadable primitives would enable separate compilation of primitives and the shell runtime, and allow users of an already-built es to extend the shell to their preference. Separate compilation would also allow primitives to be written to a different C standard or in a different programming language entirely.

A namespacing and/or versioning mechanism could make it easier to create not just additional primitives, but also alternative primitives. This could be used for a number of things: primitives with specialized implementations for the APIs of certain OSes (e.g., using a potential namespaced-primitive syntax I'm not sure I like, $&linux/fork), user-contributed alternate versions of primitives (a $&jpco/pipe which has lastpipe-like behavior), or primitives keyed by es version, allowing users to "upgrade" or "downgrade" primitives at will (e.g., $&es-0.9.3/time instead of $&es-0.10.0/time). This last use case would probably also require moving the existing primitives out of "the core runtime" and into a newly-formally-defined "shell primitive layer".

Taking this idea far enough begins to move the shell into the "librarified" territory the authors were considering long ago. The es build process could involve creating an es binary which includes little more than the core runtime and a $&dumpinitial primitive performing the behavior currently in dump.c, and then the final shell would be a second es binary which on startup runs the $&loadinitial primitive that $&dumpinitial had produced. This same sort of bootstrapping could produce an es binary doesn't use any of the shell primitives at all, but instead just has primitives useful for running a simple web server, or a window manager, or a debugger, or whatever else; at a certain point, es stops being a shell and starts actually being a flexible Scheme-based command language, which can happen to be configured as a shell.

---

So, clearly, this is a large area to explore, which would require answering a huge pile of design questions, and some of the more "sci-fi" ideas would involve lots of other changes in the shell runtime to support flexibility that simply doesn't exist today.

What I would propose doing right now, though, is just the most modest version, in order to make it easier to include or exclude certain primitive-defining files at build time. What I would want to make possible is a new line in the Makefile that looks like

PRIMFILES = readline.c $(ADDPRIMFILES)

This would let people control which primitives are in the shell via the normal make-configuring mechanisms. In addition, and I won't push for this immediately, I think all of the existing primitive files except for access.c and proc.c (which both have code used by parts of the "core" shell runtime) could be changed to be PRIMFILES-based right away. (access.c could be moved if #241 were merged; proc.c would be trickier.)

On top of this basic mechanism, I would like to create a place in the repo where new primitive-defining files can be put, like a contrib/ directory. I am picturing that code in this directory would be built into the shell on an experimental, opt-in basis, and therefore could be more freely contributed without the same level of concern around backwards compatibility or whether it violates the principle of minimalism in es.

My hope is that creating this build-time flexibility along with a "safe" place to contribute code could help bridge the gap between all the contributions that have been locked away in peoples' personal (and often abandoned) forks of es and the much smaller amount of code that manages to get upstreamed and shared with others.