Improve workflow with master components

[merryman]

This is more a note to myself, but please feel free to add in ideas.

I have observed that the current state of master components still has a bunch of caveats to it and would like to cover some possible features we can implement to alleviate various issues.

I already touched upon an issue that is related to a missing nested component feature in #223 and will not cover this here.
Rather this is more about improving the workflow with master components for developers who are more "code drive", which is to say, "prefer to have everything in human readable source code".

Async Hell
The first issue I would like to cover is the asynchronous nature that master components introduce to the code.
When for instance a designer has crafted a master component for us we as a developer would like to utilize for our tool,
we can not make any assumptions about the submorphs and properties being properly initialized until the master component has been loaded successfully:

this.master = 'styleguide://someWorld/path/to/master'
await this.master.whenApplied();
// now we can proceed and assume that everything is the way its *supposed* to be

This, I think one can say quite reasonably, sucks and forces asynchronous code where it really should not be.
How could we remove the asynchronous nature? Well actually quite simply. Assuming the master component was already resolved somehow previously, we can assume it can synchronously apply itself to the morph it is assigned to since the only reason we are asynchronous is that master components are potentially fetched from files:

// some time earlier
const masterComponent = await resource('styleguide://someWorld/path/to/master').read()

// some time somewhere later
this.master = masterComponent;
// by here the master component has already applied itself fully and initialised all submorphs

So it all boils down to those nasty files required to be fetched. But wait, wasnt there another thing that comes from files all the time? That's right, modules do. But why then do they not introduce the asyncronous problem all over the place?
Well for one at some point they do, which is when we utilize dynamic imports:

await System.import("lively.ide/code-search.js");

Yet most of the time, the modules are declared as imports, which informs the module system to only run the module fully once all of the depending modules have been loaded:

import A from "moduleA";
import { B } from "moduleB"

This essentially hides the asyncronous evaluation from the module scope and moves it into the module system itself removing any kind of asyncronous code from our module. Nice! So can we do the same with master components?
In principle we can! lively.modules allows us to almost arbitrarily instrument our module system, and we can very well make it support the following without much issues:

import masterComponentA from "styleguide://someWorld/path/to/master"

Once we then extend the freezer to properly resolve said imports, we should be able alleviate the asynchronous master component code from the codebase once and for all.

Cyclical Graph Hell
Right now the workflow for master components essentially forces developers to define parts in worlds, which for many, especially non design folks, constitutes a shift to a different mode of working, which is somewhat inconvenient. (I will not go over existing bugs which arise, only the conceptual issues with the approach). Furthermore, we still have the issue that platforms such as github etc. will not allow us to really create meaningful changesets and pullrequests between worlds, since they constitute graphs and not human readable data formats (whatever that really is).

Now @rksm and I have spent a great deal thinking about this issue in the past, and it always comes back to the simple answer that in order to embrace a more what I would call coder and platform friendly approach we need to move to a declarative, ideally simplified representation of morphs or in this case components. In an optimal world we would also like to achieve said feat without abandoning all the nice things we get from working in worlds, which is predominantly direct manipulation and the ability to craft interfaces visually rather than in code.

Well let's first note the fact, that right now nothing prevents the user to just define master components inside modules instead of worlds and use them directly:

const masterA = morph({
  isComponent: true,
  name: "button/red",
  fill: Color.red,
  borderRadius: 5,
  submorphs: [{ name: "button label", type: 'label', fontSize: 14, value: "Hello World!" }]
});

// and then just use or export this component

export { masterA }; // import and use in other modules

aMorph.master = masterA; // directly assign as master

// use in custom morph definitions
class SomeButton extends Morph {
   static get propeties() {
     return {
         master: {
             initialize() { this.master = masterA; }
         },
         submorphs: {
             initialize() {
                 this.master.reconcileSubmorphs(); // returns syncronously!
             }
         }
     }
   }
}

I think for people who work from code, it is quite obvious what benefits we get from resorting to this definition of master components: We keep the nice separation of appearance from code, remaining nicely modular. Further the components are now formidably suited to be managed in "classical" text based version tools and platforms a la github. However we loose the ability to manipulate master components inside modules via halos or other primarily visual tools. This is quite unfortunate since for one it excludes people who don't know how to code and even worse: It is just a plain old nightmare to design good interfaces from a code first approach. Freely drawing or manipulating elements on the famous "blank canvas" is I think crucial for getting to high quality and well working designs as fast as possible. The reason the tools designers prefer to use are so heavily visual is in fact not, as I sometimes believe a lot of programmers like to believe, "because they cant code" but instead due the complete inadequacy of the code first approach for this part of application development.

I think this is the perfect opportunity to revive a Lively idea from a couple years back, were we came up with mapping direct manipulation to source code transformations. One of the main problems Transmorphic faced back in the days, was to reconcile properly with abstractions such as functions calls, loops or conditionals. This is now more or less gone, since these code bits are usually never part of a master component definition. A master component is by design something that is separate from accompanying behaviour. While we can choose to ship a component with custom behaviour, we are not required to and in fact the way code based master components definitions work, we do not really need to worry about messing up abstractions of any kind. In fact master components should not ship with any custom behavior at all. This should be the role of stateful parts that live in worlds and serialize their state when they are supposed to be stored away. For more on that topic see one of the posts below about state junkies vs code people.

Concretely I am envisioning an alternative "view" provided by the system browser, where a module that defines master components can be viewed as a canvas upon which all the defined master components are rendered and displayed as morphs and can in turn be directly manipulated. Those changes can then be directly mapped back to the changes in the code of the module. This should be exceedingly straight forward due to the already existing AST support in lively which allows us to work with code as data and conveniently resolve master components to source code ranges and changes.

I will refine this idea a bit more in the near future and provide some more examples before we flesh out the details for a concrete implementation.

[merryman]

Update with some more concrete implementation details:

Definition of file based master component would happen inside .cp.js files which can then be interpreted in a custom way by the system browser.
These files consist of exported variables that each denotes a master component definition like so:

export const masterA = morph({
  isComponent: true,
  name: "button/red",
  fill: Color.red,
  borderRadius: 5,
  submorphs: [{ name: "button label", type: 'label', fontSize: 14, value: "Hello World!" }]
});

export const masterB = morph({
  isComponent: true,
  name: "button/green",
  fill: Color.green,
  borderRadius: 0,
  submorphs: [{ name: "button label", type: 'label', fontSize: 14, value: "Hello World!" }]
});

...

For these files, the system browser would in turn display either a symbolic representation (i.e. code) or alternatively a visual representation that supports direct manipulation.
The browser would ensure that visual and symbolic representation are kept in sync. Further the instances of the master components need to be kept consistent in the system. That is, saving a change to the code or manipulating a component should not lead to a detachment of a master component from existing derived morphs.

It is also important that the browser ensures that only "vanilla" morphs are used in a component definition. Utilising classes that later on reference the master components themselves within the master components, leads to component definitions that work fine during development in lively, but fail to get bundled successfully since these cyclical references can not be split up by the bundler.

[merryman]

Braindump: this approach can also be used to reduce the overall "serialized blob" of a world and instead split it up into a folder structure where the package code and master components are stored in modules while the "world State" constitutes a work in Progress that is Not intended to be shared with other worlds(projects). This would 1.) remove the need for a separate component cache/index that needs to be updated. 2.) make the "juicy" part of worlds be easily managed via git since it's just another directory and the spec based component definitions can be diffed in a meaningful way.

[merryman]

On State Junkies and Code People.

This is somewhat related to a thing that @rickmcgeer mentioned a while back in #319. I call the approach he mentions in that issue the State Junkie approach. This kind of developer likes to start off with already prepared morphs and quickly slap behavior onto them to quickly get results. This is maximum start to finish implementation of a fully working GUI and especially killer for quick prototyping or small little apps. Unfortunately this approach quickly loses its steam when we reach larger scale applications with 10 > Formulars and a complex domain model that needs to be managed. This is where what I like to call the code driven approach works best. This means that we retreat to describing our App in a more descriptive manner where we cleanly separate model from view, etc… However we loose the appealing drag and drop creation of UIs and the convenient incremental process from the state junkies. This is where this new master component approach can bridge a gap:
In this way we defer the state-management business which can easily become complex to the code drive people who write their abstractions away from classes/functions and what not and just reference the master components that have been crafted by drag and drop (WYSIWYG folks). Meanwhile the State junkies who enjoy tinkering and building stuff from building blocks can equally utilize the components. For the State junkies there will in particular be some tool support that allows them to dynamically apply different master components or quickly scaffold a submorph hierarchy based off a master component. I can even imagine both development workflows to work in tandem. Here the junkie approach would serve as the exploration phase. After that domain is explored to a satisfying extent we can transition over to the planning/code driven phase but still utilize the master components who had been crafted in the beginning.

[merryman]

Braindump:

As of now the development of an app can be devided into three stages: Design, Tinkering and Planning.
One problem that comes from transitioning from the components in files and without customized behavior (i.e. Design -> Planning) approach is that we need to replicate the morph structure still in the planner world, to define where our custom UI logic should go in the hierarchy that is described/scaffolded by the master components. This annoying duplication can be alleviated by a simple VMMV separation, where the morphs automatically dispatch events into a viewmodel object that can be dynamically bound via a submorph name scheme. We then do not have to attach our logic into morph classes. The viewmodel can be described fully in the planner world. While going from Components to tinkering (Design -> Tinkering) with parts is easy, we can provide a automatic conversion that translates "spontaneously created subclasses" in a tinker session into the aforementioned viewmodel architecture (Tinkering -> Planning). This is basically a one way street but process wise a fairly useful one. Tinker mode world allows us to experiment and mix and match. Once we are content, we can transition automatically to planner world where we can refine our implementation and make it more modular and decoupled. The model separation makes it somewhat harder to tear apart apps from the planner world but the result of a lot of stuff in planner world can be essentially widgets which then can be treated as an atomic building block to tinker with on a higher level.