unification of extensions apis

[dragoncoder047]

My thinking about a better streams api (#64), opaque objects (#70), and also more than 1 extension support using metatables, I was thinking that these different aspects could all be stored in lookup tables in flash just like the core uLisp function lookup table. Something like this for streams (like in this comment of mine but with static lookup tables):

typedef struct {
  PGM_P name;
  pfun_t pfun;
  gfun_t gfun;
  // more fields as needed
} stream_entry_t;

const char serialstream[] PROGMEM = "serial";
// etc
const stream_entry_t streams_table[] PROGMEM = {
  { serialstream, pserial, gserial /* etc */},
  // etc
};

And then gstreamfun() and pstreamfun() could use the lookup table in the same way as the main function lookup routines. This might even reduce the code size in the long run compared to two functions with if-statment pseudo-lookup-table things.

Additionally, the opaque types for the extension could also be put in a table, or at least the count of them or something.

And the metatable entries could be put in a statically allocated array (in RAM!), too, to avoid DMA on platforms that don't support it like AVR. Something like 8 or 16 slots for extensions seems appropriate.

p.s. what program do you use to draw the box-and-pointer diagrams on the uLisp website?

[technoblogy]

These are all interesting ideas - I'll need to think about them!

p.s. what program do you use to draw the box-and-pointer diagrams on the uLisp website?

I use Adobe Illustrator CS5, which has been my favourite drawing program for some time.

[dragoncoder047]

I use Adobe Illustrator CS5, which has been my favourite drawing program for some time.

Dang it. I was hoping you used some special program that takes a cons list and produces an output -- e.g. takes (array . (((0 . 0) . (0 . 0)) . ((0 . 0) . (0 . nil)))) and produces this image:

[technoblogy]

No, but that's a great idea! Perhaps a Lisp program that generates SVG?

[dragoncoder047]

If I'm going to write it I would write it in Python and use Hy as a Lisp reader. But the biggest issue I can see with some sort of program like this is that there's no way to specify the layout (side-by-side? tree? etc?) without supplying additional parameters. Although there are plenty of Javascript diagram libraries that allow you to draw something like this and there are Scheme interpreters in Javascript too, so maybe a Web-based GUI program would be appropriate. But that's not your responsibility.

I also have some ideas for how to make uLisp object-oriented like Janet (https://janet-lang.org/docs/object_oriented.html) using keywords and something like the opaque objects idea, but I'll hold off on that for now because I don't want to overload you with ideas.

[technoblogy]

I also have some ideas for how to make uLisp object-oriented

Have you seen this?

Simple object system

[technoblogy]

I was thinking that these different aspects could all be stored in lookup tables in flash just like the core uLisp function lookup table

I don't yet quite understand opaque types, but this suggestion is a great idea for tidying up the currently messy code for dealing with streams. I'll have a go at implementing a rough version.

[technoblogy]

By the way, I'll probably try to sort out the problems with LastChar at the same time, since that involves streams too.

[dragoncoder047]

I was thinking that these different aspects could all be stored in lookup tables in flash just like the core uLisp function lookup table

I don't yet quite understand opaque types, but this suggestion is a great idea for tidying up the currently messy code for dealing with streams. I'll have a go at implementing a rough version.

I was alos thinking along the same lines -- streams could be representad as some sort of opaque-pointer structure, perhaps pointing to an entry in an array of Arduino File objects or something like that. Maybe a maximum of 32 open streams/files on platforms with no DMA. And maybe also on the dynamic arrays like these you can have a compile #define called dynamicmemory that switches out these arrays and the functions that deal with them for malloc()/realloc()'ed arrays, removing the limit on the number of concurrent streams or maximum number of extensions, etc.

[dragoncoder047]

I don't yet quite understand opaque types, but this suggestion is a great idea for tidying up the currently messy code for dealing with streams. I'll have a go at implementing a rough version.

status update?

no rush.

[technoblogy]

I've been thinking about how it might work. No one else is complaining, so it's not a high priority for me - more like a fun project!

[dragoncoder047]

any more progress?

Just wondering the current state of this proposal. Once it is implemented the graphics and wifi extensions could ostensibly be both moved to separate extensions rather than being built-in.

Also, if you're willing to tell, what is the next "big feature" you're working on for uLisp?

[technoblogy]

Hi - good questions!

Although I enjoy writing code for uLisp I'm actually trying to resist the temptation to keep extending it further. I think it has reached an ideal point: it meets my original objective, which is an interpreted, interactive language that's a more user-friendly alternative to C for programming microcontroller boards. It also still supports my original target board, the Arduino Uno. I think there are benefits in keeping the specification relatively compact so, for example, the language reference fits in a single web page.

There are some exceptions to this: I still want to make bug fixes, and make improvements that don't extend the specification, but make uLisp more reliable, or run faster, or are just more elegant.

But my main concern at the moment is to make uLisp more accessible to the target audience. Currently, judging by the feedback I get by email and on the forum, uLisp has quite a small number of users, mainly among people who are already fans of Lisp. I think other users are put off by the brackets and unfamiliar syntax of Lisp, and so stick to C, or choose MicroPython rather than uLisp. I believe that one thing that would help is an easier way to edit and upload uLisp programs to a microcontroller board, like a simple stand-alone IDE, and this is what I'm currently thinking about.

This is not to say that I don't welcome extensions by other users, such as your macros, etc. But although they will be welcomed by Lisp users, I don't think they will widen to appeal of uLisp to general users.

I like your idea of a streams API that would make it possible to implement the graphics and wi-fi extensions as separate extension files. However, I'm not sure how easy it would be to do this, and another problem is that the extension file mechanism I've implemented currently only supports one extension file. I had an initial version working with unlimited extension files, but there was a performance penalty, so I decided to go with a single extension file for the first release.

Look forward to your feedback on this.

[dragoncoder047]

I believe that one thing that would help is an easier way to edit and upload uLisp programs to a microcontroller board, like a simple stand-alone IDE, and this is what I'm currently thinking about.

I wrote one in Python in my repository that runs in the terminal. It probably doesn't help you that it's written in Python 😉 but it might be a good starting point. It's still pretty basic.

I like your idea of a streams API that would make it possible to implement the graphics and wi-fi extensions as separate extension files. However, I'm not sure how easy it would be to do this, and another problem is that the extension file mechanism I've implemented currently only supports one extension file. I had an initial version working with unlimited extension files, but there was a performance penalty, so I decided to go with a single extension file for the first release.

I'm not really sure why there was a performance hit, as I implemented unlimited extensions (using malloc) on my fork and I don't see any performance issues. Perhaps that may be more apparent on slower processors (I'm still using an esp32) or maybe you implemented it differently than I did.

The key function for multi-extension support in mine is the function getentry which works as follows: start at table index 0. if the builtin index is greater than the size of table N, subtract the size of table N and move to table N+1, repeat until it is in range, then dereference the pointer. Maybe you tried this and it doesn't work, I wouldn't know.

Anyway, in terms of making uLisp more accessible, one thing you could do is have a global variable for the reading meachanism *standard-input*, *standard-output*, and a nonstandard one *reader*. The functions that expect streams could, when not given a stream, look for this variable, or use the existing default if not set. The *reader* variable's default would be the existing lisp reader, and the (read) function would simply call the *reader* with the stream. That way someone could plug a reader that e.g. reads in Dylan-like code and produces Lisp; or a reader that simply allows the math operators to be put as infix operators; or use whitespace and indentation to delimit forms á la Python; or just about anything else. The only cost would be workspace usage.

As for a well-defined streams and multi-extension API, I think something like having a "metatable" holding pointers to extension structs; the first one could be core uLisp.

show big code

typedef struct {
  tbl_entry_t* table;
  size_t table_size;
  stream_type_t* streams_table;
  size_t streams_size;
  void (*gc_hook)(); // called after each GC
  const char* extension_name; // could be used for initializing *features* list etc.
  // add more pointers for opaque object management, etc.
} extension_t;

#ifdef ULISP_USE_DMA
extension_t** Extensions;
size_t num_extensions = 0;
void add_extension (extension_t* ext) {
  Extensions = realloc(Extensions, ++num_extensions);
  Extensions[num_extensions-1] = ext;
}
#else
#define MAX_EXTENSIONS 16
extension_t* Extensions[MAX_EXTENSIONS]
size_t num_extensions = 0;
void add_extension (extension_t* ext) {
  num_extensions++;
  if (num_extensions >= 16) {
    Serial.println("too many extensions");
    while (1); // Halt
  }
  Extensions[num_extensions-1] = ext;
}
#endif

As for the streams, each stream needs 6 functions (open/close, read/write, and seek/tell if applicable), plus a buffer to hold the last few characters (to eliminate LastChar mix-ups). This might be a starting point.

show big code

typedef struct {
  stream_t* (*open)(object* args);
  char (*read)(stream_t*, bool*); // bool parameter is to indicate failure
  bool (*write)(stream_t*, char); // False == failed to write
  bool (*seek)(stream_t*, ssize_t, bool); // like fseeko()
  size_t (*tell)(stream_t*);
  bool (*close)(stream_t*);
  const char* type;
} stream_type_t;

#define STREAM_BUFFER_SIZE 8
typedef struct {
  bool open;
  stream_type_t* type; // POINTER to statically allocated one, saves RAM
  uint8_t num_buffered;
  char buffer[STREAM_BUFFER_SIZE];
  void* context; // pointer to cons pair of head & tail for stringstream, not used on gfxstream, File* for sd-stream, Wire or Wire1, Serial, Serial1, Serial2, Serial3, current wifi-client, etc.
} stream_t;

#ifndef ULISP_USE_DMA
#define MAX_STREAMS 16
stream_t Streams[MAX_STREAMS];
#endif

// Replacement for isstream() and gstreamfun()
char stream_getchar (object* stream) {
  if (!streamp(stream)) error(notastream, stream);
  stream_t* str = stream->stream;
  if (str->num_buffered > 0) {
    str->num_buffered--;
    return str->buffer[str->num_buffered];
  }
  bool ok = true;
  char c = str->type->read(str, &ok);
  if (!ok) error(PSTR("failed to read"), stream);
  return c;
}

Also, one other thing that I think might make uLisp more accessible to the wider Arduino community is making uLisp a file/library to be included, not its own sketch. You don't have to split it up into separate files, but having everything uLisp be in one file that is not the .ino would probably help. All that you really need to do there is rename it to ulisp.hpp, move setup() and loop() back to the .ino, and and add #include <Arduino.h> plus forward references so it will compile without complaint. (I did that, you might even be able to use my forward references list to start with.)

I hope I'm not overwhelming you with code or ideas.