Emerald - A Ruby VM

Emerald is a Ruby compiler & virtual machine written in Go.

Building Emerald

Run the following from your command line:

git clone git@github.com:mathiashsteffensen/emerald.git
cd emerald && ./scripts/install

This will build emerald, iem, and emerald-sandbox-worker executables in the current directory.

To run a source file of Ruby code:

./emerald main.rb

To start the Emerald REPL:

./iem

Embedding

Use Engine for a stateful, full runtime. Host inputs are explicit:

engine := emerald.New()
value, err := engine.EvalWithOptions(source, emerald.EvalOptions{
    Args:     []string{"one", "two"},
    LoadPath: []string{"/app/lib"},
})

Engine keeps globals, classes, and methods between evaluations and is not safe for concurrent evaluation.

Use Sandbox for isolated scripts with no filesystem, process output, networking, sleep, or clock APIs:

sandbox, err := emerald.NewSandbox(emerald.SandboxOptions{
    Timeout:          250 * time.Millisecond,
    MemoryLimitBytes: 256 << 20,
    WorkerPath:       "/usr/local/bin/emerald-sandbox-worker",
})
value, err := sandbox.Eval(source)

if errors.Is(err, emerald.ErrSandboxTimeout) {
    // The complete evaluation exceeded its deadline.
}

Each sandbox evaluation uses a fresh runtime, so state is not shared and one Sandbox may be used concurrently. EvalFile accepts a source name for error reporting; it does not read that file. The worker executable must be installed alongside the host application: make install installs emerald-sandbox-worker at /usr/local/bin, or deploy the built sidecar and set WorkerPath to its absolute path.

Results are transferred between processes, so Eval* supports only nil, booleans, integers, floats, strings, symbols, and acyclic arrays and hashes of those values. Classes, blocks, user instances, regexps, aliases, and cycles are rejected.

An abrupt worker exit, protocol error, or unrecoverable memory failure matches ErrSandboxWorkerFailed; an untransferable return value matches ErrSandboxUnsupportedResult.

On Linux, the worker receives a hard address-space limit before it receives untrusted source. On macOS, GOMEMLIMIT and RSS controls are best effort only. Each evaluation runs in a separate worker process, so VM defects cannot take down the embedding process; this is not a security boundary against arbitrary native-code compromise.

Architecture

Lexer

The lexer/tokenizer is located in the ./parser/lexer package. I have tried to keep it simple, but when implementing string templates (This is a #{template}), there are basically only 2 options.

1. Don't have a separate lexer & parser.
2. Turn the lexer into a stack machine since templates can be infinitely nested (or as deep as the stack allows)

And since I prefer the performance of having the parser & lexer separate (they run in parallel) I went with option 2, but this significantly increases the complexity of the lexer.

Parser

The parser is located in the ./parser package. It is a Pratt parser. Basically a recursive descent, operator precedence parser.

Compiler

The compiler is located in the ./compiler package. The compiler transforms the AST generated by the parser, into compact bytecode. The bytecode is, as the name may suggest, simply an array of bytes, they represent Opcodes & operators. All Opcode definitions can be found in the ./compiler/bytecode.go file.

Virtual Machine

The virtual machine is located in the ./vm package. It is a virtual stack machine and does not make use of any registers. This keeps the implementation & Opcode definitions simple, but it does mean that we need more bytecode to perform the same operations a register machine would. This ultimately means more execution cycles for the equivalent result.

Supported language features

This is still quite far away from being a real implementation. The below is a list of the features on the roadmap and the ones that have already been implemented.

NOTE: To say they have been implemented does not mean any features are guaranteed to be compatible with the reference Ruby implementation.

• everything is an object
  • allow method calls on everything
  • operators are method calls
• full UTF8 support
  • Unicode identifier
  • Unicode symbols
• method definitions
  • with parens
  • without parens
  • without parens with args
  • return keyword
  • default values for parameters
  • keyword arguments
  • block arguments
  • yield
• method calls
  • with parens
  • without parens
  • without parens with args
  • with block arguments
  • keyword arguments
• conditionals
  • if
  • if modifier
  • if/else
  • if/elif/else
  • ternary ? :
  • unless
  • unless modifier
  • unless/else
  • case
  • ||
  • &&
• control flow
  • for loop
  • while loop
  • until loop
  • break
  • next
  • redo
  • flip flop
• numbers
  • integers
    • integer arithmetics
    • integers 1234
    • integers with underscores 1_234
    • decimal numbers 0d170, 0D170
    • octal numbers 0252, 0o252, 0O252
    • hexadecimal numbers 0xaa, 0xAa, 0xAA, 0Xaa, 0XAa, 0XaA
    • binary numbers 0b10101010, 0B10101010
  • floats
    • float arithmetics
    • 12.34
    • 1234e-2
    • 1.234E1
    • floats with underscores 2.2_22
• booleans
• strings
  • double quoted
  • single quoted
  • %q{}
  • %Q{}
  • heredoc
    • without indentation (<<EOF)
    • indented (<<-EOF)
    • “squiggly” heredoc <<~
    • quoted heredoc
      • single quotes <<-'HEREDOC'
      • double quotes <<-"HEREDOC"
      • backticks <<-`HEREDOC`"
  • escaped characters
    • \a bell, ASCII 07h (BEL)
    • \b backspace, ASCII 08h (BS)
    • \t horizontal tab, ASCII 09h (TAB)
    • \n newline (line feed), ASCII 0Ah (LF)
    • \v vertical tab, ASCII 0Bh (VT)
    • \f form feed, ASCII 0Ch (FF)
    • \r carriage return, ASCII 0Dh (CR)
    • \s space, ASCII 20h (SPC)
    • \\ backslash, \
    • \nnn octal bit pattern, where nnn is 1-3 octal digits ([0-7])
    • \xnn hexadecimal bit pattern, where nn is 1-2 hexadecimal digits ([0-9a-fA-F])
    • \unnnn Unicode character, where nnnn is exactly 4 hexadecimal digits ([0-9a-fA-F])
    • \u{nnnn ...} Unicode character(s), where each nnnn is 1-6 hexadecimal digits ([0-9a-fA-F])
    • \cx or \C-x control character, where x is an ASCII printable character
    • \M-x meta character, where x is an ASCII printable character
    • \M-\C-x meta control character, where x is an ASCII printable character
    • \M-\cx same as above
    • \c\M-x same as above
    • \c? or \C-? delete, ASCII 7Fh (DEL)
  • interpolation #{}
• arrays
  • array literal [1,2]
  • array indexing arr[2]
  • splat
  • array decomposition
  • implicit array assignment
  • array of strings %w{}
  • array of symbols %i{}
• nil
• hashes
  • literal with => notation
  • literal with key: notation
  • indexing hash[:foo]
  • every Ruby Object can be a hash key
• symbols
  • :symbol
  • :"symbol"
  • :"symbol" with interpolation
  • :'symbol'
  • %s{symbol}
  • singleton symbols
• regexp
  • /regex/
  • %r{regex}
• ranges
  • .. inclusive
  • ... exclusive
• procs ->
• variables
  • variable assignments
  • globals
  • Ruby globals ($ notation)
• operators
  • +
  • -
  • /
  • *
  • !
  • <
  • >
  • ** (pow)
  • % (modulus)
  • & (AND)
  • ^ (XOR)
  • >> (right shift)
  • << (left shift, append)
  • == (equal)
  • != (not equal)
  • === (case equality)
  • =~ (pattern match)
  • !~ (does not match)
  • <=> (comparison or spaceship operator)
  • <= (less or equal)
  • >= (greater or equal)
  • assignment operators
    • +=
    • -=
    • /=
    • *=
    • %=
    • **=
    • &=
    • |=
    • ^=
    • <<=
    • >>=
    • ||=
    • &&=
• error handling
  • begin
  • rescue
  • ensure
  • retry
• constants
• scope operator ::
  • Constant access MyMod::MyClass
  • Method access String::new
• classes
  • class objects
  • class Class
  • instance variables
  • class variables
  • class methods
  • instance methods
  • method overrides
  • private
  • protected
  • public
  • inheritance
  • constructors
  • new
  • self
  • singleton classes (also known as the metaclass or eigenclass) class << self
  • singleton methods def self.method
  • assigment methods
  • super in methods
• modules
• object main
• comments '#'
• backtraces (unwinding of the VM call stack for debugging)
• C extension compatability