Discussion

[LadyBeGood]

Hello.

I recently came across Lox2 and I like the direction you took with it, turning Lox from more of a toy language into something that feels more complete and practical.

I'm currently building my own language that transpiles to JavaScript. While searching for resources on creating a transpiler, I found your project and it has been really interesting to go through your repo. Although I primarily work with JavaScript and don't have much experience with lower-level languages like C, I can still somewhat follow how you have structured your code.

Up until now, I have mostly been looking at the TypeScript compiler for inspiration. However, it has been very difficult to learn from because of its size and complexity. There are many layers and abstractions, and I often get lost trying to understand the overall flow.

I would really like to understand how you approached extending Lox into something more complete, and how I could apply similar ideas in my own transpiler. If you are open to discussing it, I would really appreciate the chance to ask a few questions, especially about the type checker, and learn from your experience.

Thanks for sharing your project.

[HallofFamer]

Hi @LadyBeGood

Of course I am open to talk about this, though I feel it’s a very big topic so I ain’t sure if I can cover everything in a single comment. What I can suggest is to think about what features you want to include to your language, and why these features are needed.

For Lox2, I want to build a modern full fledged object oriented programming language. The object model was inspired by Smalltalk and Ruby, while it borrowed many features from different languages. It is important to have a vision, otherwise your language will evolve organically like JS and PHP. You also want to think about how features fit with each other.

Once you have a design philosophy and a list of features to implement, you will need to think about how to implement them. It is nice to have a roadmap, and to follow a step by step approach. Start with the simplest and the most fundamental changes you want to make, and gradually approach the harder and more challenging aspects. You may also want to study the compilers and VMs of existing languages for inspirations.

With this being said, I was still a beginner of this subject when I got started, so I was also learning by doing. Fortunately, Lox2 is an experimental and educational language, so I can safely make corrections and breaking changes across versions if deemed necessary. The same may not hold true for your language, if backward compatibility is important. I was also limited by some design choices made by the original Lox language, which I would have done differently if starting from scratch.

[LadyBeGood]

Thank you for taking the time to respond.

To be honest, I've more or less settled on the overall design and semantics of my language. The part that feels most daunting to me is the semantic analysis / type checker phase. I am comfortable with the other stages of the pipeline, lexing, parsing, tranforming and code generation, but my language's semantics does not allow me to move forward with tranformation without a proper analysis phase.

The type system I am aiming for is somewhat unusual. Utkrisht is a statically and strogly typed language but all types are inferred, so you don't need (or I should say, you can't) explicitly declare the type of a variable, there's just no syntax for it. I have recently discovered that F# follows a somewhat similar approach which makes me believe it is possible to design such a system.

However, looking at the TypeScript compiler's 53,000-line type checker has been quite intimidating and not particularly helpful from a learning perspective.

How did you approach the analysis phase in Lox2? More specifically:

• How did you design and implement the resolver and typechecker phase?
• Did you follow any paticular reference / guide?

I will greatly appreciate any insight into your process.

[HallofFamer]

In Lox2, the semantic analyzer(resolver) and type checker are as simple as additional passes that walk the AST, in a similar way as how the code generator/compiler walks AST to generate bytecode. The resolver performs basic semantic analysis, and inserts names/symbols into symbol table. The type checker is a bi-directional type analyzer that handles both type checking and inference.

I did read quite a few books and references before implementing the resolver and type checker, though it was also helpful looking into the compiler and VM of existing languages. If you want to explore the source code for Lox2, I’d suggest using the branch for v2.0 as it had the simplest form of type checking without higher order types like callable and generic types.

For F#, it uses a completely different type system called Hindley–Milner (HM). It is powerful and elegant, allowing you to infer everything without even specifying type annotations at all. However, it wouldn’t fit into Lox2 as the language is full of dynamic code while HM type system works best for fully statically typed code. I am not an expert on HM type system and I won’t be able to tell you much about this, I suggest reading some books and research papers about it to understand how it works for F# and ML languages.