Inside TongaLang:
from text to execution.

When someone writes a program, the computer does not understand the source file as a human does. The text has to be broken into meaningful units, checked against a grammar, represented in a structure the runtime can process, and then evaluated. That pipeline is the heart of an interpreter.

For TongaLang, I implemented that pipeline in Python using PLY. The project is an educational language with Tonga-derived keywords, but its internal architecture uses the same broad ideas found in many language implementations: lexical analysis, parsing, an abstract syntax tree and an execution environment.

1. The lexer: recognising the language's vocabulary

The lexer reads source characters and groups them into tokens. A token has a type and a value. For example, a variable declaration, a number, an arithmetic operator and a conditional keyword should not all be treated as interchangeable strings.

In TongaLang, the lexer defines token categories for identifiers, numbers, strings, arithmetic and comparison operators, logical operators, grouping symbols and reserved words. It also maps the language's reserved vocabulary to token types. This gives later stages a consistent representation to work with.

Identifiers are case-insensitive: names such as Age, AGE and age resolve to the same normalized identifier. The lexer also handles comments, string escapes and line tracking. When it encounters an invalid character or malformed string, it raises a language-specific lexical error with a location and an explanatory message.

2. The parser: checking structure

Tokens alone do not tell us whether a program is valid. The parser checks how those tokens fit together. A number followed by an operator and another number can form an expression; a conditional keyword must be followed by a condition and a block; a function declaration has its own required structure.

TongaLang uses PLY's Yacc component to define grammar productions. The grammar describes statements such as variable declarations, assignments, output, conditionals, loops, function declarations, returns and expressions. Operator precedence is also defined so that expressions are grouped consistently.

For example, multiplication and division bind more tightly than addition and subtraction. Logical OR and AND have their own precedence levels, and unary operators are handled separately. Without these rules, the same expression could be interpreted in different ways depending on how the parser happened to reduce it.

3. The AST: representing meaning, not punctuation

After parsing, TongaLang builds an abstract syntax tree. The AST removes much of the surface punctuation and represents the program in terms of operations and relationships: a variable declaration has a name and initializer; a conditional has branches; a loop has a condition or range and a body; a binary expression has a left operand, operator and right operand.

This structure is useful for more than execution. It gives the IDE something meaningful to display in its AST explorer, and it provides a place to attach source locations so the interface can connect a node back to the code the learner wrote.

4. The runtime: environments and evaluation

The interpreter evaluates the AST. Runtime environments hold variable bindings and support scope. When a declaration is evaluated, the interpreter stores its value in the appropriate environment. When an expression is evaluated, it resolves its operands and applies the requested operation. Function calls create the execution context needed for parameters and return values.

Control-flow nodes determine which statements run. A conditional evaluates its condition and selects a branch. A loop repeatedly evaluates its body according to its condition or range. Return and break operations alter the normal flow of execution and need to be handled deliberately rather than treated as ordinary expression values.

The language engine also provides native functions for interaction with the outside world, including output and input. The bala() input workflow is particularly important in the desktop IDE because execution must pause for a value without making the entire interface unresponsive.

5. Why the engine is separate from the IDE

I kept the lexer, parser, AST and interpreter outside the Tkinter interface. The command-line runner and the graphical IDE use the same language engine. That means a program should not behave differently simply because it was launched from a different front end.

It also makes testing more direct. Language rules can be tested without opening a window, while GUI tests can focus on editor behaviour, diagnostics, input and output. This separation reduces the number of moving parts in each test and makes failures easier to locate.

6. Errors are part of the language design

A parser that rejects invalid code is doing its job, but an educational language should also help a learner understand what went wrong. TongaLang's lexical and syntax errors carry line and column information, stable error codes, English and Tonga explanations, and hints where the implementation can provide them.

That is why source locations are captured during parsing rather than added as an afterthought. The IDE can use them to highlight the relevant code and connect a diagnostic to the learner's current file.

7. Testing the pipeline

The repository includes example programs that exercise variables, conditions, loops, functions, input, arithmetic, Boolean logic and scope. The documented test suite also covers regressions, adversarial cases and GUI workflows. The current README records 179 passing automated tests.

A passing test suite is evidence of tested behaviour, not proof that every possible program is correct. The point is to make the implementation repeatable and to catch known failure modes as the language evolves.

Explore the implementation

• Lexer implementation
• Parser and grammar productions
• Language engine source
• Complete TongaLang repository

Building an interpreter taught me that a programming language is not just a list of keywords. It is a coordinated set of rules, data structures and runtime decisions. The vocabulary is what a learner sees first; the engineering underneath is what makes the language behave consistently.

Written by Leo Mabuku · Software developer and Computer Science graduate · Get in touch