Tokenize Go Language source code

TODO:

• [X] Semicolon insertion (currently ';' is required) - DONE!
• [ ] Unicode identifiers (currently '#[...]' can be used)

Nonstandard TopLevelPrel production

This is not part of the standard, but is here to abstract away some of the details of possible extensions to the language.

Standard Expression

Technically, the Go spec says

• Expression = UnaryExpr | Expression binary_op UnaryExpr .
• UnaryExpr = PrimaryExpr | unary_op UnaryExpr .

but we combine these into one production here.

Standard PrimaryExpr

PrimaryExpr is occurs in many places:

• Expression,
• TypeSwitchGuard,
• and in its own definition.

Therefore, it is useful to have a separate datatype for it, since otherwise we would have to repeat ourselves. This is the responsibility goPrimary below. The only thing we do here is convert the AST one level, so it's an expression.