demo-proper-tokenizing
123456789
say .parse: $*IN.slurp given grammar {
rule TOP { <identifier> <identifier> '=' <expression> }
token ws { [ <comment> | \s+ ]* <!ww> }
token comment { '#[' <-[\]]>* ']' }
token identifier { \w+ }
rule expression { '(' <expression> ')' | <number> | <string> | ... }
token number { \d+ }
token string { '"' <-["]>* '"' }
}
Raku
INFO