Spaces can't be ignored when interpreting the language.

[patrickfeltes]

For example Disp "X", which is a display command with multiple whitespaces before the argument will cause an error. So we need to tokenize spaces when they are appropriate (i.e. not in strings).