json-tokens- Tokenize JSON

Safe HaskellNone




data Token Source #

A token in a JSON document.

Eq Token Source # 
Instance details

Defined in Json.Token


(==) :: Token -> Token -> Bool #

(/=) :: Token -> Token -> Bool #

Show Token Source # 
Instance details

Defined in Json.Token


showsPrec :: Int -> Token -> ShowS #

show :: Token -> String #

showList :: [Token] -> ShowS #

decode :: Bytes -> Either TokenException (SmallArray Token) Source #

Decode a sequence as JSON tokens. This allows token sequences that would be rejected by the ABNF given in RFC 7159:

>>> decode (Bytes.fromAsciiString "[ , true }")
Right [ LeftBracket, Comma, BooleanTrue, RightBrace ]

It is up to the user to reject such malformed JSON when they parse the token sequence. More surprisingly, this tokenizer accepts some unnatural juxtapositions of token sequences without whitespace. For example:

>>> decode (Bytes.fromAsciiString "55truefalse")
Right [ Number 55, BooleanTrue, BooleanFalse ]
>>> decode (Bytes.fromAsciiString "null\"hello\"")
Right [ Null, String "hello" ]

Acceptance of such samples simplifies the implementation of this tokenizer. These unnatural juxtapositions always result in token sequences that should be rejected anyway in the subsequent parsing done by the user. Consequently, their acceptance is not considered harmful.