protobuf-0.2.0.2: Google Protocol Buffers via GHC.Generics

Safe HaskellNone
LanguageHaskell2010

Data.ProtocolBuffers

Contents

Description

An implementation of Protocol Buffers in pure Haskell.

Extensive documentation is available at https://developers.google.com/protocol-buffers/docs/overview and Google's reference implementation can be found at http://code.google.com/p/protobuf/.

It is intended to be used via GHC.Generics and does not require .proto files to function. Tools are being developed that will convert a Haskell Protobuf definition into a .proto and vise versa.

Given a message definition:

{-# LANGUAGE DeriveGeneric #-}

 import Data.Int
import Data.ProtocolBuffers
import Data.Text
import GHC.Generics (Generic)
import GHC.TypeLits

 data Foo = Foo
   { field1 :: Required '1' (Value Int64) -- ^ The last field with tag = 1
   , field2 :: Optional '2' (Value Text) -- ^ The last field with tag = 2
   , field3 :: Repeated '3' (Value Bool) -- ^ All fields with tag = 3, ordering is preserved
   } deriving (Generic, Show)

 instance Encode Foo
instance Decode Foo
 

It can then be used for encoding and decoding. The Encode and Decode instances are derived automatically using DeriveGeneric and DefaultSignatures as outlined here: http://www.haskell.org/haskellwiki/GHC.Generics#More_general_default_methods.

To construct a message, use putField to set each field value. Optional, Repeated and Packed fields can be set to their empty value by using mempty. An example using record syntax for clarity:

>>> let msg = Foo{field1 = putField 42, field2 = mempty, field3 = putField [True, False]}

To serialize a message first convert it into a Put by way of encodeMessage and then to a ByteString by using runPut. Lazy ByteString serialization is done with runPutLazy.

>>> fmap hex runPut $ encodeMessage msg
"082A18011800"

Decoding is done with the inverse functions: decodeMessage and runGet, or runGetLazy.

>>> runGet decodeMessage =<< unhex "082A18011800" :: Either String Foo
Right
  (Foo
    { field1 = Field {runField = Required {runRequired = Always {runAlways = Value {runValue = 42}}}}
    , field2 = Field {runField = Optional {runOptional = Last {getLast = Nothing}}}
    , field3 = Field {runField = Repeated {runRepeated = [Value {runValue = True},Value {runValue = False}]}}
    }
  )

Use getField to read fields from a message:

>>> let Right msg = runGet decodeMessage =<< unhex "082A18011800" :: Either String Foo
>>> getField $ field1 msg
42
>>> getField $ field2 msg
Nothing
>>> getField $ field3 msg
[True,False]

Some Protocol Buffers features are not currently implemented:

  • Default values for Optional fields
  • Extension fields
  • Storing unknown fields, those without a mapped field tag in message record
  • Tag-delimited Groups, deprecated in lieu of Message

Synopsis

Message Serialization

Encoding

class Encode a where

Minimal complete definition

Nothing

Methods

encode :: a -> Put

Instances

Encode (HashMap Tag [WireField])

Untyped message encoding

encodeMessage :: Encode a => a -> Put

Encode a Protocol Buffers message.

encodeLengthPrefixedMessage :: Encode a => a -> Put

Encode a Protocol Buffers message prefixed with a varint encoded 32-bit integer describing it's length.

Decoding

class Decode a where

Minimal complete definition

Nothing

Methods

decode :: HashMap Tag [WireField] -> Get a

Instances

Decode (HashMap Tag [WireField])

Untyped message decoding, decode = id

decodeMessage :: Decode a => Get a

Decode a Protocol Buffers message.

decodeLengthPrefixedMessage :: Decode a => Get a

Decode a Protocol Buffers message prefixed with a varint encoded 32-bit integer describing it's length.

Fields

Tags

Restricted type aliases of Field. These are used to attach a field tag (a numeric id) to a field. Each tag must be unique within a given message, though this is not currently checked or enforced.

type family Required n a :: *

Required fields. Parsing will return empty if a Required value is not found while decoding.

Instances

type family Optional n a :: *

Optional fields. Values that are not found will return Nothing.

Instances

type Repeated n a = Field n (RepeatedField [a])

Lists of values.

type Packed n a = Field n (PackedField (PackedList a))

Packed values.

Accessors

Fields tend to have rather complex types that are unpleasant to interact with. HasField was designed to hide this complexity and provide a consistent way of getting and setting fields.

class HasField a where

Functions for wrapping and unwrapping record fields. When applied they will have types similar to these:

getField :: Required '1' (Value Text) -> Text
putField :: Text -> Required '1' (Value Text)

getField :: Optional '2' (Value Int32) -> Maybe Int32
putField :: Maybe Int32 -> Optional '2' (Value Int32)

getField :: Repeated '3' (Value Double) -> [Double]
putField :: [Double] -> Repeated '3' (Value Double)

getField :: Packed '4' (Value Word64) -> [Word64]
putField :: [Word64] -> Packed '4' (Value Word64)
 

Minimal complete definition

getField, putField

Associated Types

type FieldType a :: *

Methods

getField :: a -> FieldType a

Extract a value from it's Field representation.

putField :: FieldType a -> a

Wrap it back up again.

field :: Functor f => (FieldType a -> f (FieldType a)) -> a -> f a

An isomorphism lens compatible with the lens package

Selectors

Follow these rules to define fields supported by the generic encoder/decoder:

  • The n phantom type parameter specifies the Protocol Buffers field tag (id).
  • Field tags must be an instance of Nat.
  • Field selectors must be an instance of Foldable to support encoding.
  • Value selectors must be an instance of Monoid to support decoding.

data Field n a

Fields are merely a way to hold a field tag along with its type, this shouldn't normally be referenced directly.

This provides better error messages than older versions which used Tagged

Instances

Functor (Field n) 
Foldable (Field n) 
Traversable (Field n) 
Bounded a => Bounded (Field n a) 
Enum a => Enum (Field n a) 
Eq a => Eq (Field n a) 
Ord a => Ord (Field n a) 
Show a => Show (Field n a) 
Monoid a => Monoid (Field n a) 
NFData a => NFData (Field n a) 
HasField (Field n (PackedField (PackedList (Enumeration a))))

Iso: FieldType (Packed n (Enumeration a)) = [a]

HasField (Field n (PackedField (PackedList (Value a))))

Iso: FieldType (Packed n (Value a)) = [a]

HasField (Field n (RepeatedField [Enumeration a]))

Iso: FieldType (Repeated n (Enumeration a)) = [a]

HasField (Field n (RepeatedField [Value a]))

Iso: FieldType (Repeated n (Value a)) = [a]

HasField (Field n (OptionalField (Last (Enumeration a))))

Iso: FieldType (Optional n (Enumeration a)) = Maybe a

HasField (Field n (OptionalField (Last (Value a))))

Iso: FieldType (Optional n (Value a)) = Maybe a

HasField (Field n (RequiredField (Always (Enumeration a))))

Iso: FieldType (Required n (Enumeration a)) = a

HasField (Field n (RequiredField (Always (Value a))))

Iso: FieldType (Required n (Value a)) = a

HasField (Field n (RepeatedField [Message a]))

Iso: FieldType (Repeated n (Message a)) = [a]

HasField (Field n (OptionalField (Maybe (Message a))))

Iso: FieldType (Optional n (Message a)) = Maybe a

HasField (Field n (RequiredField (Always (Message a))))

Iso: FieldType (Required n (Message a)) = a

Typeable (Nat -> * -> *) Field 
type FieldType (Field n (PackedField (PackedList (Enumeration a)))) = [a] 
type FieldType (Field n (PackedField (PackedList (Value a)))) = [a] 
type FieldType (Field n (RepeatedField [Enumeration a])) = [a] 
type FieldType (Field n (RepeatedField [Value a])) = [a] 
type FieldType (Field n (OptionalField (Last (Enumeration a)))) = Maybe a 
type FieldType (Field n (OptionalField (Last (Value a)))) = Maybe a 
type FieldType (Field n (RequiredField (Always (Enumeration a)))) = a 
type FieldType (Field n (RequiredField (Always (Value a)))) = a 
type FieldType (Field n (RepeatedField [Message a])) = [a] 
type FieldType (Field n (OptionalField (Maybe (Message a)))) = Maybe a 
type FieldType (Field n (RequiredField (Always (Message a)))) = a 

Values

Selectors

Each field value needs to specify the way it should be encoded.

There are three built-in value selectors: Value, Enumeration and Message.

If you're unsure what value selector to use, Value is probably the correct one.

data Value a

Value selects the normal/typical way for encoding scalar (primitive) values.

Instances

Functor Value 
Foldable Value 
Traversable Value 
Bounded a => Bounded (Value a) 
Enum a => Enum (Value a) 
Eq a => Eq (Value a) 
Ord a => Ord (Value a) 
Show a => Show (Value a) 
Monoid a => Monoid (Value a) 
NFData a => NFData (Value a) 
DecodeWire a => DecodeWire (Last (Value a)) 
DecodeWire a => DecodeWire (Maybe (Value a)) 
DecodeWire (PackedList (Value Bool)) 
DecodeWire (PackedList (Value Double)) 
DecodeWire (PackedList (Value Float)) 
DecodeWire (PackedList (Value Int32)) 
DecodeWire (PackedList (Value Int64)) 
DecodeWire (PackedList (Value Word32)) 
DecodeWire (PackedList (Value Word64)) 
DecodeWire (PackedList (Value (Fixed Int32))) 
DecodeWire (PackedList (Value (Fixed Int64))) 
DecodeWire (PackedList (Value (Fixed Word32))) 
DecodeWire (PackedList (Value (Fixed Word64))) 
DecodeWire (PackedList (Value (Signed Int32))) 
DecodeWire (PackedList (Value (Signed Int64))) 
DecodeWire a => DecodeWire (Always (Value a)) 
DecodeWire a => DecodeWire (Value a) 
EncodeWire a => EncodeWire [Value a] 
EncodeWire a => EncodeWire (Last (Value a)) 
EncodeWire a => EncodeWire (Maybe (Value a)) 
EncodeWire (PackedList (Value Bool)) 
EncodeWire (PackedList (Value Double)) 
EncodeWire (PackedList (Value Float)) 
EncodeWire (PackedList (Value Int32)) 
EncodeWire (PackedList (Value Int64)) 
EncodeWire (PackedList (Value Word32)) 
EncodeWire (PackedList (Value Word64)) 
EncodeWire (PackedList (Value (Fixed Int32))) 
EncodeWire (PackedList (Value (Fixed Int64))) 
EncodeWire (PackedList (Value (Fixed Word32))) 
EncodeWire (PackedList (Value (Fixed Word64))) 
EncodeWire (PackedList (Value (Signed Int32))) 
EncodeWire (PackedList (Value (Signed Int64))) 
EncodeWire a => EncodeWire (Always (Value a)) 
EncodeWire a => EncodeWire (Value a) 
HasField (Field n (PackedField (PackedList (Value a))))

Iso: FieldType (Packed n (Value a)) = [a]

HasField (Field n (RepeatedField [Value a]))

Iso: FieldType (Repeated n (Value a)) = [a]

HasField (Field n (OptionalField (Last (Value a))))

Iso: FieldType (Optional n (Value a)) = Maybe a

HasField (Field n (RequiredField (Always (Value a))))

Iso: FieldType (Required n (Value a)) = a

Typeable (* -> *) Value 
type Required n (Value a) = Field n (RequiredField (Always (Value a))) 
type Optional n (Value a) = Field n (OptionalField (Last (Value a))) 
type FieldType (Field n (PackedField (PackedList (Value a)))) = [a] 
type FieldType (Field n (RepeatedField [Value a])) = [a] 
type FieldType (Field n (OptionalField (Last (Value a)))) = Maybe a 
type FieldType (Field n (RequiredField (Always (Value a)))) = a 

data Enumeration a

Enumeration fields use fromEnum and toEnum when encoding and decoding messages.

data Message m

The way to embed a message within another message. These embedded messages are stored as length-delimited fields.

For example:

data Inner = Inner
   { innerField :: Required '1' (Value Int64)
   } deriving (Generic, Show)

 instance Encode Inner
instance Decode Inner

 data Outer = Outer
   { outerField :: Required '1' (Message Inner)
   } deriving (Generic, Show)

 instance Encode Outer
instance Decode Outer
 

It's worth noting that Message a is a Monoid and NFData instance. The Monoid behavior models that of the Protocol Buffers documentation, effectively Last. It's done with a fairly big hammer and it isn't possible to override this behavior. This can cause some less-obvious compile errors for paramterized Message types:

data Inner = Inner{inner :: Required '2' (Value Float)} deriving (Generic, Show)
instance Encode Inner
instance Decode Inner

data Outer a = Outer{outer :: Required '3' (Message a)} deriving (Generic, Show)
instance Encode a => Encode (Outer a)
instance Decode a => Decode (Outer a)
 

This fails because Decode needs to know that the message can be merged. The resulting error implies that you may want to add a constraint to the internal GMessageMonoid class:

/tmp/tst.hs:18:10:
  Could not deduce (protobuf-0.1:GMessageMonoid (Rep a))
    arising from a use of `protobuf-0.1: Decode .$gdmdecode'
  from the context (Decode a)
    bound by the instance declaration at /tmp/tst.hs:18:10-39
  Possible fix:
    add an instance declaration for
    (protobuf-0.1:GMessageMonoid (Rep a))
  In the expression:
    (protobuf-0.1:Decode.$gdmdecode)
  In an equation for decode:
      decode = (protobuf-0.1:Decode .$gdmdecode)
  In the instance declaration for `'Decode' (Outer a)'

The correct fix is to add the Monoid constraint for the message:

- instance (Encode a) => Decode (Outer a)
+ instance (Monoid (Message a), Decode a) => Decode (Outer a)

Instances

Functor Message 
Foldable Message 
Traversable Message 
Eq m => Eq (Message m) 
Ord m => Ord (Message m) 
Show m => Show (Message m) 
(Generic m, GMessageMonoid (Rep m)) => Monoid (Message m) 
(Generic m, GMessageNFData (Rep m)) => NFData (Message m) 
Decode m => DecodeWire (Message m) 
(Foldable f, Encode m) => EncodeWire (f (Message m)) 
HasField (Field n (RepeatedField [Message a]))

Iso: FieldType (Repeated n (Message a)) = [a]

HasField (Field n (OptionalField (Maybe (Message a))))

Iso: FieldType (Optional n (Message a)) = Maybe a

HasField (Field n (RequiredField (Always (Message a))))

Iso: FieldType (Required n (Message a)) = a

type Required n (Message a) = Field n (RequiredField (Always (Message a))) 
type Optional n (Message a) = Field n (OptionalField (Maybe (Message a))) 
type FieldType (Field n (RepeatedField [Message a])) = [a] 
type FieldType (Field n (OptionalField (Maybe (Message a)))) = Maybe a 
type FieldType (Field n (RequiredField (Always (Message a)))) = a 

Wire Coding

Some primitive values can be more compactly represented. Fields that typically contain negative or very large numbers should use the Signed or Fixed wrappers to select their respective (efficient) formats.