Thursday, February 23, 2012

Using JAXB for XML and JSON APIs in Scala Web Applications

In the past, I already mentioned several times the implementation of RESTful XML and JSON APIs in Scala web applications using JAXB, without going into details. In this blog post I want to shed more light on this approach together with some links to more advanced examples. A JAXB-based approach to web APIs can be useful if you want to support both XML and JSON representations but only want to maintain a single binding definition for both representations. I should also say that I'm still investigating this approach, so see the following as rather experimental.

First of all, JAXB is a Java standard for binding XML schemas to Java classes. It allows you to convert Java objects to XML documents, and vice versa, based on JAXB annotations on the corresponding Java classes. JAXB doesn't cover JSON but there are libraries that allow you to convert Java objects to JSON (and vice versa) based on the very same JAXB annotations that are used for defining XML bindings. One such library is Jersey's JSON library (jersey-json) which internally uses the Jackson library.

As you'll see in the following, JAXB can also be used together with immutable domain or resource models based on Scala case classes. There's no need to pollute them with getters and setters or Java collections from the java.util package. Necessary conversions from Scala collections or other type constructors (such as Option, for example) to Java types supported by JAXB can be defined externally to the annotated model (and reused). At the end of this blog post, I'll also show some examples how to develop JAXB-based XML and JSON APIs with the Play Framework.


In the following, I'll use a model that consists of the single case class Person(fullname: String, username: Option[String], age: Int). To define Person as root element in the XML schema, the following JAXB annotations should be added.

@XmlRootElement makes Person a root element in the XML schema and @XmlAccessorType(XmlAccessType.FIELD) instructs JAXB to access fields directly instead of using getters and setters. But before we can use the Person class with JAXB a few additional things need to be done.
  • A no-arg constructor or a static no-arg factory method must be provided, otherwise, JAXB doesn't know how to create Person instances. In our example we'll use a no-arg constructor.

  • A person's fullname should be mandatory in the corresponding XML schema. This can be achieved by placing an @XmlElement(required=true) annotation on the field corresponding to the fullname parameter.

  • A person's username should be an optional String in the corresponding XML schema i.e. the username element of the complex Person type should have an XML attribute minOccurs="0". Furthermore, it should be avoided that scala.Option appears as complex type in the XML schema. This can be achieved by providing a type adapter from Option[String] to String via the JAXB @XmlJavaTypeAdapter annotation.

We can implement the above requirements by defining and annotating the Person class as follows:

Let's dissect the above code a bit:
  • The no-arg constructor on the Person class is only needed by JAXB and should therefore be declared private so that it cannot be accessed elsewhere in the application code (unless you're using reflection like JAXB does).

  • Placing JAXB annotations on fields of a case class is a bit tricky. When writing a case class, usually only case class parameters are defined but not fields directly. The Scala compiler then generates the corresponding fields in the resulting .class file. Annotations that are placed on case class parameters are not copied to their corresponding fields, by default. To instruct the Scala compiler to copy these annotations, the Scala @field annotation must be used in addition. This is done in the custom annotation type definitions xmlElement and xmlTypeAdapter. They can be used in the same way as their dependent annotation types XmlElement and XmlJavaTypeAdapter, respectively. Placing the custom @xmlElement annotation on the fullname parameter will cause the Scala compiler to copy the dependent @XmlElement annotation (a JAXB annotation) to the generated fullname field where it can be finally processed by JAXB.

  • To convert between Option[String] (on Scala side) and String (used by JAXB on XML schema side) we implement a JAXB type adapter (interface XmlAdapter). The above example defines a generic OptionAdapter (that can also be reused elsewhere) and a concrete StringOptionAdapter used for the optional username parameter. Please note that annotating the username parameter with @xmlTypeAdapter(classOf[OptionAdapter[String]]) is not sufficient because JAXB will not be able to infer String as the target type (JAXB uses reflection) and will use Object instead (resulting in an XML anyType in the corresponding XML schema). Type adapters can also be used to convert between Scala and Java collection types. Since JAXB can only handle Java collection types you'll need to use type adapters should you want to use Scala collection types in your case classes (and you really should). You can find an example here.

We're now ready to use the Person class to generate an XML schema and to convert Person objects to and from XML or JSON. Please note that the following code examples require JAXB version 2.2.4u2 or higher, otherwise the OptionAdapter won't work properly. The reason is JAXB issue 415. Either use JDK 7u4 or higher which already includes this version or install the required JAXB version manually. The following will write an XML schema, generated from the Person class, to stdout:

The result is:

Marshalling a Person object to XML can be done with

which prints

Unmarshalling creates a Person object from XML.

We have implemented StringOptionAdapter in a way that an empty <username/> element or <username></username> in personXml1 would also yield None on Scala side. Creating JSON from Person objects can be done with the JSONJAXBContext from Jersey's JSON library.

which prints the following to stdout:

Unmarshalling can be done with the context.createJSONUnmarshaller.unmarshalFromJSON method. The JSONConfiguration object provides a number of configuration options that determine how JSON is rendered and parsed. Refer to the official documentation for details.

Play and JAXB

This section shows some examples how to develop JAXB-based XML and JSON APIs with the Play Framework 2.0. It is based on JAXB-specific body parsers and type class instances defined in trait JaxbSupport which is part of the event-sourcing example application (Play-specific work is currently done on the play branch). You can reuse this trait in other applications as is, there are no dependencies to the rest of the project (update: except to SysError). To enable JAXB-based XML and JSON processing for a Play web application, add JaxbSupport to a controller object as follows:

An implicit JSONJAXBContext must be in scope for both XML and JSON processing. For XML processing alone, it is sufficient to have an implicit JAXBContext.

XML and JSON Parsing

JaxbSupport provides Play-specific body parsers that convert XML or JSON request body to instances of JAXB-annotated classes. The following action uses a JAXB body parser that expect an XML body and tries to convert it to a Person instance (using a JAXB unmarshaller).

If the unmarshalling fails or the request Content-Type is other than text/xml or application/xml, a 400 status code (bad request) is returned. Converting a JSON body to a Person instance can be done with the jaxb.parse.json body parser.

If the body parser should be chosen at runtime depending on the Content-Type header, use the dynamic jaxb.parse body parser. The following action is able to process both XML and JSON bodies and convert them to a Person instance.

JaxbSupport also implements the following related body parsers
  • jaxb.parse.xml(maxLength: Int) and jaxb.parse.json(maxLength: Int)

  • jaxb.parse(maxLength: Int) and jaxb.parse(maxLength: Int)

  • jaxb.parse.tolerantXml and jaxb.parse.tolerantJson

  • jaxb.parse.tolerantXml(maxLength: Int) and jaxb.parse.tolerantJson(maxLength: Int)

XML and JSON Rendering

For rendering XML and JSON, JaxbSupport provides the wrapper classes JaxbXml, JaxbJson and Jaxb. The following action renders an XML response from a Person object (using a JAXB marshaller):


renders a JSON response from a Person object. If you want to do content negotiation based on the Accept request header, use the Jaxb wrapper.

Jaxb requires an implicit request in context for obtaining the Accept request header. If the Accept MIME type is application/xml or text/xml an XML representation is returned, if it is application/json a JSON representation is returned. Further JaxbSupport application examples can be found here.

Friday, January 20, 2012

Building an Event-Sourced Web Application - Part 2: Projections, Persistence, Consumers and Web Interface

UPDATE: A successor of the example application described in this blog post is available here. It is based on the eventsourced library, a library for building reliable, scalable and distributed event-sourced applications in Scala.

A few weeks ago I started a blog post series to summarize my experiences in building an event-sourced web application using Scala and Akka. This was done based on an example application (see branch part-1). There, I gave an overview of the application architecture and presented some details of the immutable domain model and the service layer. Since then, the example application has been extended (see branch part-2) with a number of new features and enhancements. Here's an overview:

Enhancements are:
  • The STM-based state management was completely revised and generalized into the traits UpdateProjection and EventProjection. An UpdateProjection is similar to an Akka Agent: it applies state transition functions asynchronously and can participate in STM transactions. The major difference is that an UpdateProjection is specialized on domain object updates and can log captured events to a persistent event log. By default, update events are logged before state changes are visible via STM references (this is an important difference to the service layer of part 1). UpdateProjection implementors (such as the example application's InvoiceService) are domain event producers. EventProjection implementors, on the other hand, are domain event consumers. They internally use plain Akka Agents to manage state and derive new state values from received domain events (using an application-defined event projection function). EventProjection implementors are usually components that manage read models or coordinate business processes in event-driven architectures, for example.
  • The domain model was enhanced by introducing the domain classes DraftInvoice, SentInvoice and PaidInvoice to represent the states an invoice can have. Valid state transitions are defined by the methods on these domain classes and can therefore be checked by the compiler. This approach is explained more detailed here although the implementation used in our example application slightly differs. In part 1, we only had a single Invoice class and valid state transitions had to be checked at runtime.

New features include:
  • A revised EventLog trait together with two implementations: JournalioEventLog is based on Journal.IO and BookkeeperEventLog on Apache BookKeeper. An EventLog supports synchronous and asynchronous appending of events as well as iterating over stored events, either from the beginning or from an application-defined position in the event history. Event log entries are also assigned sequence numbers so that event consumers can detect gaps in event streams or re-order (resequence) them, if needed.
  • Event consumers. One example is InvoiceStatistics, an EventProjection that derives invoice update statistics from domain events. Here, a separate read model is used (following the CQRS pattern) to serve invoice statistic queries. Another example is InvoiceReplicator, an EventProjection that simply reconstructs the invoice map (as maintained by the InvoiceService) from invoice events at a different location. It can be used to replicate application state across different nodes and to serve (eventually consistent) reads. The replicated state could also be used by a snapshot service to take snapshots of application state. InvoiceReplicator needs to receive events in the correct order and is therefore configured to resequence the received event stream. A third example is the PaymentProcess. It coordinates the activities of InvoiceService and PaymentService. Instead of having these services sending commands to each other, it is the PaymentProcess that sends commands to (i.e. calls methods on) these services in reaction to domain events. This event-driven approach to implementing business processes not only decouples the services from each other but also lets other components extend (or monitor) the business process by subscribing to and reacting on the relevant domain events. The PaymentProcess is currently stateless. Processes that need to maintain state should implement EventProjection and recover the process state during application start (or failover) from the event history.
  • A RESTful web interface for invoices and invoice statistics with support for HTML, XML and JSON representations. These can be negotiated with the HTTP Accept header. The web layer is based on the Jersey web framework (the JAX-RS reference implementation). HTML representations are rendered with the Scalate template engine. The mapping between XML and JSON representations and immutable domain classes is based on JAXB annotations. A JAXB-based XML provider must be supported by any JAX-RS implementation but Jersey additionally comes with a JAXB-based JSON provider so that the same metadata (JAXB annotations) can be used to generate both XML and JSON representations. Following some simple rules, it is possible to JAXB-annotate Scala case classes without polluting them with Java collection types or getters and setters. One major drawback of the current JAX-RS specification is that it doesn't support asynchronous responses yet. This will change with JAX-RS 2.0 and then we can make full use of the asynchronous InvoiceService responses.
  • A communication Channel for connecting domain event producers to consumers. The example application provides a SimpleChannel implementation for local communication. Alternative implementations, for example, could connect to a distributed event bus to communicate events across components of a distributed application. 

Running the example application

The example application can be started with

sbt run-main dev.example.eventsourcing.server.Webserver

Two classes relevant for starting the application are:
  • Appserver: configures the event log, services, read models and processes and connects them via an event channel. It also recovers application state from the event history.
  • Webserver: configures the web service layer and starts an embedded web server (Jetty).
To experiment with the BookKeeper based event log, you need to replace JournalioEventLog with BookkeeperEventLog in Appserver and additionally start a test BookKeeper instance with

sbt run-main dev.example.eventsourcing.server.Zookeeper
sbt run-main dev.example.eventsourcing.server.Bookkeeper

Examples how to interact with the RESTful web interface can be found here.

Service Layer Enhancements

In the service layer implementation from part 1 we've seen how to keep the order of logged events in correspondence with the order of updates to the application state. We used a transient event log that could participate in STM transactions. After the transaction commits, the events from the transient event log have been transferred to a persistent event log. A drawback of this approach is that one can loose updates in case of crashes after an STM reference has been updated but before the changes have been written to the persistent event log. This can lead to situations where clients can already see application state that cannot be recovered from the event log. While some applications may tolerate this, others may require that any visible application state must be fully recoverable from the event log. Therefore, an alternative approach must be chosen.

We need a way to write events, captured during domain object updates, to a persistent event log before the STM reference is updated. But writing to the persistent event log must be done outside an STM transaction for reasons explained in part 1. Updates must also be based on the current (i.e. latest) application state. We therefore need to
  1. Get the current state value from a transactional reference (STM transaction 1)
  2. Update domain object(s) obtained from the current state (no STM transaction)
  3. Write the captured update event(s) to a persistent event log (no STM transaction)
  4. Compute a new state value from the domain object update and write it to the transactional reference (STM transaction 2)
Since steps 1-4 are not a single transaction we must prevent their concurrent execution. This can be achieved using a single actor, for example. This actor is the single writer to the transactional reference. This is actually very similar to how Akka Agents work internally. The main difference in our case is that the computation of a new state value occurs outside an STM transaction, whereas Akka Agents apply update functions within STM transactions. This difference allows us to have side effects, such as writing to a persistent event log. The example application provides the above functionality in the UpdateProjection trait:
  • Instances of UpdateProjection manage (part of) application state with a transactional ref of type Ref[S] where S is the state value type. Clients concurrently read application state via currentState. Sequential writes to the transactional ref are done exclusively by the updater actor (more on write concurrency below).
  • UpdateProjection implementors change application state with the transacted method. The update parameter is a function that computes a domain object Update[Event, B] from current state S where B is a domain object type. The update result (either Success[B] or Failure[DomainError]) is returned as future value from the transacted method.
  • The update function and the underlying Future implementation object (promise) are sent to the updater actor with an ApplyUpdate message. The updater then reads the current state and applies the update function to it. If the update succeeds, it writes the captured events to an EventLog and projects the update result onto the current state. The projection is done with the project function. It creates a new state value from the current state and the update result. The new state value is then finally set on the transactional ref and the promise is completed with the update result.
  • Furthermore, the transacted method can participate in STM transactions. If there's an enclosing transaction, the updater actor will only be triggered if the enclosing transaction successfully commits. If there's no enclosing transaction the updater actor will always be triggered.
The example application uses UpdateProjection to implement the stateful InvoiceService. The state is of type Map[String, Invoice] i.e. a single map containing draft, sent and paid invoices. Here's a simplified version of InvoiceService:

The updateInvoice method uses the transacted method of the UpdateProjection trait. It tries to get an invoice with given invoiceId from the current state and applies the supplied update function f to it. The updateInvoice method is used by updateDraftInvoice for updating draft invoices in the invoice map. The updateDraftInvoice method is used by the service methods addInvoiceItem and sendInvoiceTo. Adding an item to an existing draft invoice yields a future value of an updated draft invoice (return type Future[DomainValidation[DraftInvoice]]). Sending an existing draft invoice, on the other hand, causes a state transition of that invoice to a sent invoice (return type Future[DomainValidation[SentInvoice]]) i.e. the service methods make use of the newly introduced domain object types. The InvoiceService must also implement the abstract members project, initialState and eventLog (declared by Projection and UpdateProjection).
  • The project implementation projects an updated invoice onto the current state by simply adding it to the map of invoices (replacing an old invoice if present).
  • An initialState (empty map by default) and an EventLog instance are provided during construction of an InvoiceService.
The above InvoiceService implementation supports concurrent reads but only sequential writes to the whole invoice map. This may be sufficient for many applications but if a higher degree of write concurrency is needed, one could choose to have a separate UpdateProjection instance per invoice object (which is comparable to have a separate Akka Agent for each invoice object). This allows both concurrent reads and writes to the invoices of an application. The following snippet shows the general idea (not part of the example project).

Here, an InvoiceService maintains a map of PersistentInvoice instances where a PersistentInvoice is an UpdateProjection that contains a reference to a single (draft, sent or paid) invoice. Consequently, updates to different invoices can now be made concurrently. The projection function degenerates to a function that simply returns the updated invoice.

Event Logs


CQRS and Consistency


Business Processes


Web Interface

TODO (see also this blog post)