PXB: Perl XML data Binding
Maxim Grigoriev
This document presents and describes API designed for binding RelaxNG (Compact) or XML schema, represented by RelaxNG Compact notation into the perl class library. The main idea behind such binding is to provide flexible and scalable API which will allow developers of the document literal based web services easily describe any service messaging profile by writing simple perl data structures. This approach allows you to work with data from any XML message using your own class structures, so your XML elements are binded into the perl objects, therefore it called Perl XML data Binding (or PXB). The PXB framework will handle all the details of converting data to and from XML or DOM objects based on your instructions. The DOM objects tree is created by utilizing very fast LibXML library. PXB was designed to allow you a high degree of control over the translation process with custom callbacks and SQL mapping of the hierarchical XML entities into the flat SQL database schema. Even more importantly to note that PXB creates heavily documented, highly supportable and maintainable perl class hierarchies. It also automatically creates test suit for every created class. The the Perl::Tidy manpage and the Perl::Critic manpage profile files are included into the standard test suit in order to comply with the Best Perl Practices. If you familiar with java XML data binding packages such xmlbeans, CASTOR, METRO or older JAXB then idea of unmarshalling XML into the simple objects should be known to you.
PXB uses the power of perl data structures to define the rules of how perl
objects are converted to or from XML (the binding). Or more explicitly to and
from the DOM tree objects. Why not to use DOM instead? DOM is just a
representation of the XML tree data structure. It does not provide any facility to
hook up easily some custom callback aligned with some external data schema to
the particular element in the tree. The current spectrum of available XML APIs
for perl is limited to tree walking and XPath based search. Also, the process of refactoring heavy XML driven perl code based on
XML/XPath approach was found to be very tedious and error prone.
Especially when schemas are constantly adapting to the ever changing customer's requirements.
The PXB framework is an attempt to bring configurable callbacks into the DOM
and add some functional validation inside of such binding as well as provide
mechanism to map SQL database schema to the hierarchical structure of the DOM tree.
Importance of such mapping technique was exposed by the lack of robust support for native XML data
type among the current list of “freeware” storage engines where the SQL RDBM
is more or less a standard and performance of the native XML databases are still far
from the optimal. Also, PXB uses perl as meta-language and it uses
perl to create perl OO API which makes schema's code significantly smaller.
Any developer can concentrate on implementing semantic rules, protocols and actual
functionality instead of wasting any time on reimplementing once again XML tree walking API.
Moreover, there is an automated test suite for the each class of the generated API and
every method call is documented. The internal structure of every class was designed
to be clean, readable and easily understandable by any other perl developer.
It utilizes perl’ best practices and uses fields packages to provide tighter
encapsulation and creates explicitly named lists of accessors and
mutators. Behavioral semantic of the each class in the class tree is the same
and implements absolutely the same methods to allow transparency and
propagation of the implemented callbacks across the API.
The basic XML element definition is represented as perl hash reference:
<element-variable> = '{ attrs => { ' <attributes-definition> ',' ‘ 'xmlns’ ‘ => ‘ ' <namespace prefix> '},'
'elements => [' <elements-definition> '],'
'‘text ’ => ' <text-content> ','
'‘sql => {' <sql-mapping-definition> '}'
'}'
Where each <xxxx-definition> can be expressed in EBNF as:
<attributes-definition> = (attribute-name '‘=>’' attribute-value) (',’' <attributes-definition>)*
attribute-name = string
namespace-id = string
attribute-value = ‘scalar’| ('’enum:’'(attribute-name ( '‘,’' attribute-name)*))
<elements-definition> = ( '‘[‘' element-name '’=>'’ (
<element-variable> |
’'[' ‘ <element-variable> '’]’' |
’'[‘' ( <element-variable> '’,'’)+’ ']’' |
'’[‘' ('['‘ <element-variable> '’]’,'’?)+ ’']'’ )
',' conditional-statement? ']’')*
conditional-statement = ('‘unless’'|’'if'’)’':'’(registered-name ('‘,'’ registered-name)*)
registered-name = attribute-name|element-name
element-name = string
<text-content> = ‘scalar’|conditional-statement
<sql-mapping-definition> = (sql-table-name ‘'=>’ ‘{'‘ sql-table-entry '‘}’' ) ( '‘,'’ <sql-mapping-definition>)*
sql-table-entry = (sql-entry-name '‘=>’ ’{'‘ entry-mapping ‘'}’' ) ( '‘,’ ' sql-table-entry )*
sql-table-name = string
sql-entry-name = string
entry-mapping = '‘value’ ‘=>' ’( element-name|( '‘['‘ element-name ('‘,'’ element-name)+ ’']’')) ( ‘','’ if-condition)?
if-condition = '‘if’ ‘=>’' attribute-name '‘:’' attribute-value
Where:
elements => [parameter => [$parameter]] - defines list of 'parameter' sub-elements
elements => [parameter => $parameter] - defines single 'parameter'’ sub-element
elements => [parameters => [$parameters,$select_parameters]] - defines choice between two single
‘'parameters' sub-elements of different type
elements => [datum => [[$pinger_datum], [$result_datum]] ] - defines choice between two lists of
'datum' sub-elements of different type
In elements-definition the third member is an optional conditional statement which represents validation rule.
For example ‘unless:value’ conditional statement will be translated into the perl’ conditional statement
&& !($self->get_value) where ‘value’ must be registered attribute or sub-element name and this condition
will be placed in every piece of code where perl object is serialized into the XML DOM object or unmarshalled from it.
In order to provide the same class interface throughout the API, the same universal constructors and same set
of method calls were implemented. Currently the constructor body includes initialization part as well and this
part is slightly different for different modules, but it might be reduced to generic constructor and some custom
initialization part in the future where generic constructor might be inherited from some sort of the base Object class.
Currently every constructor “knows” how to initialize the whole class from the DOM object, XML string fragment
or from the reference to the perl hash structure of the named class fields. Also, it “knows” how to handle arrays
of the class fields with support for elements identified by id. It supports special field,
named idmap for that purpose. It “knows” how to serialize object of the class into the DOM or XML
text string and how to map contents of the object on some SQL schema. By issuing registerNamespaces call to
the root object one can obtain the reference to the hash with all namespaces utilized by every sub-element in the object.
It utilizes another special field, called nsmap for that. This field is an object of the generated helper
class and serves as container for map between local names and namespace prefixes. Every namespace is identified by
namespace prefix and the version of the schema is getting built into the generated API package pathname.
Our case is based on building interoperable SOAP document/literal based webservice for perfSONAR-PS project. Webservices are indeed wrappers around network performance monitoring tools. Where there are two types of services - Measuremnt Archive (MA) with ability to publish historical data and Measurement Point (MP) providing on-demand measurements.
The major utilization scenario of the PXB framework comes when there is a need to build MA service with SQL database based storage engine. In any case PXB framework will provide the complete solution and actual development for MP will be contained in writing real time measurements facility class and then utilizing PXB to inject measured data into the published XML message. For MA the whole SD process will be dedicated to writing the actual message handler and implementing actual protocol specifications. For example, for PingER MA it is the perfSONAR_PS::Datatypes::PingER manpage class, inherited from the perfSONAR_PS::Datatypes::Message manpage class with actual implementation of SetupData request and MetaDataKey request handlers. The the perfSONAR_PS::Datatypes::Message manpage is an abstract class, extending the perfSONAR_PS::Datatypes::v2_0::nmwg::Message manpage class. This extra class exists because the root Message class from the PXB is message type agnostic.
The root of the perfSONAR-PS schema and the root of the OO API built by PXB is the Message object.
It exists in the perfSONAR base namespace identified by nmwg id. The schema is versioned.
The most current root package name for Message class is the perfSONAR_PS::Datatypes::v2_0::nmwg::Message manpage.
The base schema is completely defined by the perfSONAR_PS::DataModels::DataModel manpage module.
This is a simple perl package and not the class, because it has only data definitions.
The current DataModel has implemented definitions from several OGF schemas, namely:
filter.rnc, nmtopo.rnc, nmbase.rnc, nmtm.rnc, nmtl3.rnc, nmtl4.rnc.
There is no SQL mapping definitions in the base data model allowed, because SQL mapping is a service specific.
Any service specific extension of this base schema must be extended in the separate data model package
as it was done for PingER service. The PingER data model can be viewed as an example and it is contained
in the the perfSONAR_PS::DataModels::PingER_Model manpage. Any other extension data model package can be created for any other service.
Let’s look on example of the parameter element. It is described in the base model as:
$parameter = {attrs => {name => 'scalar', value => 'scalar', xmlns => 'nmwg'},
elements => [],
text => 'unless:value',
};
That means the parameter element has name and value attributes, does not have any sub-elements and might have optional (only when value attribute is undefined) text content. For PingER service it was extended as:
$parameter->{'attrs'} = {name => 'enum:count,packetInterval,packetSize,ttl,valueUnits,startTime,endTime,deadline,transport,setLimit',
value => 'scalar', xmlns => 'nmwg'};
Where explicit enumeration was added for the list of allowed values of the name attribute and SQL mapping was defined as:
$parameter->{sql} = {metaData => {count => {value => ['value' , 'text'],
if => 'name:count'},
packetInterval=> {value => ['value' , 'text'],
if => 'name:packetInterval'},
packetSize=> {value => ['value' , 'text'] ,
if => 'name:packetSize'},
ttl=> {value => ['value' , 'text'],
if => 'name:ttl'},
deadline => {value => ['value' , 'text'] ,
if => 'name:deadline'},
transport => { value => ['value' , 'text'],
if => 'name:transport'},
},
time => { start => {value => ['value' , 'text'],
if => 'name:startTime'},
end => { value => ['value' , 'text'],
if => 'name:endTime'},
},
limit => { setLimit => { value => ['value' , 'text'],
if => 'name:setLimit'},
},
};
Please note that time table name was made up to allow mapping of the time range related entities.
Also, tlimit table name was made up to provide non-existed paging mechanism in order to limit the
size of the requested dataset in the response message. As it clear from the example above, the schema can be
modified in any time without affecting the rest of the service API. For example extra attribute ‘named type’ can be
added and the whole API framework rebuild in the matter of seconds. Then developer can utilize this extra
attribute type by calling ->get_type on the parameter object in the message handling class and
new functionality will be immediately supported.
Lets look again on the example of the SQL mapping definitions for PingER service from previous chapter. Every object of
the API class tree implements querySQL call. It accepts reference to the empty hash and passes this reference
through the whole objects tree while filling contents of the hash with initialized contents of the “objects of interest”.
The “object of interest” is defined by the SQL mapping definition.
For example, for definition:
{metaData => {count => {value => ['value' , 'text'], if => 'name:count'}
for Parameter object with name attribute set to ‘count’ and ‘value attribute’ or text content
set to ‘100’ it will add:
metaData => {‘count’ => { ‘eq’ => ‘100’}}
into the SQL query hash.
The resulted hash for metaData table will be returned and will look as:
{ metaData => {count => '100'}}
where it can be easily passed to any of SQL ORM frameworks in order to build proper WHERE clause.
For example the Class::DBI manpage or with minor refactoring to the the Rose::DB manpage