PsychoPath XPath 2.0 processor update

We recently implemented quite a few built in XSD numeric data types in PsychoPath XPath 2.0 processor (ref, http://www.w3.org/TR/xmlschema-2/#built-in-datatypes). Now all (I mean, really all of xs:decimal ones :)) the data types in the xs:decimal hierarchy are available in PsychoPath, and these should be available in Eclipse WTP 3.2 M1 (which should be released sometime soon after the Eclipse Galileo release, at around June 09' end).

Now almost all the major built in Schema types are available in PsychoPath, except for few subtypes of xs:string (like xs:normalizedString, xs:token etc.). These shouldn't be much difficult to add.

Dave Carver reported, that the improvements we have done recently in PsychoPath have significantly improved it's compliance to the W3C XPath 2.0 test suite.

PsychoPath processor version is now enhanced from 1.0 to 1.1.

Xerces-J XSD 1.1 assertions and PsychoPath XPath 2.0 processor update

I recently contributed few patches to the Eclipse PsychoPath XPath 2.0 engine, to support Schema aware XPath (2.0) expressions. These patches would enhance Schema aware support in PsychoPath XPath2 engine, for element and attribute nodes, for the XML Schema primitive types.

These enhancements in PsychoPath engine would make XPath expressions like following possible, to be evaluated by PsychoPath engine:

person/@dob eq xs:date('2006-12-10') // if dob is an attribute, and of schema type xs:date

person/dob eq xs:date('2006-12-10') // if dob is an element, and of schema type xs:date

@max ge @min // this would work if 'max' and 'min' have say schema types, xs:int

The patch in my local environment already exhibits these improvements. As promised by Dave Carver (the PsychoPath engine project lead), users would likely get these improvements in Eclipse WTP (Web Tools Project) 3.2.
2009-05-24: These changes are now committed to the Eclipse CVS server, and the improvements are flagged to be delivered in Eclipse WTP 3.2 M1, which should be quite sooner. Thanks to Dave Carver for testing all my patches, and committing them to the server.

PsychoPath engine already has a framework (thanks to Andrea Bittau and his team) for supporting Schema awareness (based on the Xerces-J XSD schema model). I just added in small pieces of code in attribute and element node implementations (particularly, improving the "typed value" of attribute and element nodes for built in XSD schema types), to enhance schema aware support.

I think, we are gradually moving to a more mature schema aware support in PsychoPath.

I'm also using these new PsychoPath processor capabilities, to implement schema aware XPath 2.0 evaluations in Xerces-J XSD assertions support.

I'm currently working on to construct a typed XPath data model instance, for XSD 1.1 assertions evaluations. Having this capability, would allow users to write XPath expressions like, following:

@max ge @min

or

person/@dob eq xs:date('2006-12-10')

In the absence of this (i.e, typed XDM nodes), currently users have to make explicit cast operations, like following:

xs:int(@max) ge xs:int(@min)

or

xs:date(person/@dob) eq xs:date('2006-12-10')

The XML Schema 1.1 assertions spec recommends a typed XDM instance.

We hope to provide this capability within Xerces-J, inline with the XML Schema 1.1 assertions specification.

2009-05-23: These improvements are now implemented, and I've submitted the code improvements to the Apache Xerces-J JIRA server. I'm hoping, we'll have these improvements committed on the Xerces-J SVN server some time soon.

Became Apache Xerces-J committer

As per the voting process for becoming an Apache project committer, the Apache Xerces-J team granted me the committer status for the Xerces-J project on May 5, 2009.

This gives me an opportunity to contribute to the Xerces-J codebase, in a more direct way.

It's indeed a privilege for me to be part of the core Xerces team. Starting from being an Xerces user (since long time ago :)), to becoming project committer has been a rewarding journey in numerous ways.

Apache Xerces-J assertions implementation and PsychoPath XPath 2.0 processor

I shared sometime back on this blog, on the work I am doing regarding XML Schema 1.1 assertions support in Xerces-J. The XML Schema 1.1 assertions processing requires a XPath 2.0 processor for performing Schema validation.

The Xerces-J team has opted to use the open source XPath 2.0 processor, PsychoPath. PsychoPath was developed by Andrea Bittau and his team. The PsychoPath team donated the PsychoPath code base to Eclipse community, where it is now formally used in the Eclipse, Web Tools Platform project. Future enhancements to PsychoPath are now taking place at Eclipse.

Since Xerces-J is using PsychoPath XPath 2.0 engine, we wish that PsychoPath be ideally 100% compliant to the XPath 2.0 specification, so Xerces-J users can use much of the failities of the XPath 2.0 language while using XML Schema 1.1 assertions.

After looking at the PsychoPath source code and using it quite a bit, my personal observation is, that PsychoPath has a pretty good XPath 2.0 implementation. Please refer to this documentation for knowing more about PsychoPath and the current compliance status.

The Eclipse WTP team is working actively to solve any remaining non-compliant items in PsychoPath. Incidentally, I have been working recently to help improve PsychoPath's compliance to the XPath 2.0 spec, and have contributed few patches to Eclipse.

We are also planning to run the W3C XPath 2.0 test suite on PsychoPath, and targetting PsychoPath to pass the test suite, with 100% coverage. This should give the PsychoPath adopters more confidence while using it.

WAS XML Feature Pack Open Beta

There was an annoucement recently from IBM (http://webspherecommunity.blogspot.com/2009/04/was-open-xml-feature-pack-beta.html), about availability of "WAS XML Feature Pack Open Beta" supporting XPath 2.0, XSLT 2.0 and XQuery 1.0. It was good to know this.

Therefore, users would be able to use XPath 2.0, XSLT 2.0 and XQuery 1.0 in a WAS environment, using IBM's own processors for these languages.

This is an early preview release, with more enhancements expecting to come later.

I'm looking forward to try these language processors myself.

Xerces-J: XML Schema 1.1 assertions support

This post is related to my earlier blog post, http://mukulgandhi.blogspot.com/2008/07/assertions.html about the XML Schema 1.1 assertions implementation into Xerces-J.

Today, I reached an important milestone with all the development finished for assertions in Xerces-J, and submitted an Apache JIRA issue for review.

Here is a small example of what XML Schema 1.1 assertions means:

<xs:complexType name="book">
    <xs:sequence>
      <xs:element name="name" type="xs:string" />
      <xs:element name="author" type="xs:string" />
      <xs:element name="price" type="xs:string" />
      <xs:element name="publisher" type="xs:string" />
      <xs:element name="pub-date" type="xs:date" /> 
    </xs:sequence>
    <xs:assert test="ends-with(price, 'USD')" />
    <xs:assert test="pub-date > xs:date('2007-12-31')" />
  </xs:complexType>

With this XML Schema 1.1 fragment, the user wants to have a validation constraint that, the price string should end with literal 'USD' and pub-date should be greater than the date 2007-12-31. This is a very simple example, but it does signify the usefulness of assertions syntax. We could have unlimited (0-n) numbers of xs:assert elements in a XSD schema type (which could be a simple type or a complex type. Though the assertions facet name in simple types is named xs:assertion). The value of 'test' attribute in assertions is an XPath 2.0 expression. All the assertions have to evaluate to boolean, "true" for an element to be locally valid.

There could be many other scenarios (and some of them quite complex, like for e.g., assertions present in a Schema type hierarchy) for writing assertions in XML Schema 1.1. It's difficult to specify all of them here. I'd ask the reader, to read the article [2] below, for learning about many of other, XML Schema 1.1 assertions scenarios.

With assertions in XML Schema 1.1 language, we could express much more involved XML validation constraints, that were almost impossible to specify in XML Schema 1.0. Using assertions, we can specify relationships between elements (like element names, contents etc), between elements and attributes, between attributes, and perhaps much more.

The assertions processing in XML Schema 1.1 works as follows:
When a XML Schema (1.1) processor encounters an element in the XML instance document, it must validate the element (if the user has requested validation) with it's associated type in the Schema (which could be a simple type or a complex type). The element's type declaration could be anonymous, or it could be a named type (which has a "name" attribute, and they are globally defined in the schema) declaration in the Schema. The XML Schema processor builds a XPath data model (XDM) tree rooted at this element (with Xerces, a XDM tree is built only if any assertions (which could be, 1-n in numbers) are associated with an element's type. If schema types of XML attributes have assertion facets, then these assertion facets work upon the attribute's value, and no XDM tree is constructed in this case). The XDM tree consists of the root element, it's attributes and all it's descendants. When an element validation is going on within Xerces, assertions evaluation also takes place as part of the validation process. Each assertion is evaluated on the XDM tree rooted at a given context element. Therefore, also any attempt by the assert XPath expression to access any node outside this element tree will not succeed.

We also have a wiki page for Xerces assertions implementation, http://wiki.apache.org/xerces/XML_Schema_1.1_Assertions. It describes a bit of implementation details of assertions in Xerces.

I'm happy to share that we expect Xerces-J to support the whole of assertions implementation in a near future release. And of course, Xerces would support lot of other XML Schema 1.1 features as well.

Following are few nice articles related to XML Schema 1.1, which are worth reading:
1. Overview of XML Schema 1.1 language
2. XML Schema 1.1 co-occurence constraints using XPath 2.0

Data modeling approaches

With reference to my earliest blog post on "Relational/XML hybrid database design",
http://mukulgandhi.blogspot.com/2009/01/relationalxml-hybrid-database-design.html,

I have some more thoughts on this topic.

I think, an information model of an enterprise should consider XML as a first class data storage mechanism, along side the relational mechanism.

When XML was not there, using relational storage (and some other forms of EIS data) was the only choice. Given the benefits of XML (some of which are mentioned in my previous blog post cited above), it should be considered equally important as the relational mechanism, for data persistence needs.

There should be a mixture of XML and relational data in the overall data model of an enterprise. This would bring best of the worlds of relational and XML storage, to the enterprise's data.

I have seen that, today XML is used a lot for various kinds of property/configuration files, and for sending messages. But I feel, that wherever it makes good business sense, the core business entities should be modeled as XML along side the relational models. With the current hybrid XML/Relational databases (like, DB2 with pureXML), we can have the relational and XML data in the same transactional context.