Mukul Gandhi

Using XSD 1.1 assertions on complexType mixed contents

2012-01-26T15:39:00.001+05:30

There were some interesting ;) thoughts coming to my mind lately, and not surprisingly again related to XSD. I was playing with XSD 1.1 assertions once again to try to constrain an XSD complexType{mixed} content model and I'm sharing some of my findings ... (I guess, I hadn't written about this particular topic on this blog before or on any other forum. If you find any duplicacy of information in this post with any information I might have written elsewhere, kindly ignore the earlier things I might have written). I come to the topic now.

What is XSD mixed content (you may ignore reading this, if you already know about this)?
I believe, this isn't really an XSD only topic. It is something which is present in plain XML (there can be a good old well-formed XML document, which might have "mixed" content and needn't be validated at all -- i.e in a schema free XML environment), but XSD allows to report such an XML instance document as 'valid' (more importantly, XSD would report a "mixed" content model XML instance as 'invalid' if validated by an "element only" content model specified by an XSD complexType definition) and also to constrain XML mixed contents in certain ways (particularly with XSD 1.1 in some new ways, which I'll try to talk about further below).

Example of "element only" (content of element "X" here) XML content model [X1]:

<X>
  <Y/>
  <Z/>
</X>

Example of "mixed content" (content of element "X" here) XML content model [X2]: 

<X>
  abc
  <Y/>
  123
  <Z/>
  654
</X>

Therefore, "mixed content" allows "non whitespaced" text nodes as siblings of element nodes.

XSD 1.0 schema definition that allows "mixed" content [XS1]:

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="X">    
        <xs:complexType mixed="true">
             <xs:sequence>
                 <xs:element name="Y"/>
                 <xs:element name="Z"/>
             </xs:sequence>
        </xs:complexType>
    </xs:element>
    
</xs:schema>

This schema (XS1) would report the XML document "X2" above as 'valid' (since that instance document has "mixed" content, and this schema allows "mixed" content via a property "mixed = 'true'" on a complexType definition).

But in the schema document "XS1" above, if we remove the property specifier "mixed = 'true'" or set the value of attribute "mixed" as 'false' (which is also the default value of this attribute), then such a modified schema would report the XML instance document "X2" above as 'invalid' (but the XML document "X1" above would be reported as 'valid' -- since it doesn't has "mixed" content).

New capabilities provided by XSD 1.1 to constrain XML "mixed" content further:

Following is a list of new features supported by XSD 1.1 for XML "mixed" contents, that currently come to my mind,

a)

XSD 1.1 schema "XS2":

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="X">    
       <xs:complexType mixed="true">
          <xs:sequence>
             <xs:element name="Y"/>
             <xs:element name="Z"/>
          </xs:sequence>          
          <xs:assert test="deep-equal(text()[matches(.,'\w')]/normalize-space(.), ('abc','123','654'))"/>
       </xs:complexType>
    </xs:element>
    
</xs:schema>

The <assert> element in this schema (XS2) constrains the mixed content in XML instance document to be a list (with order of list items been significant) of only few specified values. The assertion is written only to illustrate the technical capabilities of an assertion here, but not with any application in mind.
Following are few of other things, which XSD 1.1 assertions could achieve in an XML "mixed" content model's context:

b)

<xs:assert test="((text()[matches(.,'\w')]/normalize-space(.))[2] castable as xs:integer)
                    and
                 ((text()[matches(.,'\w')]/normalize-space(.))[3] castable as xs:integer)"/>

This assertion constrains specific items of an XML "mixed" content model list to be of a specified XSD schema type -- here the 2nd and 3rd items of the list need to be typed as xs:integer, whereas the first item is "untyped".

c)

<xs:assert test="count((text()[matches(.,'\w')]/normalize-space(.))[. castable as xs:integer])
                    =
                 count(text()[matches(.,'\w')]/normalize-space(.))"/>

This assertion constrains all items of the XML "mixed" content model list to be of the same type (xs:integer in this case) -- this uses a well defined pattern "count of xs:integer items is equal to the count of all the items".

d)

<xs:assert test="every $x in text()[matches(.,'\w')][position() gt 1]
                   satisfies 
                (number(normalize-space($x)) gt number($x/preceding-sibling::text()[matches(.,'\w')][1]))"/>

This assertion constrains the list of XML "mixed" content model to be in ascending numeric order (assuming that all items in the list are numeric. Though it should be possible to specify a numeric order on a heterogeneously typed list, and specify numeric order only for numeric list items).

Summary: XSD 1.0 allowed an "untyped" XML mixed content, that was uniformly available anywhere within the scope of an XML element that was validated by an XSD complexType. No further constraints on "mixed" content were possible in an XSD 1.0 environment. XSD 1.1 allows some new ways to constrain XML "mixed" content further (some of these capabilities were illustrated in examples above). To my opinion, the likely benefits of constraining XML "mixed" content in some of the ways as illustrated above, is to allow the XML document authors to model certain semantic content in "mixed" content scope and make this knowledge available to the XML applications. All examples above were tested with Apache Xerces (I hope that these examples would also be compliant with other XSD validators, notably Saxon currently which also supports XSD 1.1).

I hope that this information was useful.

XPath 2.0 and XSD schemas : sharing experiences

2011-08-23T19:57:00.034+05:30

I was just playing with XPath 2.0 and thought of sharing my observations, about a specific use case.

We start with the following XSD schema document,

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
    
    <xs:element name="X">
       <xs:complexType>
          <xs:sequence>
             <xs:element name="a" type="xs:integer"/>
          </xs:sequence>
          <xs:attribute name="att1" type="xs:boolean"/>
       </xs:complexType>
    </xs:element>

</xs:schema>

This schema intends to validate an XML instance document like following,

<X att1="0">
  <a>100</a>
</X>

I wrote an XPath (2.0) expression like following [1],

/X[if (@att1) then true() else false()]/a/text() AND ran this after enabling validation of the input document.

I though that this would not return any result (i.e an empty sequence).

But the XPath expression above ([1]) returns the result "100". At first thought, I was little amazed by this result. I thought, that since attribute "att1" was declared with type xs:boolean in the schema, the "if condition" should return 'false' in this case. But that's not the correct interpretation of the XPath expression written above ([1]). Following is a little more explanation about this.

The reference @att1 in the XPath expression above (i.e if (@att1) ..) is a node reference (an attribute node) and is not a boolean value (which I thought initially, and I was wrong -- I incorrectly thought, that atomization of the expression @att1 would take place in this case; more about this below).

The XPath 2.0 spec says, that if the first item in the sequence is a non null node, then effective boolean value of such a sequence is 'true' (this interpretation is unaffected by the fact, if the input XML document was validated or not with the XSD schema). And in the expression like above (i.e if (@att1) ..), the effective boolean value of the sequence {@att1} is used to determine IF the "if condition" returns 'true' or not (in this case, this sequence has one item [which is also the first item of this sequence] which is an attribute node whose name is "att1", which makes the effective boolean value as 'true' -- and hence the XPath predicate evaluates to 'true'). I think this explains, why the "if condition" {if (@att1)} would return true for the above XML instance document (even if it was validated by the schema given above, and the XPath 2.0 expression above [1] was run in a schema aware mode).

To write the XPath expression correctly, as I wanted (i.e the expression of the "if condition" should return 'true' if the instance document had value true/1 for the attribute, and 'false' otherwise AND an XSD validation of instance document took place prior to the evaluation of the XPath expression), the XPath expression would need to be modified to either of the following styles [2],

/X[if (data(@att1)) then true() else false()]/a/text()

/X[if (@att1 = true()) then true() else false()]/a/text()

To understand why the expressions given above ([2]) work correctly, one needs to understand the XPath 2.0 "data" function (for the first correct variant above, [2] -- this returns the typed value of the argument of the "data" function) and the process of atomization (for the second correct variant above, [2] -- in this case the attribute node "att1" is atomized to return a sequence of kind {xs:boolean}) as described by the XPath 2.0 spec.

That's all about this. I hope that my experience with this may be helpful to someone (to understand this, one just has to know the XPath [2.0] spec correctly, and how it interacts with XSD schemas!).

Thanks for reading this post.

@2011-11-11: updated in place, to correct few factual errors.

[revisiting] Xerces-J XSModel serializer

2011-07-26T19:26:00.004+05:30

I started playing a bit with Xerces-J XSSerializer utility (it's actually a sample within Xerces-J and was introduced in Xerces-J 2.10.0 -- the version in SVN is slightly better and will be released with a future Xerces release; and it serializes an in-memory Xerces XSModel instance into a lexical XSD syntax), and thought of writing something about it's features.

XSModel serializer has following two important (and currently the only ones) serialization features/options:
1. Selecting the XSD language version, the XSModel serializer should work with. By default this is XSD 1.0, but it can be set to XSD 1.1 via the following command line parameter, {-version 1.1}. There are very few XSD 1.1 features that the XSModel serializer currently supports. We'll try to add more XSD 1.1 features in future to the XSModel serializer. But the XSD 1.0 support with Xerces's XSModel serializer is fairly complete.
2. The XSD language prefix during serialization output can be configured with the option, {-prefix <prefix-value>}. For e.g "-prefix xsd". If this option is not specified, the prefix "xs" is generated as default during XSModel instance serialization.

I've had few interesting observations while using the Xerces XSSerializer (illustrated with small examples below),

1. I supplied the following XSD document (only the element declaration is shown, since this is the focus of this point) to the XSModel serializer,

<xs:element name="E1">
   <xs:simpleType>
      <xs:list>
         <xs:simpleType>
           <xs:restriction base="xs:string">
              <xs:minLength value="5"/>
           </xs:restriction>
         </xs:simpleType>
      </xs:list>
   </xs:simpleType>
</xs:element>

and the XSModel serializer echoed this element instance (the XSModel serializer converted the lexical schema into XSModel instance, and then serialized the XSModel again to lexical XSD syntax) to following,

<xs:element name="E1">
   <xs:simpleType>
      <xs:list>
         <xs:simpleType>
            <xs:restriction base="xs:string">
               <xs:whiteSpace value="preserve"/>
               <xs:minLength value="5"/>
            </xs:restriction>
         </xs:simpleType>
      </xs:list>
   </xs:simpleType>
</xs:element>

The interesting thing I notice in this example is, the generation of the built in facet "whiteSpace" for the XSD type xs:string.

2. Serializing the following XSD element,

<xs:element name="E1">
   <xs:simpleType>
      <xs:list>
         <xs:simpleType>
            <xs:restriction base="xs:integer">
               <xs:minInclusive value="5"/>
            </xs:restriction>
         </xs:simpleType>
      </xs:list>
   </xs:simpleType>
</xs:element>

produces the following round-trip output with the XSModel serializer,

<xs:element name="E1">
   <xs:simpleType>
      <xs:list>
         <xs:simpleType>
            <xs:restriction base="xs:integer">
               <xs:whiteSpace value="collapse"/>
               <xs:fractionDigits value="0"/>
               <xs:minInclusive value="5"/>
               <xs:pattern value="[\-+]?[0-9]+"/>
            </xs:restriction>
         </xs:simpleType>
      </xs:list>
   </xs:simpleType>
</xs:element>

this shows the built in facets for the XSD type xs:integer ("whiteSpace", "fractionDigits" and others).

I personally like this feature of XSModel serializer, that it is able to generate certain hidden properties of XML Schema components, which the schema authors normally don't specify while writing the schema documents for applications.

3. I provided the following XSD Schema fragment to XSModel serializer (a complexType referring to a model group),

<xs:element name="E1">
  <xs:complexType>
     <xs:group ref="gp1"/>
  </xs:complexType>
</xs:element>
   
<xs:group name="gp1">
   <xs:sequence>
      <xs:element name="x" type="xs:string"/>
      <xs:element name="y" type="xs:string"/>
   </xs:sequence>
</xs:group>

and the XSModel serializer generated the following round-trip serialization result,

<xs:element name="E1">
   <xs:complexType>
      <xs:sequence>
         <xs:element name="x" type="xs:string"/>
         <xs:element name="y" type="xs:string"/>
      </xs:sequence>
   </xs:complexType>
</xs:element>

<xs:group name="gp1">
   <xs:sequence>
      <xs:element name="x" type="xs:string"/>
      <xs:element name="y" type="xs:string"/>
   </xs:sequence>
</xs:group>

The global "model group" is serialized as expected. But the complexType within the element declaration was serialized with it's element declarations expanded. The lexical group reference is not present in the serialized output.

At first this may look odd (i.e the absence of the model group reference) in the serialized output. But the fact is, that Xerces XSModel instance in it's complete compiled form, doesn't know whether a group particle (in this case xs:sequence) comes from a group reference. And I had to live with this XSModel serialization characteristic. But the serialized schema output in this example is equivalent to the original schema document (which was supplied to the XSModel serializer) from validation perspective (but the global group definition in the output in this case is redundant from validation perspective, and it's just a characteristic of the XSModel serializer currently).

That's all I have to say now. Thanks for reading this post.

Dealing with multiple roots within an XML Schema

2011-06-04T06:59:00.005+05:30

I've been thinking on this problem for a while, and have collected some opinions, which I'm presenting here.

We'll be working with the following XML Schema documents:

a.xsd [1]

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="x" type="xs:string"/>

    <xs:element name="y" type="xs:string"/>

    <xs:element name="z">
       <xs:complexType>
          <xs:sequence>
             <xs:element ref="x"/>
             <xs:element ref="y"/>
          </xs:sequence>
       </xs:complexType>
    </xs:element>

</xs:schema>

b.xsd [2]

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:include schemaLocation="c.xsd"/>

    <xs:element name="z">
       <xs:complexType>
          <xs:sequence>
             <xs:element ref="x"/>
             <xs:element ref="y"/>
          </xs:sequence>
       </xs:complexType>
    </xs:element>

</xs:schema>

c.xsd [3]

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="x" type="xs:string"/>

    <xs:element name="y" type="xs:string"/>

</xs:schema>

The schema documents [1] and [2] are equivalent for the purpose of validating an XML instance document (it's just that the schema document b.xsd includes c.xsd).

Our application requires the following XML document to be successfully validated, by the schemas [1] or [2] above:

z.xml [4]

<z>
   <x>hello</x>
   <y>world</y>
</z>

All of this is just fine, and XML document [4] get's successfully validated by the schemas [1] or [2] above.

But the above schema design (either [1] or [2]), may present following problems sometimes:

The side effect of schema documents [1] or [2] is to also successfully validate the following XML documents,

Since elements "x" and "y" are also valid roots defined in the schema (due to the global declarations of elements "x" and "y" in the schema). But the purpose of defining elements "x" and "y" in the schema, is to include them by reference else where in the schema document (as in element declaration "z" in schemas [1] or [2]).

This kind of schema design is sometimes necessary, for the reasons of modularity (for e.g using one declaration at multiple places) and re-usability (for e.g. by including a foreign schema in our own schema) -- this design can be more beneficial, if the complexity of the schema (for e.g with more schema components, and more & deep nesting of schema components) is more.

So how do we live with following design trade-off,
i.e having schema like [1] or [2] above (which gives us benefits of modularity and re-usability) and also a side effect of these schema documents validating multiple root elements (which risks an application to accept invalid XML documents -- in this example, the roots "x" and "y" are invalid for the application, while the root "z" is valid).

In this use case, if we desire that the application must reject XML documents with roots "x" or "y" but should accept documents with root "z", then to my opinion this problem cannot be solved completely with XML Schema language (there's no way currently in the XML Schema language, to forbid validating the top level XML element in instance document, with a global schema element declaration).

Solving this problem would require a little bit of non schema solution (for e.g a SAX java add-on along with schema validation).

Here's a sketch of a java SAX application which can be and-ed with the XML Schema validation (using schemas above), to achieve the desired overall XML validation effect (i.e successful validation for the root element "z" and prohibiting the XML roots "x" and "y"),

(java imports are omitted to keep the text short)

class SAXUtil extends DefaultHandler {

     String[] excludedElems = new String[] {"x", "y"};

     private boolean isRootElemOK(String docUri) {  
        boolean rootElemOk = true;
  
        try {
           SAXParserFactory saxParserFactory = SAXParserFactory.newInstance();
           saxParserFactory.setNamespaceAware(true);
           SAXParser saxParser = saxParserFactory.newSAXParser();
           saxParser.parse(docUri, this);
        }
        catch(SAXException ex) {
           if (ex instanceof RootElementSAXException) {
              RootElementSAXException expObj = (RootElementSAXException) ex;
              if ("100".equals(expObj.getErrCode())) {
                 rootElemOk = false;  
              }
           }
        }
  
        return rootElemOk;  
     }

     public void startElement(String uri, String localName, String qName, Attributes attributes) throws SAXException {
        if (!isElementAllowed(localName)) {
           throw new RootElementSAXException("100"); 
        }
        throw new RootElementSAXException("101");
     }

     private boolean isElementAllowed(String localName) {    
        boolean elemAllowed = true;       
        for (int elemIdx = 0; elemIdx < excludedElems.length; elemIdx++) {
           if (localName.equals(excludedElems[elemIdx])) {
              elemAllowed = false;
              break;
           }
        }       
        return elemAllowed;       
     }

     class RootElementSAXException extends SAXException {
        String errorCode;
  
        public RootElementSAXException(String errorCode) {
           this.errorCode = errorCode;
        }
  
        public String getErrCode() {
           return errorCode;
        }
     }

} // class SAXUtil

Following is an algorithmic summary of the above java validation add-on,

1) A SAX parser is instantiated and parsing is invoked/triggered with the parse() method.

2) The SAX parser cannot go beyond parsing the root element -- the algorithm is intentionally designed in this way (since the SAX "startElement" callback method would always throws an exception [user defined exception, RootElementSAXException], upon encountering the top most element). The constructor parameter to the exception ("100" or "101" in this case) RootElementSAXException determines, whether the top most element was allowed or not (which is determined by an element name forbidden-list "excludedElems", defined in the above java class).

Notes:

1) To terminate the SAX parsing prior to completing parsing the whole of XML document, a SAXException can be thrown from the SAX call back methods. A custom exception class (like RootElementSAXException in the above example) is desirable, to distinguish our application designed exception from the built in SAXException events.

2) It's recommended to use SAX API for this use case, since it'll be much more efficient than for e.g using DOM APIs, which would load the whole of XML document in memory (which doesn't look a sensible approach to me, for just knowing the name of top most element of XML document).

3) The exclude element name list can be externalized from the java application, to make the above program reusable for any kind of XML documents.

4) We may use something like the java JAXP validation APIs, to help achieve the "and" of the two validation steps (i.e, schema validation and the SAX application step) described here, if we want to integrate this approach in a java application.

5) The java code snippet presented above can be made XML namespace aware (i.e if the XML elements are in namespace), by considering the namespace name parameter in SAX callback methods (for e.g the method parameter "String uri", in the startElement callback method).

I hope that this post is useful.

2011-06-26:
The explanation given by me in this blog post originally, seems to convey that multiple global element declarations in XML Schema documents are allowable by the XML Schema language, and this is inherently a bad/wrong design present within the XML Schema language. One of the solution to prohibit certain XML elements to be global in an XML instance document, was presented earlier in this blog post (using an additional restricted SAX parsing step in an application).

All this is fine. But I wanted to follow up on my thoughts written earlier in this post, arguing now, that multiple global element declarations allowable in XML Schema language is not a bad/wrong design present in XML Schema language. One of the features of XML Schema language, which requires multiple global element declarations is XML Schema "substitution groups" (i.e one element substituting for another) -- and "substitution groups" is a core and important concept within XML Schema language.

Of-course, if not working with XML Schema "substitution groups" or otherwise, one could use the SAX add-on technique I presented earlier to prohibit certain global element declarations to validate the XML instance root element, if that suits someones application design.

XML Schema: facets constraining the cardinality of simpleType->list

2011-04-30T12:33:00.002+05:30

I thought I should write a little clarification of a point I mentioned in my blog post, http://mukulgandhi.blogspot.com/2010/10/xsd-11-xml-schema-design-approaches.html.

I seem to have suggested in the above cited post, that XML Schema 1.1 assertions are probably necessary to impose restrictions on cardinality of an XML Schema simpleType list instance. But this fact doesn't appear to be true, after I realized this reading the XML Schema spec lately; which allows the following constraining facets on XML Schema simpleType's with variety list:
[1]
<xs:length ../>
<xs:minLength ../>
<xs:maxLength ../>

(ref, http://www.w3.org/TR/xmlschema11-2/#defn-coss which says, "If {variety} is list, then the applicable facets are assertions, length, minLength, maxLength, pattern, enumeration, and whiteSpace")

These constraining facets [1], on simpleType with variety list were available in XML Schema 1.0 too.

These facets [1] may serve the design purpose (and should probably be even efficient than using assertions, since assertions require compiling the XPath expressions in their "test" attribute's, and to build quite a bit of context information for XPath expression evaluation) I had mentioned in the above cited post.

Also to mention, that an assertion facet for simpleType with variety list, could be found useful for other purposes (i.e they are not without purpose!), for example as follows:

<xs:assertion test="count($value) mod 2 = 0"/>

(the list instance must have even number of items)

Thanks for reading this post!

Happy New Year 2011

2011-01-01T09:13:00.002+05:30

I wish readers of this blog a very Happy New Year 2011.

My new year resolutions are to have more interactions with the online community, particularly with folks at XML, XML Schema, XSL and XQuery forums. And I do wish to see W3C-standards progress on XML Schema 1.1, XPath 3.0, XSLT 3.0 and XQuery 3.0 languages (these are great new XML languages which I'm following-up with recently). I'm also reading through the discussions at the newly setup HTML-XML convergence task force (I hope we'll see few nice decisions emerging there)!

And needless to mention, I'm looking to work more closely with Eclipse community.

Schema based XML compare

2010-12-28T17:30:00.001+05:30

David A. Lee (producer of XMLSH -- A command line shell for XML) raised an interesting discussion a while ago on XML-DEV mailing list, about how to do XML Schema aware XML document comparison. The whole of this discussion thread can be read here. Michael Kay suggested to use the XPath 2.0 function deep-equal (where the input document trees need to be validated by a schema -- to enable type-aware comparison, before doing a comparison by this function) for this kind of use case. Following Michael's idea I was playing with this concept using IBM's XPath 2.0 engine (which is XML Schema aware and is a component of WebSphere Application Server feature pack for XML). For the interest of readers, here's a minimal Java program illustrating this program design.

package com.ibm.xpath2;

import javax.xml.namespace.QName;
import javax.xml.transform.stream.StreamSource;

import com.ibm.xml.xapi.XDynamicContext;
import com.ibm.xml.xapi.XFactory;
import com.ibm.xml.xapi.XPathExecutable;
import com.ibm.xml.xapi.XSequenceCursor;
import com.ibm.xml.xapi.XSequenceType;
import com.ibm.xml.xapi.XStaticContext;

public class XMLCompare {

    public static void main(String[] args) throws Exception {
        String dataDir = System.getProperty("dataDir.path");
  
        XFactory factory = XFactory.newInstance();
        factory.setValidating(XFactory.FULL_VALIDATION);
        factory.registerSchema(new StreamSource(dataDir + "/test.xsd"));
        
        XStaticContext staticContext = factory.newStaticContext();
        staticContext.declareVariable(new QName("doc1"), factory.getSequenceTypeFactory().                      documentNode(XSequenceType.OccurrenceIndicator.ONE));
        staticContext.declareVariable(new QName("doc2"), factory.getSequenceTypeFactory().                                      documentNode(XSequenceType.OccurrenceIndicator.ONE));
        XDynamicContext dynamicContext = factory.newDynamicContext();
        dynamicContext.bind(new QName("doc1"), new StreamSource(dataDir + "/test1.xml"));
        dynamicContext.bind(new QName("doc2"), new StreamSource(dataDir + "/test2.xml"));
                
        XPathExecutable executable = factory.prepareXPath("deep-equal($doc1, $doc2)", staticContext);
        XSequenceCursor result = executable.execute(dynamicContext);
        if (result.exportAsList().get(0).getBooleanValue()) {
           System.out.println("deep-equal == true");
        }
        else {
           System.out.println("deep-equal == false");
        }
    }
}

Following are the XML and XML Schema documents used for the above example.

test1.xml

<test>10.00</test>

test2.xml

<test>10</test>

test.xsd

<?xml version="1.0" encoding="UTF-8"?>
<schema xmlns="http://www.w3.org/2001/XMLSchema">
  <element name="test" type="double" />
</schema>

For the above examples, if the schema type of element node "test" is xs:double then both the XML documents above are reported deep-equal (since the values 10 and 10.00 are same double values, and the element node was annotated with schema type xs:double and deep-equal function did a type aware comparison of XML documents). But if say the schema type of element node "test" is xs:string, then the XML documents shown above would be reported not deep-equal.

I hope that this post is useful.

Apache Xerces-J 2.11.0 released

2010-12-18T20:00:00.001+05:30

I am happy to extend the announcement made by Apache Xerces team few days ago, for the release of new version of Xerces-J (2.11.0) (ref http://markmail.org/message/oom75s3wpebfywh5). This Xerces release specifically improves compliance to the XML Schema 1.1 language (the detailed release notes for Xerces-J 2.11.0 are available at, http://xerces.apache.org/xerces2-j/releases.html).

On behalf of Xerces team I hope that this Xerces-J release would be found useful by the XML and XML Schema community.

Refrences to XML Schema language:
1. http://www.w3.org/XML/Schema (XML Schema WG Home Page)
2. http://www.w3.org/TR/xmlschema11-1/ (XML Schema 1.1 Structures specification)
3. http://www.w3.org/TR/xmlschema11-2/ (XML Schema 1.1 Datatypes specification)

XML Schema 1.1: complexType restriction rules

2010-11-27T09:01:00.075+05:30

I've been excited enough to write now about the new rules that have been specified in XML Schema 1.1 spec regarding type derivations between XML Schema complexType definitions and what is Xerces-J's (it's XML Schema 1.1 engine) current compliance about this area of XML Schema language. In this blog post I'm currently covering XML schema complexType restriction derivations. I'll try to write about complexType extensions sometime later. I thought that this post might find audience interested in this topic (anyone is invited to write a comment to this blog post, which will help me to learn more about type derivations between XML schema complex types -- "i'm interested in both restriction and extension derivations", and can also give Xerces team useful feedback to improve Xerces in desired and compliant ways). Below are my findings from the XML Schema 1.1 spec about this topic, and also Xerces's compliance status in this regard (I acknowledge that my understanding may yet not be complete about these areas of the XML Schema language :).

In XML Schema 1.0 language complex type restriction derivation rules are defined by schema particle restriction rules specified here, http://www.w3.org/TR/xmlschema-1/#coss-particle. There's a 5x5 table in this section which describes what constitute valid restrictions (and what schema type restrictions are forbidden) of XML schema particles.

In XML Schema 1.1 all of these complexType derivation rules are replaced by sections 3.4.6.3 Derivation Valid (Restriction, Complex) and 3.4.6.4 Content Type Restricts (Complex Content). In XML Schema 1.1 a mapping table (the 5x5 table) for particle restrictions is removed, and now a generic algorithm of subsumption relationship (a kind of containment or association relationship) of default bindings (which is an abstract notion for element and attributes declarations along with wild-card attributes "strict", "lax" and "skip") is specified. The XML Schema 1.1 complexType subsumption rules are simpler and easy to remember, than the corresponding type derivation rules from XML Schema 1.0 spec. My personal understanding so far is that, the improved default binding particle subsumption rules in XML Schema 1.1 make XML Schema 1.1 complexType restriction derivations largely compatible with corresponding type derivation rules in XML Schema 1.0, but the rules are now specified with better wordings.

Below are various XML schema complexType restriction cases I've studied so far (and these have corresponding implementations in Xerces; the upcoming Xerces-J 2.11.0 release would have these features), the characteristics of which are also described and I'm trying to discover more of the rules in these areas of XML Schema language.

xs:sequence, xs:choice and xs:all are possible compositors (which signify the notion of how we can compose schema particles in XML schema complexType definitions) in schema complexType's.

A) SEQUENCE TO SEQUENCE RESTRICTIONS
a.1 xs:element is derived from xs:any wild-card (both of these particles are part of an XML Schema sequence compositor). In this scenario cardinality of particles takes precedence than presence of a concrete element in derived type, when determining valid particle derivations.

For e.g <xs:element name="x" type="xs:string" minOccurs="0"/> is not a valid restriction of <xs:any processContents="lax" />, since the effective cardinality of element "x" (minOccurs="0" means that particle "x" is optional) is more than that of the wild-card particle (is mandatory).

a.2 There must be a similar (i.e X-to-X where X is a positive numerical value) mapping of particles from a schema 'base' to 'derived' type. i.e a derived type cannot have less number of particles than those in base type, and a particle in derived type must validly derive (i.e is subsumed validly as per rules specified in XML Schema 1.1 spec) from the corresponding particle in base schema type.

B) ALL TO SEQUENCE RESTRICTIONS
b.1 This is a valid schema compositor (and of particles in them) restriction (i.e ordered from unordered restriction).

For e.g sequence(b, a) and sequence(a, b) {order of particles in derived type doesn't matter} are valid restrictions of all(a, b).

b.2 Identity of particles (recognized by QName of the particles) is recognized by the XML schema validator, and corresponding such particles must obey rules of restriction by cardinality (i.e an optional characteristic of particle does not make particle a valid restriction of a mandatory particle, where QName's of corresponding such particles in base and derived types are same).

C) ALL TO ALL RESTRICTIONS
c.1 This is an unordered to unordered kind restriction. Concrete element particle is an valid derivation of a wild-card particle.

c.2 Cardinality of identical particles (having same QName's) in derived type must be same or less (which makes the derived particle validly derive from the corresponding particle from base type) than that in base type. Particle cardinalities take precedence over generic/concrete relationship between particles, when determining valid particle subsumptions.

c.3 Number of leaf particles (which are essentially xs:element and xs:any wild-card's) in derived and base types must be equal.

D) SEQUENCE TO ALL RESTRICTIONS
This is not a valid schema compositor restriction (i.e from ordered to unordered).

E) CHOICE TO SEQUENCE RESTRICTIONS
e.1 Here are few examples explaining some of the rules for this category.

  <xs:sequence>
     <xs:element name="c" type="xs:string" />
  </xs:sequence>

is a valid restriction of

  <xs:choice>  
     <xs:any processContents="lax" />
     <xs:element name="b" type="xs:string" />
  </xs:choice>

(the element particle "c" is subsumable by the wild-card)

e.2

  <xs:sequence>
     <xs:any processContents="lax" />
  </xs:sequence>

is not a valid restriction of

  <xs:choice>         
     <xs:element name="a" type="xs:string" />
     <xs:element name="b" type="xs:string" />
  </xs:choice>

This is so because a wild-card is not a valid subsumption of an element particle (i.e generic derivations from concrete elements is not a valid restriction, which in fact looks like an "type extension" concept).

F) SEQUENCE TO CHOICE RESTRICTIONS
Here's an example I can think over that correspond to use case of such kinds.

   
   <xs:restriction base="TYPE_BASE">
      <xs:choice>
         <xs:group ref="myGroup" />
      </xs:choice>
   </xs:restriction>
   
   is a valid restriction of
   
   <xs:complexType name="TYPE_BASE">
      <xs:group ref="myGroup" />
   </xs:complexType>
   
   <xs:group name="myGroup">
      <xs:sequence>
         <xs:element name="a" type="xs:string" />
         <xs:element name="b" type="xs:string" />
      </xs:sequence>
   </xs:group>

But this is not a useful schema type restriction, since the result of choice (i.e the schema particle produced from xs:choice) in derived type results only in one option, which is same as the contents of the sequence of the base type.

Other than the above example I cannot envision any other useful example for practical scenarios for "sequence to choice" restriction. I would imagine that schema authors must not bother much about "sequence to choice" restriction scenarios, as this doesn't looks a good and useful schema design scenario (but I don't deny that people may find valid uses of this as well :).

G) CHOICE TO CHOICE RESTRICTIONS
Here are few of the examples I can think of that satisfy this use-case (these I've found to be working fine with Xerces as well):

g.1 choice(a, c) is not a valid restriction of choice(a, b). Because element "c" in derived type doesn't have a corresponding element particle in the base type.

g.2
- choice(a, b) is a valid restriction of choice(a, wild-card processContents="lax"). If the wild-card can resolve to an element declaration that doesn't match element declaration "b", then this is NOT-A-VALID restriction.
- choice(a, b) is a valid restriction of choice(a, wild-card processContents="strict") if wild-card can resolve to an element decleration for "b" OTHER-WISE not.

g.3 choice(group name="myGroup", a) is a valid restriction of choice(group name="myGroup", xs:any processContents="lax"). Here model group instance is considered as a particle. But if the wild-card resolves to an element declaration that doesn't match element declaration "a", then this is NOT-A-VALID restriction.

g.4 choice(group name="myGroup", a) is not a valid restriction of choice(group name="myGroup", <xs:any/>). But this is a valid restriction if wild-card <xs:any> can find definition of element "a" which can derive (i.e is a valid subsumption) to element "a" in the derived type.

These are all the cases I can think of at the moment (enumerated A to G) which might occur for restriction between XML Schema 1.1 complexType's. I believe there would be few more complexType restriction cases which I'll try to post on this blog as I discover them.

I hope that this post was useful.

XSD: schema type definition for empty XML content models

2010-10-23T17:10:00.042+05:30

I'm inclined to write a little post, suggesting a correction (perhaps a better schema design) to an XML schema document I wrote in the blog post, http://mukulgandhi.blogspot.com/2010/07/xsd-11-xml-schema-design-approaches.html [1].

In this post [1], I suggested the following XML schema type definition for empty content models (I assume there would not be any attributes on an element):

  <xs:complexType name="EMPTY"> 
     <xs:complexContent> 
        <xs:restriction base="xs:anyType" /> 
     </xs:complexContent> 
  </xs:complexType>

Instead of the above schema type definition, I find the following (which is simpler I believe) schema type definition [2] (intending to constrain an XML element) to be better instead:

  <xs:element name="X">
    <xs:complexType/>
  </xs:element>

The element definition [2] above intends to validate an XML fragment like following:

<X/>

In the above example, I intend to suggest that there must not be any child nodes (and neither any XML attributes on an element) within element "X". Interestingly (nothing new really for people knowing XML schema language :) the XML Schema language, only allows constraining XML element and attribute nodes (and optionally these being XML namespace aware) and it doesn't bother about other XML infoset components like comments, processing-instructions and so on (which are present in XPath data model for example) [A] -- this means that any other kinds of nodes, than XML elements and attributes are ignored by XML Schema language and a compliant XML schema validator. This nature [A] of XML schema language is OK as I've learnt (there have been some nice discussions about all of this at XML-DEV list in recent past).

2010-10-26: Here's another variant for definition of empty XML content models.

  <xs:simpleType name="EMPTY">
     <xs:restriction base="xs:string">
        <xs:maxLength value="0"/>
     </xs:restriction>
  </xs:simpleType>

This defines an XML schema 'simpleType' -- and enforces content emptiness with the schema 'maxLength' facet on type xs:string, instead of a complex type as defined in the previous example. I'm more inclined to define element emptiness by an simpleType like above, since intent (and semantics) of schema simple types is never to define XML attributes, but those of complexType are.

I hope the corrections I've shared in this post is appreciated by folks who've read my earlier post cited above [1].

XSD 1.1: XML schema design approaches cotd... PART 4

2010-10-10T22:15:00.059+05:30

In this blog post i'm trying to describe (I find the subject matter here interesting enough to have a new blog post!) few more XML Schema (i'm trying to cook-up XSD 1.1 examples :) use-cases - using largely XSD 1.1 assertions which are now solvable with XML Schema 1.1 (for example constraining cardinality of XML Schema xs:list items as described below), and as per my view-point couldn't be solved with XML Schema 1.0.

I hope, XML Schema community might find few of the things here interesting.

This post can be considered the PART 4 of the XML Schema 1.1 design series that I started couple of weeks ago. The previous parts of this series are available here:

1) PART 1
2) PART 2
3) PART 3

I'm using latest XML Schema 1.1 code-base from Xerces-J SVN repos.

Use-case: (A)
The examples in this post illustrate, how we can constrain the cardinality of XML Schema 1.1 xs:list instance members, and optionally constraining (just to verify myself how XSD 1.1 assertions behave in various combinations) few aspects of list members (like for example that, list items need to be even integers).

Here's an XML instance document (this describes a simple enough list of integers encapsulated in an XML element "X"), which I'll use for illustrations in this post:

[XML 1] (named temp.xml)

  <X>2 4 6 5 10 3</X>

Below are few XML Schema 1.1 examples (with Schema 1.1 instructions highlighted with different color), and explanations from my point of view thereafter:

[XML Schema 1]

  <?xml version='1.0'?>
  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
   
    <xs:element name="X">
       <xs:complexType>
         <xs:simpleContent>
            <xs:restriction base="INT_LIST">
              <xs:assertion test="count($value) le 5" />
            </xs:restriction>
         </xs:simpleContent>
       </xs:complexType>
    </xs:element>
   
    <xs:complexType name="INT_LIST">
       <xs:simpleContent>
         <xs:restriction base="xs:anyType">
            <xs:simpleType>
               <xs:list itemType="xs:integer" />          
            </xs:simpleType>
            <xs:assert test="every $x in $value satisfies ($x mod 2 = 0)" />
         </xs:restriction>
       </xs:simpleContent> 
    </xs:complexType>

  </xs:schema>

[XML Schema 2]

  <?xml version='1.0'?>
  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
   
     <xs:element name="X">
        <xs:simpleType>
          <xs:restriction base="INT_LIST">
             <xs:assertion test="$value mod 2 = 0" />
          </xs:restriction>
        </xs:simpleType>
     </xs:element>
   
     <xs:simpleType name="INT_LIST">
       <xs:list itemType="xs:integer" />
     </xs:simpleType>

  </xs:schema>

[XML Schema 3]

  <?xml version='1.0'?>
  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
   
    <xs:element name="X">
      <xs:complexType>
        <xs:simpleContent>
          <xs:extension base="INT_LIST">
             <xs:assert test="count($value) le 5" />
          </xs:extension>
        </xs:simpleContent>
      </xs:complexType>
    </xs:element>
   
    <xs:simpleType name="INT_LIST">
       <xs:list itemType="xs:integer" />
    </xs:simpleType>

  </xs:schema>

Here are results of XML instance (of document [XML 1]) validation, with the specified schema's:

1. When XML document ([XML 1]) is validated by the schema [XML Schema 1], we get following validation outcomes with Xerces:
[Error] temp.xml:1:20: cvc-assertion.3.13.4.1: Assertion evaluation ('every $x in $value satisfies ($x mod 2 = 0)') for element 'X' with type 'INT_LIST' did not succeed.
[Error] temp.xml:1:20: cvc-assertion.3.13.4.1: Assertion evaluation ('count($value) le 5') for element 'X' with type '#anonymous' did not succeed.

2. When XML document ([XML 1]) is validated by the schema [XML Schema 2], we get following validation outcomes (with Xerces):
[Error] temp.xml:1:20: cvc-assertion.3.13.4.1: Assertion evaluation ('$value mod 2 = 0') for element 'X' with type '#anonymous' did not succeed. Assertion failed for an xs:list member value '5'.
[Error] temp.xml:1:20: cvc-assertion.3.13.4.1: Assertion evaluation ('$value mod 2 = 0') for element 'X' with type '#anonymous' did not succeed. Assertion failed for an xs:list member value '3'.

3. When XML document ([XML 1]) is validated by the schema [XML Schema 3], we get following validation outcomes (with Xerces):
[Error] temp.xml:1:20: cvc-assertion.3.13.4.1: Assertion evaluation ('count($value) le 5') for element 'X' with type '#anonymous' did not succeed.

Here's some quick analysis from my point of view, with regards to what I wanted to achieve with these use-cases (A):

The XML Schema 1.1 assertions XPath 2.0 context variable "$value" has a type annotation xs:anyAtomicType*.

1. The first validation result (1. above) illustrates that every item of xs:list needs to be an even integer, and number of list items are constrained to be maximum "5" (this is a sample "max" limit on number of list items).

2. I intended to use validation results 2. and 3. in combination doing an boolean "AND" of them, essentially to have same XML instance validation objective as case 1. The boolean "AND" of two schema validations can be achieved with for example, Java JAXP validation API. I wrote XML Schema document, [XML Schema 2] to have the XML Schema validator return each individual list item, which do not pass test of mathematical evenness (this was not entirely achieved with schema document [XML Schema 1] -- where the schema detected an evenness failure for whole list instance, but didn't report every individual list item which didn't pass evenness test).

I hope the intent of the use-case described here, and the solutions offered are explained clear enough for XML Schema audience.

Thanks for reading, and as usual I hope that this blog post was interesting!

XSD 1.1: Xerces-J implementation updates

2010-09-05T22:48:00.037+05:30

Over the past one or two months, there have been few interesting changes happening at Xerces-J XML Schema 1.1 implementation. I feel obliged to share these enhancements with the XML Schema community, and also with folks at Eclipse WTP (where we enhanced few "schema aware" components of PsychoPath XPath 2.0 engine, to support these recent Xerces enhancements -- I think we improved the design of typed values of XML element and attribute XDM nodes in PsychoPath XPath2 engine, in case the XDM node has a type annotation of kind XML Schema simpleType, with varieties list or union).

Here's a summary of XML Schema 1.1 implementation changes that have recently been completed with Xerces (available at Xerces SVN repos as of now), which are planned to be part of the Xerces-J 2.11.0 release, planned to take please during November 2010 time frame.

1. Xerces-J now has a complete implementation of XML Schema 1.1 conditional inclusion functionality. The Xerces-J 2.10.0 release had implementation of XML Schema 1.1 conditional inclusion vc:minVersion and vc:maxVersion attributes. Xerces-J now supports all of "conditional inclusion" attributes as specified by the XML Schema 1.1 spec. The "conditional inclusion" attributes that are now newly supported in Xerces-J are: vc:typeAvailable, vc:typeUnavailable, vc:facetAvailable and vc:facetUnavailable. All of XML Schema 1.1 built-in types and facets are now supported by Xerces-J related to XML Schema 1.1 "conditional inclusion" components.

2. There are few interesting changes that have happened to Xerces-J XML Schema 1.1 assertions implementation as well, that are planned to be part of Xerces-J 2.11.0 release. Xerces now has an improved assertions evaluation processing on XML Schema (1.1) simple types, with varieties 'list' and 'union'.

2.1 Enhancements to assertions evaluation on simpleType -> list:

Here's an example of XML Schema 1.1 assertions on an xs:list schema component:
[XML Schema 1]

   <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

      <xs:element name="Example" type="EXAMPLE_LIST" />
   
      <xs:simpleType name="EXAMPLE_LIST">
         <xs:list>
            <xs:simpleType>
               <xs:restriction base="xs:integer">
                  <xs:assertion test="$value mod 2 = 0" />
               </xs:restriction>
            </xs:simpleType>
         </xs:list>
      </xs:simpleType>
   
   </xs:schema>

If an XML instance document has a structure something like following:
[XML 1]
<Example>1 2 3</Example>

And if this XML instance document ([XML 1]) is validated by the above XML schema ([XML Schema 1]), Xerces-J would report error messages like following (assuming the name of XML document was, test.xml):
[Error] test.xml:1:25: cvc-assertion.3.13.4.1: Assertion evaluation ('$value mod 2 = 0') for element 'Example' with type '#anonymous' did not succeed. Assertion failed for an xs:list member value '1'.
[Error] test.xml:1:25: cvc-assertion.3.13.4.1: Assertion evaluation ('$value mod 2 = 0') for element 'Example' with type '#anonymous' did not succeed. Assertion failed for an xs:list member value '3'.

An assertion must evaluate on every 'simpleType -> list' item (which is validated by the itemType of xs:list) in an XML instance document. Xerces now does this, and needed error messages are displayed in case of schema assertion failures.

2.2 Enhancements to assertions evaluation on simpleType -> union:

Here's an example of XML Schema 1.1 assertions on an xs:union schema component:
[XML Schema 2]

   <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
   
      <xs:element name="Example">
         <xs:simpleType>
            <xs:union memberTypes="MYDATE xs:integer" />
         </xs:simpleType>
      </xs:element>
   
      <xs:simpleType name="MYDATE">
         <xs:restriction base="xs:date">
            <xs:assertion test="$value lt current-date()" />
         </xs:restriction>
      </xs:simpleType>

   </xs:schema>

If an XML instance document has a structure something like following:
[XML 2]
<Example>2010-12-05</Example>

And this instance document is validated by the schema document, [XML Schema 2] the following error message is displayed by Xerces:
[Error] temp.xml:1:30: cvc-assertion.union.3.13.4.1: Element 'Example' with value '2010-12-05' is not locally valid. One or more of the assertion facets on an element's schema type, with variety union, have failed.

Xerces tried to validate an atomic value '2010-12-05' both with schema types xs:integer and MYDATE. Since none of these types could successfully validate this atomic value, and an assertion failed in the process of these validation checks, the relevant assertion failure was reported by Xerces.

If the XML schema, [XML Schema 2] tries to validate the XML instance document:
<example>10</Example>

no validation failures are reported in this case, since an atomic value '10' conforms to the schema type xs:integer, which results in an overall validation success of the atomic value with an 'union' schema type.

I'm ending this blog post now. Stay tuned for more news here :)

And I hope, that this post was useful.

XSD 1.1: XML schema design approaches cotd... PART 3

2010-07-17T08:24:00.009+05:30

I'm continuing with the XML Schema design thoughts series, with the third part here. The first two parts are available here:
1) PART 1
2) PART 2

All the examples here have been tested with Xerces-J 2.10.0.

(I'm disclaiming in the beginning, that examples presented in this blog post are somewhat fictitious and may not serve a real life use-case. These examples are kind of cooked-up to only illustrate XML Schema 1.1 constructs, and some of design thinking behind them. I also refer at lot of places a phrase "element particles". This simply means XML elements, but "particles" is a formal term defined by the XML Schema spec, designating XML schema components having minOccurs and maxOccurs attributes -- if minOccurs/maxOccurs attributes are absent, then these have default values for the relevant schema components)

I'm presenting a sample 1.1 XML schema with corresponding XML document first, and then attempting trying to reflect on the inherent design from my point of view in these examples:

XML Schema 1.1 specific constructs are emphasized with a different color.

[XML1]

  <Book>
     <name>XML in a Nutshell</name>
     <ISBN>AB-1001</ISBN>
     <author>Jimmy</author>
     <NoPages>100</NoPages>
  </Book>

[XML Schema 1]

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
    
     <xs:element name="Book">
        <xs:complexType>
          <xs:complexContent>
            <xs:extension base="BOOK_FRAGMENT">
               <xs:openContent>
                 <xs:any processContents="lax" />
               </xs:openContent>
               <xs:assert test="not(* except (name, author, ISBN, NoPages)) and 
                                 (if (ISBN)
                                    then not(ISBN/*) 
                                    else true()) and 
                                 (if (NoPages) 
                                     then (not(NoPages/*) and (NoPages/text() castable as xs:positiveInteger))
                                     else true())" />    
            </xs:extension>
          </xs:complexContent>          
        </xs:complexType>
     </xs:element>
   
     <xs:complexType name="BOOK_FRAGMENT">
        <xs:sequence>
          <xs:element name="name" type="xs:string" />
          <xs:element name="author" type="xs:string" />
        </xs:sequence>
     </xs:complexType>

  </xs:schema>

The following use-case requirements motivated me to write this sample (I'm also trying to reflect on the schema design choices I've made, about which I surely invite comments from the readers -- if you've patience to read this post and respond!):
1. XML Schema 1.0 has a limitation that, when a complex type (having sequence or choice particles) is derived by extension then a derived complex type can only add element particles at the end of an element list (within the base type). Supposing that we want to re-use a complex type (having a sequence of element particles) by deriving it with extension, and need to add additional element particles say any-where in between the elements of the base type. This is what the above XML schema (XML Schema 1) example intends to do; and the above schema does indeed validates successfully the corresponding XML document presented above (XML1).

2. A key design decision in the above schema (XML Schema 1) is to use the XML Schema 1.1 "openContent" instruction (newly introduced in 1.1 version). The use of XSD 1.1 assertions here is optional, but is very practical to do so (which I'll try to explain!). An XML schema "openContent" instruction is essentially a wrapper around xs:any wild-card, producing the same effect as xs:any wild-card but has an interleave or a suffix appending behavior (please feel free to read the XML Schema 1.1 spec to learn more about XSD 1.1 open contents. Or perhaps if you want a lighter [but brilliant] explanation, you may read Roger L. Costello's XML Schema 1.1 write-up available here).
The XML Schema 1.1 spec defines an "openContent" instruction as following:

  <openContent
     id = ID
     mode = (none | interleave | suffix) : interleave
     {any attributes with non-schema namespace . . .}>
     Content: (annotation?, any?)
  </openContent>

It is an openContent instruction with "interleave" mode (which is the default openContent mode), which enables adding additional element particles interspersed between base type's element particles.

3. In the above example, the XML elements "ISBN" and "NoPages" are added to the base type's element particles which are not appended at the end of base type's elements, but can be added anywhere within the resulting XML content model. For this particular example, the placement of XML elements coming from the derived complex type are arbitrary, and is done to only illustrate the workings of "openContent" instruction in "interleave" mode.

4. It's interesting to see the benefit of XSD 1.1 assertions here. The assertions here are able to impose certain constraints on the resultant content model (otherwise the content model is kind-of wide open with no restrictions). The assertions in the above schema document (XML Schema 1) mean:
a) The resulting content model can only have XML elements -> "name", "author", "ISBN" and "NoPages".
b) The element "ISBN" needs to be an atomic string value, and the element "NoPages" needs to be an xs:positiveInteger value.

I'm presenting below another XML schema variant (than the example above -- XML Schema 1), which solves the same problem as described above, but in a slightly different way (with advantages and disadvantages described after the example):

[XML Schema 2]

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
    
      <xs:element name="Book">
         <xs:complexType>
            <xs:complexContent>
               <xs:extension base="BOOK_FRAGMENT">
                  <xs:openContent>
                     <xs:any processContents="strict"/>
                  </xs:openContent>
                  <xs:assert test="count(distinct-values(for $elem in (* except (name, author)) return $elem/name())) = count(for $elem in (* except (name, author)) return $elem/name())"/>       
               </xs:extension>
            </xs:complexContent>          
         </xs:complexType>
      </xs:element>
   
      <xs:complexType name="BOOK_FRAGMENT">
         <xs:sequence>
           <xs:element name="name" type="xs:string"/>
           <xs:element name="author" type="xs:string"/>
         </xs:sequence>
      </xs:complexType>
   
      <xs:element name="ISBN" type="xs:string" />
   
      <xs:element name="NoPages" type="xs:positiveInteger" />

   </xs:schema>

The example XML document for this schema (XML Schema 2) remains same (XML1). Here are the advantages (and unfortunately a little disadvantage as well, with a suggested workaround for the drawback...) of the sample, XML Schema 2:
1. Here we are using xs:any wild-card with processContents="strict" mode (the earlier example used the wild-card with "lax" mode) and providing the corresponding element declarations in the schema (the last two element declarations). This approach has advantage that, the content model of elements "ISBN" and "NoPages" are enforced natively by the XML schema engine, and the schema author doesn't have to implement the content model constraints herself/himself (for example, that an element is empty and has an atomic value) -- say via assertions. This approach is more robust, than trying to achieve the similar effect with assertions.

2. The assertion in schema document, [XML Schema 2] enforces that elements in the sequence could occur only once. This is accomplished by this simple algorithm:
count(distinct-values(names...)) = count(names...)

3. The only drawback I foresee with XML Schema 2, is that elements "ISBN" and "NoPages" are now global elements (which is necessary to have xs:any wild-card to work with processContents="strict" mode). This has implication that following XML documents would be reported valid as well, by the schema document XML Schema 2:

  <ISBN>AB-1001</ISBN>

AND

  <NoPages>100</NoPages>

This is a side-effect of schema document XML Schema 2, which I myself personally don't seem to like :(

To solve this limitation, I can imagine there could be a workaround as following:
We could perform two validations in sequence. One with the schema document, [XML Schema 2] (let's call this validation V1) and the second one with the following schema document (let's call this validation result V2):

[XML Schema 3]

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
      
      <xs:element name="ISBN" type="xs:string" />
   
      <xs:element name="NoPages" type="xs:positiveInteger" />

  </xs:schema>

This is kind of a little validation pipeline. The complete/end-to-end (which usually means, that this has domain meaning) schema validation succeeds in entirety, if validation V1 succeeds but V2 doesn't (I imagine, that this kind-of pipeline operation could be enforced by a host language, like Java using the XML Schema JAXP APIs).

Thanks for reading!

I hope that this post is useful.

XSD 1.1: XML schema design approaches cotd... PART 2

2010-07-11T10:00:00.009+05:30

I'm continuing with the XML Schema design approaches series, I started in the previous blog post. Here's the second post in this series.

Here's a description of the use-case I'll be illustrating in this post, with both XML Schema 1.0 and 1.1 examples:

We need to write an XML Schema for the following XML content model:

  colors
    -> (violet | indigo | blue | green | yellow | orange | red)+

Here the words "colors", "violet" etc represent XML elements, and they have no attributes and are empty. The above content model means, that children of element "colors" can repeat and are unordered, and at-least one of them is required.

Therefore following XML document is a valid instance according to this content model:

[XML1]

  <colors>
     <violet/>
     <indigo/>
     <blue/>
     <green/>
     <yellow/>
     <orange/>
     <red/>
  </colors>

AND for example, the following XML document is valid as well, as per the content model described above (here the element "colors" have less children than the previous example, and some of children of "colors" occur more than once):

[XML2]

  <colors>
     <violet/>
     <indigo/>
     <blue/>
     <green/>
     <green/>
  </colors>

Here are two XML schema examples that express the above XML content model constraints:

[XML Schema 1] (written in XML Schema 1.0)

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
    
     <xs:element name="colors">
        <xs:complexType>
           <xs:choice maxOccurs="unbounded">
              <xs:element name="violet" type="EMPTY" />
              <xs:element name="indigo" type="EMPTY" />
              <xs:element name="blue" type="EMPTY" />
              <xs:element name="green" type="EMPTY" />
              <xs:element name="yellow" type="EMPTY" />
              <xs:element name="orange" type="EMPTY" />
              <xs:element name="red" type="EMPTY" />     
           </xs:choice>
        </xs:complexType>
     </xs:element>
   
     <xs:complexType name="EMPTY"> 
        <xs:complexContent> 
          <xs:restriction base="xs:anyType" /> 
        </xs:complexContent> 
     </xs:complexType>

  </xs:schema>

[XML Schema 2] (written in XML Schema 1.1 -- the 1.1 specific constructs are displayed with a different color)

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
    
     <xs:element name="colors">
        <xs:complexType>
           <xs:sequence>
             <xs:any maxOccurs="unbounded" processContents="lax" />
           </xs:sequence>
           <xs:assert test="every $x in */name() satisfies ($x = 
                              ('violet','indigo','blue','green','yellow','orange','red'))" />
           <xs:assert test="every $x in * satisfies not($x/node())" />
        </xs:complexType>
     </xs:element>

  </xs:schema>

Here's some quick analysis from my point of view, about the differences between the above schema approaches, and if any of the above approaches is better than the other one:
1) "XML Schema 1" is written in a familiar 1.0 style, so people who want to stick with 1.0 can still adopt this technique. We can observe, that the first schema is a little more verbose than the second one, which I see as one of the advantage of the second one.

2) If you are comfortable writing the XPath 2.0 expressions, then there are virtually too many possibilities to express schema validation constraints with XSD 1.1 assertions, which is really lots of power in the hands of the schema author!

3) Personally speaking, I find the second way of writing the XML schema ("XML Schema 2") a really cool NEW way to express these validation constrains. I'm not suggesting that the 1st way is not really good! That technique has great value, in it's own sense and has stood the tests of time. I find the second technique a more natural description from the schema author, to express the logic of the use-case in question.

4) One the possibilities I now foresee with XML Schema 1.1, is that schema author could impose quite a bit of constraints on xs:any wild-card instruction via assertions (which is particularly useful with processContents="lax" mode of the xs:any wild-card). A point worth observing is that with processContents="strict" mode of the xs:any wild-card, assertions are not really useful because, the schema validator would strictly validate the XML element with an element declaration, which must be provided by the schema author to satisfy the processContents="strict" mode of the wild-card (and assertions here would actually interfere with the available element declarations, which to my opinion is not a good design). With processContents="skip" mode of the xs:any wild-card, assertions would always fail (and the XML instance would become invalid), because the concerned XML elements would be discarded by the XML schema validator, and consequently these elements would not be part of the XPath data-model tree, on which assertions operate.

And needless to mention, Xerces-J handles all the above examples fine!

I hope that this post is useful.

XSD 1.1: XML schema design approaches in XSD 1.1 world... PART 1

2010-07-03T20:05:00.008+05:30

I'm thinking to write a series of posts (since writing too many ideas in one post could be boring to read, and could be quite voluminous for one post. I'll try to make sure, that these blog posts starting from this one have cross-references between them for related issues AND I'll convey in some future blog post, when I'm stopping writing this series!) only on XML schema design, given the XML Schema 1.1 constructs. I'll try to reflect why XML Schema 1.1 is essential for certain use-cases, and where XML Schema 1.0 falls short.

It is possible, that there may be a blog post unrelated to this series between these posts. When this series completes, I'll try to summarize the ideas at the end, to make the whole series available as an unit.

I'm disclaiming in the beginning, that any advice offered here may not necessarily be best. Improvements are generally always possible! Any feedback would be great (about the correctness of anything described here, alternative ideas OR anything else).

To start with the 1st post in this series, here's a little background about the use-case I'm describing in the subsequent paragraphs:
I've been reading the book "DB2 pureXML Cookbook: Master the Power of the IBM Hybrid Data Server" [1] recently by Matthias Nicola (a member of DB2 pureXML team). This book describes an example (in chapter 2) as follows.
A physical object could be described by two possible XML content models as follows:

a) Metadata as values, aka Name/Value Pairs (often bad):

  <object type="car">
    <field name="brand" value="Honda" />
    <field name="price" value="5000" />
    <field name="currency" value="USD" />
    <field name="year" value="2002" />
  </object>

b) Metadata as element names (good):

  <car>
    <brand>Honda</brand>
    <price currency="USD">5000</price>
    <year>1996</year>
  </car>

I wouldn't describe here why one of the above XML design approaches is good or bad. This is described well in the book cited above [1], which I would encourage folks to read (the books has some nice explanation about DB2 pureXML as well).

Let's say we want to build an XSD schema, for the XML document (a) above. To start with, one of the design decisions I took is, to define a set of XML Schema types with a hierarchy as following:

  OBJECT
     -> OBJECT_ON_SALE
            -> CAR
            -> BOOK

The other few design decisions I've taken are as following:
1) We'll use XSD 1.1 type-alternatives to select between different schema types, depending on value of the attribute object/@type.
2) We'll use an hierarchy of XSD 1.1 assertions definitions, to enforce certain validation constraints.

The meaning of these design decisions would likely become clear to us, by looking at the XML instance and schema document I'm describing below:

The sample XML instance I propose is as following:
[XML1] (this is same as XML instance "a" above, and is repeated here for convenience)

  <object type="car">
     <field name="brand" value="Honda" />
     <field name="price" value="5000" />
     <field name="currency" value="USD" />
     <field name="year" value="2002" />
  </object>

OR

[XML2]

  <object type="book">
     <field name="title" value="XML in a Nutshell" />
     <field name="author" value="Jimmy" />
     <field name="author" value="Nick" />
     <field name="publisher" value="Prentice Hall" />
     <field name="price" value="15" />
     <field name="currency" value="USD" />
     <field name="year" value="2008" />
  </object>

I propose the following XSD 1.1 schema (the 1.1 specific constructs are highlighted with a different color), that is designed to validate both of above XML instance documents (after which I'll try to analyze few design elements of this schema):

[SCHEMA1]

  <?xml version="1.0" encoding="UTF-8"?>
  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
             xmlns:xerces="http://xerces.apache.org">
 
     <xs:element name="object" type="OBJECT">
       <xs:alternative test="@type='book'" type="BOOK" />
       <xs:alternative test="@type='car'" type="CAR" />
     </xs:element>
 
     <xs:complexType name="BOOK">
        <xs:complexContent>
           <xs:extension base="OBJECT_ON_SALE">
              <xs:assert test="field/@name = 'title' and
                               field/@name = 'author' and
                               field/@name = 'publisher' and
                               field/@name = 'year'"
                         xerces:message="For a book the fields title/author/publisher/year are mandatory" />
              <xs:assert test="xs:int(field[@name = 'year']/@value) gt 1900 and 
                               xs:int(field[@name = 'year']/@value) lt 2011"
                         xerces:message="A book's publication year must be between 1900 and 2011" />
              <xs:assert test="count(field[not(@name = 'author')]) = 
                               count(distinct-values(field[not(@name = 'author')]/string(@name)))" 
                         xerces:message="A book can have multiple authors, but none of other fields of a book can occur twice" />
           </xs:extension>
        </xs:complexContent>
     </xs:complexType>
 
     <xs:complexType name="CAR">
        <xs:complexContent>
           <xs:extension base="OBJECT_ON_SALE">
              <xs:assert test="field/@name = 'brand' and
                               field/@name = 'year'" 
                         xerces:message="For a car the fields brand/year are required" />
              <xs:assert test="xs:int(field[@name = 'year']/@value) gt 2000 and 
                               xs:int(field[@name = 'year']/@value) lt 2011" 
                         xerces:message="A car's manufacture year must be between 2000 and 2011" />
              <xs:assert test="count(field) = count(distinct-values(field/string(@name)))" 
                         xerces:message="None of the fields of an object 'car' can occur twice" />
           </xs:extension>
        </xs:complexContent>
     </xs:complexType> 
 
     <xs:complexType name="OBJECT_ON_SALE">
        <xs:complexContent>
          <xs:extension base="OBJECT">
             <xs:assert test="field/@name = ('price','currency')" 
                        xerces:message="An object that can be sold, must have the fields price/currency" />
             <xs:assert test="if (field/@name = 'price') then 
                              (field[@name = 'price']/xs:decimal(@value) gt 0 and
                              field/@name = 'currency')
                              else true()" 
                         xerces:message="If a field price is present, the currency field should exist. The value of price must be greater than 0." />
          </xs:extension>
        </xs:complexContent>
     </xs:complexType>
 
     <xs:complexType name="OBJECT">
       <xs:sequence>
         <xs:element name="field" minOccurs="0" maxOccurs="unbounded">
            <xs:complexType>
               <xs:attribute name="name" type="xs:string" />
               <xs:attribute name="value" type="xs:string" />
            </xs:complexType>
         </xs:element>
       </xs:sequence>
       <xs:attribute name="type" type="xs:string" />    
     </xs:complexType>

  </xs:schema>

The key design elements in the above schema document are the inheritance hierarchy and assertions/type-alternatives. The element "object" is validated by the schema type "CAR" or "BOOK". The set of assertions applicable on the type CAR/BOOK are the assertions on this type, as well as assertions inherited from the base types. The schema type applicable on the element "object" is controlled by the type-alternative switch (which works upon the value of attribute "type").

When the XML document, [XML1] is validated (with Xerces-J 2.10.0 -- actually with the latest code-base on Xerces SVN as of today, because there was a minor bug [which affects this particular example] that got fixed few days after Xerces-J 2.10.0 got released) by the schema document [SCHEMA1] the validation succeeds (as there are no validation errors).

Let's try to introduce some data errors in the XML instance document, and see what happens upon validation with the same schema document.

Here's a modified XML instance document:
[XML3]

  <object type="car">
    <field name="price" value="-100" />
    <field name="currency" value="USD" />
    <field name="year" value="1999" />
  </object>

If the above XML document ([XML3] -- named "test.xml") is validated by the schema document, [SCHEMA1] we get following validation errors with Xerces-J:
[Error] test.xml:5:10: cvc-assertion.failure: Assertion failure. If a field price is present, the currency field should exist. The value of price must be greater than 0.
[Error] test.xml:5:10: cvc-assertion.failure: Assertion failure. For a car the fields brand/year are required.
[Error] test.xml:5:10: cvc-assertion.failure: Assertion failure. A car's manufacture year must be between 2000 and 2011.

If we remove all of xerces:message attributes from assertions above, the following error messages are printed by Xerces for the above scenario:
[Error] test.xml:5:10: cvc-assertion.3.13.4.1: Assertion evaluation ('if (field/@name = 'price') then (field[@name = 'price']/xs:decimal(@value) gt 0 and field/@name = 'currency') else true()') for element 'object' with type 'OBJECT_ON_SALE' did not succeed.
[Error] test.xml:5:10: cvc-assertion.3.13.4.1: Assertion evaluation ('field/@name = 'brand' and field/@name = 'year'') for element 'object' with type 'CAR' did not succeed.
[Error] test.xml:5:10: cvc-assertion.3.13.4.1: Assertion evaluation ('xs:int(field[@name = 'year']/@value) gt 2000 and xs:int(field[@name = 'year']/@value) lt 2011') for element 'object' with type 'CAR' did not succeed

It's up-to the user's, that which error format is appropriate for them (the error format without xerces:message prints more error-context information. While the format with xerces:message could be useful to print user-friendly error messages, upon assertions failure).

I won't describe in much detail now, the domain meaning of error messages above and the problem scenario itself. I believe, this fictitious problem domain is simple enough to understand these examples.

I would end this post now. I'll take up the case of schema type "BOOK" in a future blog post (I imagine, the concepts I'm trying to illustrate here for the domain object CAR would be similar for the object BOOK).

I'll try to write few more XSD 1.1 examples in subsequent posts!

I hope that this post is useful.

XSD 1.1: assertions failure error messages

2010-06-27T08:50:00.000+05:30

Here's some info for XML Schema users!

As refered on my earlier blog post (ref, http://mukulgandhi.blogspot.com/2010/04/xsd-11-xsprecisiondecimal-assertions.html), XSD 1.1 assertions error message was not a standard feature of the XML Schema language spec, and XML Schema WG was deliberating on this issue at that point of time.

As of now, the XML Schema WG has taken a decision on this, which is as following (which is an response from David Ezell, XML Schema WG chair):
<quote>
The WG believes that this topic should be covered in a separate note describing best practices for how to handle this issue. Liam suggests following the i18n practice of publishing "articles" to recommend best practice.

See http://www.w3.org/International/ for examples.
</quote>

Therefore, keeping in view this observation from XML Schema WG, the Xerces-J implementation would not change about assertions error messages. Or unless, there are any specific comments from Xerces users about assertions error reporting.

Xerces-J 2.10.0 released!

2010-06-19T09:55:00.001+05:30

I'm pleased to extend the announcement of Xerces-J 2.10.0 release (released today. ref, http://markmail.org/message/m73xwkrmyacppu3l).

Xerces-J 2.10.0 is now available on the Xerces site, http://xerces.apache.org/xerces2-j/ to be used by community. This is a release after nearly two and a half years of the previous Xerces-J release, 2.9.1. This is a significant milestone at Xerces, with lot's of new features & bug fixes/enhancements.

Xerces-J 2.10.0 provides two versions of the Xerces distributables:
1. Xerces2 Java 2.10.0 - this is a maintainence release for 2.9.1 release (having bug fixes & enhancements), with essentially the same parsing & API support as 2.9.1 (except I think addition of support for StAX 1.0 event API & Element Traversal API -- more details are available in Xerces release notes).
2. Xerces2 Java 2.10.0 (XML Schema 1.1) (Beta) - this release supports a partial experimental implementation of the XML Schema 1.1 Structures and Datatypes Working Drafts (December 2009), along with all the changes available in "Xerces2 Java 2.10.0" release (point 1).

PS: I would also like to extend special thanks to the Eclipse/PsychoPath XPath 2.0 team members, Dave Carver & Jesper S Møller for helping producing an excellent XPath 2.0 engine, which underlies Xerces XML Schema 1.1 assertions and type-alternatives implementation. Special thanks also to Andrea Bittau, who originally authored the PsychoPath engine.

I hope the community likes the Xerces 2.10.0 release!

Please feel free to discuss about Xerces-J at:

http://xerces.apache.org/mail.html#xerces-j-user

http://xerces.apache.org/mail.html#xerces-j-dev

XSD 1.1: schema versioning, and Xerces-J support

2010-05-27T06:35:00.004+05:30

XML Schema 1.1 provides a nice schema composition feature, called "Conditional inclusion" which allows us to include/exclude Schema components, during schema processing, based on values of certain special control attributes (minVersion & maxVersion), specified on the schema components.

Here are two simplistic examples, illustrating this feature:

Example 1

XML document [1]:
<test>3</test>

XSD 1.1 document [2]:
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
   xmlns:vc="http://www.w3.org/2007/XMLSchema-versioning">

   <xs:element name="test">
   <xs:simpleType>
   <xs:restriction base="xs:positiveInteger">
   <xs:assertion test="$value mod 2 = 0" vc:minVersion="1.1" />
   </xs:restriction>
   </xs:simpleType>
   </xs:element>

</xs:schema>

In the above schema document [2], the attribute vc:minVersion on xs:assertion instruction specifies, that the assertion instruction would only be processable by XSD processors, which support 1.1 and a higher level of the XSD schema language. If this schema document [2] is run by an XSD 1.1 (and possible a higher language version in future) processor in XSD 1.0 mode, the assertion instruction would be ignored by the XSD engine. The schema versioning features allows us to have a XSD engine, ignore certain schema components (in entirety along with their descendant instructions).

Example 2

XML document [3]:
<address ver="V2">
   <street1></street1>
   <street2>XX</street2>
   <city>XX</city>
   <state>XX</state>
   <country>XX</country>
</address>

XSD 1.1 document [4]:
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
   xmlns:vc="http://www.w3.org/2007/XMLSchema-versioning">

   <xs:element name="address" type="Address">
   <xs:alternative test="@ver = 'V2'" type="AddressV2" vc:minVersion="1.1" />
   </xs:element>

   <xs:complexType name="Address">
   <xs:sequence>
   <xs:element name="street1" type="xs:string" />
   <xs:element name="street2" type="xs:string" />
   <xs:element name="city" type="xs:string" />
   <xs:element name="state" type="xs:string" />
   <xs:element name="country" type="xs:string" />
   </xs:sequence>
   </xs:complexType>

   <xs:complexType name="AddressV2">
   <xs:complexContent>
   <xs:extension base="Address">
   <xs:attribute name="ver" type="xs:string" />
   <xs:assert test="not(normalize-space(street1) = '')" />
   </xs:extension>
   </xs:complexContent>
   </xs:complexType>

</xs:schema>

Similarly, the above schema document ([4]) ignores the type-alternative instruction, if the XSD 1.1 processor is run in a XSD 1.0 mode. I believe, the intent of the above schema and the XML document should be clear enough (we are using an "address" element in XML document, which needs to be validated by a corresponding XML Schema type. The complex type, "AddressV2" extends the type "Address", and has an assertion specification to constrain the contents of the element "street1" -- in this particular example, the assertion on type "AddressV2" constrains the element, "street1" to have some significant white-space characters).

Xerces-J runs these examples fine.

Summarizing this post: The XSD 1.1 schema versioning features, allows us to write a XSD schema containing mix and match of XSD 1.0 and 1.1 instructions (and XSD instructions beyond XSD 1.1 level, for future!), and have the XSD 1.1 engine ignore certain XSD instructions at run-time depending, at which XSD language level, the XSD 1.1 engine was invoked.

I hope that this post is useful.

XSD 1.1: negative "pattern" facets and assertions

2010-04-25T21:41:00.004+05:30

While exploring more of XSD 1.1 assertions, I've been pretty convinced that much of the limitations of XSD "pattern" facet can be overcome with assertions (and of-course one of real benefits of XSD 1.1 assertions is the ability to specify co-occurrence constraints, in XML Schema documents -- here's a nice article explaining XML Schema 1.1 co-occurrence constraints).

I think, one of the things which might get quite difficult to express in XML Schema 1.0, is specifying a negative word list.

For example, if we have this simple XML document:

  <fruit>apple</fruit>

And we want that, the XML element "fruit" must not contain say the words "cherry" or "guava". Although, this looks a pretty straight-forward regex use-case, but unfortunately it might get quite cumbersome to express this seemingly straightforward regex pattern, with the available XSD 1.0 regular-expression syntax.

My quick try to express this with XSD 1.0, was something like following:
<xs:pattern value="^(cherry|guava)" />

But unfortunately, the above pattern facet and quite a few similar regexes, can't accomplish this seemingly common use-case easily (I think, this is doable with XSD 1.0 regex's but certainly, it would be quite tedious to come to the right regex pattern -- of-course regex experts/gurus could do this easily, but not me at this moment!).

And now, I try to express these validation constraints with XSD 1.1 assertions. Here's a sample XSD 1.1 schema [1], using assertions to solve this, and few of similar use-cases:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

     <xs:element name="Example" type="Fruits1" />
  
     <xs:complexType name="Fruits1">
       <xs:sequence>
         <xs:element name="fruit" type="xs:string" />
         <xs:element name="exclude" type="xs:string" />
       </xs:sequence>
       <xs:assert test="not(fruit = tokenize(exclude,','))" />
     </xs:complexType>
  
     <xs:complexType name="Fruits2">
       <xs:sequence>
         <xs:element name="fruit" type="xs:string" />
         <xs:element name="exclude" type="xs:string" />
       </xs:sequence>
       <xs:assert test="not(fruit = (for $x in tokenize(exclude,',') return 
                                              normalize-space($x)))" />
     </xs:complexType>
  
     <xs:complexType name="Fruits3">
       <xs:sequence>
         <xs:element name="fruit" type="xs:string" />
         <xs:element name="exclude" type="xs:string" />
       </xs:sequence>
       <xs:assert test="not(fruit = (for $x in tokenize(exclude,',') return 
                                (string-join(tokenize($x,' '),''))))" />
     </xs:complexType>
  
   </xs:schema>

A sample XML instance document [2], that we'll validate with the above schema, is following:

  <Example>
    <fruit>apple</fruit>
    <exclude>cherry,guava</exclude>
  </Example>

As stated in the original requirements above, we want that the word in element "fruit" must not contain any of words, from the comma-separated list in the "exclude" element.

In the above XSD schema [1], the complex type "Fruits1" can successfully validate the above XML instance document [2].

The complex type "Fruits2" can validate an exclude list, where there could be white-spaces before and after the 'comma separator'. For example, the list "cherry, guava" (please note, an extra white-space after the 'comma') would be considered an appropriate exclusion list for this example. Whereas, this list variant cannot be validated by the schema type, "Fruits1".

And the complex type "Fruits3" can validate an exclude list of kind, "cherry, g u a v a" (i.e, there could be white-space characters, within a word) -- this is a figment of my imagination :). But certainly there could possibly be such lexical constraints in instance documents.

PS: All the examples in this post were tested with, Xerces-J.

I hope, that this post is useful.

XSD 1.1: xs:precisionDecimal, assertions and Xerces-J updates

2010-04-17T09:33:00.012+05:30

Section 1

Recently, I went through in sufficient detail about the XSD primitive data-type, xs:precisionDecimal (newly introduced in, XSD 1.1), and was trying to use XSD 1.1 assertions to simulate xs:precisionDecimal (just to vent my curiosity and exploring more of, XSD assertions) as a user-defined (as a restriction of xs:decimal data-type) XSD Simple Type (though I believe, a native implementation of xs:precisionDecimal should also exist in an XSD 1.1 implementation, or in language systems which may use the XSD type system -- for example, a stand-alone XPath (2.x) implementation which uses an XSD type system).

Here's an XSD 1.1 schema example, illustrating these concepts:
[1]

   <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

      <xs:element name="example" type="myPrecisionDecimal" />
  
      <xs:simpleType name="myPrecisionDecimal">
        <xs:restriction base="xs:decimal" xmlns:xerces="http://xerces.apache.org">
           <xs:totalDigits value="6" />
           <xs:fractionDigits value="4" />
           <xs:assertion test="string-length(substring-after(string($value), '.')) ge 2" 
                  xerces:message="minScale of this decimal number should be 2" />
        </xs:restriction>
      </xs:simpleType>
  
   </xs:schema>

The XSD type, "myPrecisionDecimal" defined above has following correspondences with the type, xs:decimal:
a) The facet specification, xs:totalDigits in "myPrecisionDecimal" is equivalent to the facet xs:totalDigits in xs:decimal.
b) The facet specification, xs:fractionDigits in "myPrecisionDecimal" is equivalent to the facet "maxScale" for, xs:decimal.
c) The assertion facet in, "myPrecisionDecimal" is equivalent (an user-defined attempt to equalize!) to the facet "minScale" for, xs:decimal.

When the above schema document [1], is used to validate the following XML instance:

<example>44.4</example>

The following error message is produced:
[Error] test.xml:1:24: cvc-assertion.failure: Assertion failure. minScale of this decimal number should be 2.

It's also worth noting that, the above user-defined type "myPrecisionDecimal" cannot be considered a true equivalent of XSD type, xs:precisionDecimal as defined in XSD 1.1 spec, because xs:precisionDecimal also includes values for positive and negative infinity and for "not a number", and it differentiates between "positive zero" and "negative zero" (these aspects, are not defined for xs:decimal). The above example, for "myPrecisionDecimal" only demostrates, simulating the "minScale" facet (which is not available in the type, xs:decimal) of xs:precisionDecimal.

Section 2
(Xerces-J, assertions implementation update)

Xerces-J recently implemented, an extension attribute "message" (specified in a namespace, http://xerces.apache.org, for Xerces-J XSD 1.1 implementation) on XSD 1.1 assertion instructions. The value of this attribute, needs to be an error message that will be reported by an XSD 1.1 engine upon assertions failure.

An example of this is illustrated, in the schema document above [1].

In the absence of the "message" attribute on assertions (or if it's present, but it doesn't contain any significant non-whitespace characters), the following default error message is produced by Xerces:
[Error] test.xml:1:24: cvc-assertion.3.13.4.1: Assertion evaluation ('string-l
ength(substring-after(string($value), '.')) ge 2') for element 'example' with type 'myPrecisionDecimal' did not succeed.

We could see the benefit of, the "message" attribute on assertions, which to my opinion are following:
a) For complex (& particularly, lengthy) XPath expressions in assertions, the default error messages produced by Xerces, could be quite verbose which the user's may not find convenient to view & debug. The user experience, with default assertions error messages, may be further trouble-some if there are numerous assertion evaluations for XML documents -- we could imagine the user-experience, for say maxOccurs="unbounded" specification on XML elements on which assertions apply OR let's say, there may be of the order of "> 10" different assertions.
b) We could specify, domain specific error messages with the assertions "message" attribute.

Though, the advantage of the default assertion error messages produced by Xerces is that, it prints to the user, the name of XSD type and the element/attribute involved in a particular assertions validation episode.

PS: There's been a recent issue raised with the XSD WG, which proposes addition of a "message" attribute on assertions in the XSD 1.1 language itself. The Xerces implementation of assertions "message" attribute may change in future, depending on a recommendation related to this, from the XSD WG.

I hope, that this post is useful.

playing again with XSD 1.1 assertions

2010-03-21T19:50:00.001+05:30

Some time ago, XSLT folks (including me!) were discussing on XSL-List the design of an XML schema, describing a product catalog. This post has nothing to do with XSLT, except that an earlier discussion on XSL-List enkindled me with yet another XSD schema use-case, to try out the Xerces-J XSD 1.1 assertions implementation. I wrote the following XSD 1.1 schema use-case, with a desire to find out if Xerces-J XSD 1.1 assertion implementation would succeed, for this example (and to cause no surprise to readers, I'm pleased to say, that Xerces passes this example!).

So here goes this example.

XML document:

  <?xml version="1.0" encoding="UTF-8" ?>
  <product id="100">
    <shortname>Sun Press, Java Book</shortname>
    <description>Java Language: Design and Programming</description>
    <author>James Gosling</author>
    <price>
      <value effective="2000-10-10" format="hard cover">25</value>
      <value effective="2005-10-10" format="hard cover">20</value>
      <value effective="2009-10-10" format="pdf" freeware="true">0</value>
    </price>
  </product>

An XSD 1.1 schema validating the above XML document:

  <?xml version="1.0" encoding="UTF-8" ?>
  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

   <xs:complexType name="Product">
     <xs:sequence>
       <xs:element name="shortName">
         <xs:simpleType>
           <xs:restriction base="xs:string">
             <xs:maxLength value="20"/>
           </xs:restriction>
         </xs:simpleType>
       </xs:element>
       <xs:element name="description" type="xs:string"/>
       <xs:element name="author" type="xs:string"/>
       <xs:element name="price">
         <xs:complexType>
           <xs:sequence>
             <xs:element name="value" maxOccurs="unbounded">
                <xs:complexType>
                  <xs:simpleContent>
                    <xs:extension base="xs:double">
                      <xs:attribute name="effective" type="xs:date" use="required"/>
                      <xs:attribute name="freeware" type="xs:boolean"/>
                      <xs:attribute name="format" use="required">
                        <xs:simpleType>
                          <xs:restriction base="xs:string">
                            <xs:enumeration value="hard cover"/>
                            <xs:enumeration value="pdf"/>
                          </xs:restriction>
                        </xs:simpleType>
                      </xs:attribute>
                      <xs:assert test="@effective lt current-date()" />
                      <xs:assert test="if (@freeware eq true()) then (@format eq 'pdf' and . eq 0)
                                       else true()" />                    
                    </xs:extension>
                  </xs:simpleContent>
                </xs:complexType>
             </xs:element>
           </xs:sequence>
           <xs:assert test="every $vl in value[position() lt last()] satisfies
                            ($vl gt $vl/following-sibling::value[1]) and 
                            ($vl/@effective lt $vl/following-sibling::value[1]/@effective)" />
         </xs:complexType>
       </xs:element>
     </xs:sequence>
     <xs:attribute name="ID" type="xs:positiveInteger" use="required"/>
   </xs:complexType>
   
   <xs:element name="product" type="Product"/>

  </xs:schema>

I don't wish to explain in detail the problem domain behind the above XML & XSD documents (I believe, readers familiar with XSD language & XML could easily understand the intent of the above example). In very shortest description, this example "illustrates a simple product catalog, describing a single product".
Here's a short explanation, about what the assertions -- highlighted with a different color (starting from assertion at top, to assertion at bottom) in above schema document are, intending to do:
1. The first assertion is checking, that the value of attribute "effective" (with a schema type, xs:date) is prior to today's date.
2. The second assertion is checking, that if value of attribute "freeware" is a boolean 'true', then value of attribute "format" must be 'pdf' & the numeric value of price should be 0.
3. The third assertion is checking, that price always reduces in future, & the effective date of the price is prior to the next price revision.

I enjoyed writing this example, and I'm glad that this worked with Xerces. The Eclipse/PsychoPath XPath 2.0 implementation (which is the underlying XPath 2 implementation, used by Xerces-J XSD 1.1 assertions implementation) also looks pretty compliant to the XPath 2 language.

I hope, that this post is useful.

Xerces-J: XSModel serialization

2010-03-07T07:52:00.001+05:30

There's a new API sample contributed to the Xerces-J code-base (in the schema-dev XSD 1.1, branch), which allows us to serialize a Xerces-J XSModel. This should be available in the upcoming Xerces-J release, 2.10.0.

This could be invoked by using the Java class, xs.XSSerializer.

Here's one of the use-case for this, as asked by colleagues in the community:
http://mail-archives.apache.org/mod_mbox/xerces-j-users/200611.mbox/%3c4557B05A.6000808@gael.fr%3e

XSD 1.1: some more assertions fun

2010-02-24T21:59:00.007+05:30

Here are some more XSD 1.1 assertions examples (interesting one's I guess), that I tried running with Xerces-J XSD 1.1 implementation (these ones run fine, with Xerces!):

Example 1 [1]:

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="test" type="X" />
   
    <xs:complexType name="X">
      <xs:group ref="List1" />
      <xs:assert test="a and b and d" />
    </xs:complexType>
   
    <xs:complexType name="Y">
      <xs:group ref="List1" />
      <xs:assert test="a and b and c and d" />
    </xs:complexType>
   
    <xs:group name="List1">
       <xs:sequence>
         <xs:element name="a" type="xs:string" minOccurs="0"/>
         <xs:element name="b" type="xs:string" minOccurs="0"/>
         <xs:element name="c" type="xs:string" minOccurs="0"/>
         <xs:element name="d" type="xs:string" minOccurs="0"/>
       </xs:sequence>
    </xs:group>
           
  </xs:schema>

The corresponding XML instance, document is:

<test>
    <a>hello</a>
    <b>world</b>
    <!--<c>hello..</c>-->
    <d>world..</d>
  </test>

Here's the rationale/goal, that motived me to write this XSD sample:
I wanted to define a pair of XSD complex types (something like, X & Y above), such that one of the types could reuse the element particles, from the other type. If this problem could have been solved with XSD type derivation (which I attempted initially), I wanted that only one of the elements in the derived type could become optional -- element, "c" in this example (i.e, with minOccurs = 0 & maxOccurs = 1), while the other elements from the base type should have the same occurrence indicator (i.e, a mandatory indicator -- which is, minOccurs = maxOccurs = 1).

Interestingly, this problem is unsolvable with XSD type derivation (either complex type extension, or restriction mechanism).

For this schema use-case, I came up with the XSD sample above [1], which meets my goal to be able to re-use the element particles in the XSD types. The Schema above [1], defines a global group which contains a sequence of XML element definitions. All of the elements in the group, are marked as optional. Within the complex types (X & Y), the cardinality of elements (0-1 or 1-1) is enforced with XSD assertions. Defining all elements in the group, as optional allows us to reuse this list in different XSD types easily, as we can constrain the elements (say controlling the cardinality of elements, or even the contents of elements/attributes) in different contexts/types say using, assertions.

Using the above schema example [1], therefore if one wants to use a XSD type, where element "c" is optional, one would use the type, "X". While if, one wants to use a XSD type, where all elements are mandatory, one would use the type, "Y".

After having solved the use-case I had in mind (explained above), so just for fun, I wrote another schema using some more assertions.

Here's the 2nd XSD schema:

Example 2 [2]:

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

     <xs:element name="test" type="X" />
   
     <xs:complexType name="X">
       <xs:group ref="List1" />
       <xs:assert test="a and b and d" />
     </xs:complexType>
   
     <xs:complexType name="Y">
       <xs:group ref="List1" />
       <xs:assert test="a and b and c and d" />
     </xs:complexType>
   
     <xs:group name="List1">
        <xs:sequence>
           <xs:element name="a" minOccurs="0">
             <xs:complexType>
               <xs:sequence>
                 <xs:element name="a1" type="xs:string" maxOccurs="unbounded" />
               </xs:sequence>
               <xs:attribute name="aCount" type="xs:nonNegativeInteger" />
               <xs:assert test="count(a1) eq @aCount" />
             </xs:complexType>
           </xs:element>
           <xs:element name="b" type="xs:string" minOccurs="0"/>
           <xs:element name="c" type="xs:string" minOccurs="0"/>
           <xs:element name="d" type="xs:string" minOccurs="0"/>
        </xs:sequence>
     </xs:group>
           
  </xs:schema>

The schema [2] is conceptually similar, to schema [1]. The only difference between the two schemas is, that in schema [2], element "a" has complex content, while in schema [1], element "a" is defined to have simple content (which is, xs:string). In schema, [2]'s complex type we define another assertion (which enforces the constraint that, value of attribute "aCount" is equal to the number of, "a1" children of element, "a"). The assertion definition in the complex type of element, "a" in the 2nd schema, is written only to visually increase the complexity of the element a's definition (of-course, this also does increase the functional complexity of element, "a" and subsequently the complexity of contents of the global group definition, in the 2nd schema).

The 2nd schema illustrates, that a more functionally complex list of particles (a, b, c & d here) get more benefit by the schema component re-use technique (accomplished with a XSD group, and assertions) illustrated in this post.

I hope, that this post is useful.

Xerces-J, XSD 1.1 assertions: complexType -> simpleContent -> restriction

2010-02-14T20:57:00.007+05:30

XSD 1.1 complex types are specified by the grammar given here, in the XSD 1.1 spec:
http://www.w3.org/TR/xmlschema11-1/#declare-type

XSD complex type definitions are essentially composed of three mutually exclusive definitions, as follows:

  <complexType ...
    simpleContent |
    complexContent |
    openContent?, (group | all | choice | sequence)?, ((attribute | attributeGroup)*, anyAttribute?), assert*))
  </complexType>

The assertions specification in complexType -> simpleContent -> restriction is a bit different, that all other assertions cases on complex types (as this consists of assertion facets, as well as/or assertions on the complex type).

This is specified by the following XSD 1.1 grammar:

  <simpleContent
    id = ID
    {any attributes with non-schema namespace . . .}>
    Content: (annotation?, (restriction | extension))
  </simpleContent>

  <restriction
    base = QName
    id = ID
    {any attributes with non-schema namespace . . .}>
    Content: (annotation?, (simpleType?, (minExclusive | minInclusive | maxExclusive | maxInclusive | totalDigits | fractionDigits | maxScale | minScale | length | minLength | maxLength | enumeration | whiteSpace | pattern | assertion | {any with namespace: ##other})*)?, ((attribute | attributeGroup)*, anyAttribute?), assert*)
  </restriction>

The XSD definition for xs:restriction above specifies assertions something like following:
assertion*, ..., assert*

Here, xs:assertion (with cardinality, 0-n) is a facet for the simple type value (specified by, complexType -> simpleContent). Whereas, xs:assert (with cardinality, 0-n) is an assertion definition on the complex type (which has access to the element tree, like the XML element itself, and it's attributes if there are any). xs:assertion definitions on, complexType -> simpleContent -> restriction do not have access to the element tree (on which the complex type is applicable), and can only access the simple type value (using, the implicit assertion variable $value, having a XSD type specified by the definition, <xs:restriction base = QName ...) of the element in the context.

Here's a small fictitious examples, illustrating these concepts:

XML document [1]:

  <A a="15">Example A</A>

XSD 1.1, Schema [2]:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="A">
      <xs:complexType>
        <xs:simpleContent>
          <xs:restriction base="myBase">    
            <xs:assertion test="contains($value, 'Example')" />
            <xs:assert test="@a mod 5 = 0" />    
          </xs:restriction>
        </xs:simpleContent>
      </xs:complexType>
    </xs:element>
  
    <xs:complexType name="myBase">
      <xs:simpleContent>
         <xs:extension base="xs:string"> 
           <xs:attribute name="a" type="xs:int" />  
         </xs:extension>
      </xs:simpleContent> 
    </xs:complexType>

  </xs:schema>

In the Schema above [2], there are two assertions (shown with bold emphasis) specified on the XSD type. One of assertions is a facet for the simple content, and the other is an assertion on the complex type.

I believe, the above Schema is simple enough and self-explanatory, to illustrate the points I've tried to explain in this post.

Actually, what prompted me to write this post, was that there was a minor bug in complexType -> simpleContent -> restriction facet processing in Xerces-J XSD 1.1 SVN code, which we could fix today, and the fix is now available in Xerces-J SVN repository.

Interestingly, this fix was there in Xerces-J SVN during some past Xerces SVN version. But going forward with assertions development, this bug got introduced, and now has been fixed again.

PsychoPath XPath2 processor update: fn:name() function fix

2010-02-06T10:13:00.002+05:30

While writing following blog post, http://mukulgandhi.blogspot.com/2010/01/xsd-11-wild-cards-in-compositor-and.html (dated, Jan 31, 2010) [1], I actually unearthed a bug in PsychoPath XPath 2 processor, whereby the XPath2 fn:name() function didn't evaluate properly with zero arity (it raised a "context undefined" exception, even if a context item existed).

This bug led me to use the, fn:local-name() (whose implementation was correct) function instead, for the above mentioned blog post [1].

The good news is, that now this bug with fn:name() function is fixed (ref, https://bugs.eclipse.org/bugs/show_bug.cgi?id=301539).

For the example given in the blog post [1], the given XSD 1.1 assertion could now be written like following, as well:

  <xs:assert test="(*[1]/name() = ('fname', 'lname')) and 
                 (*[2]/name() = ('fname', 'lname'))" />

(instead, of the "local-name" function as used in the mentioned blog post [1])

C. M. Sperberg-McQueen: slides about XSD 1.1

2010-02-05T21:41:00.001+05:30

I just came across this brief (but sufficient enough to give a good overview) slide presentation about XML Schema (XSD) 1.1, by C. M. Sperberg-McQueen:

http://www.blackmesatech.com/2009/07/xsd11/

Nice ones indeed, and highly recommended.

XSD 1.1: wild-cards in xs:all compositor, and assertions

2010-01-31T16:15:00.005+05:30

I was reading through the latest XSD 1.1 language draft, and one of the things that has changed between XSD 1.0 and 1.1, are some of the details of, xs:all compositor instruction.

XSD 1.1 defines <xs:all ..> compositor as follows:

  <all
    id = ID
    maxOccurs = 1 : 1
    minOccurs = (0 | 1) : 1
    {any attributes with non-schema namespace . . .}>
    Content: (annotation?, (element | any)*)
  </all>

Whereas, XSD 1.0 defined xs:all instruction as following:

  <all
    id = ID
    maxOccurs = 1 : 1
    minOccurs = (0 | 1) : 1
    {any attributes with non-schema namespace . . .}>
    Content: (annotation?, element*)
  </all>

XSD 1.1 allows xs:any wild-card to be part of xs:all (whereas, XSD 1.0 didn't allow this), which makes xs:all instruction to be more useful (because, with xs:any we could make the Schema type more open). We could also have certain Schema constraints present as assertions (as illustrated in the example below), restricting the degree of Schema openness (achieved by xs:any wild-card) to an extent we would want.

Here's an example I came up with, illustrating the use of xs:any wild-card within xs:all compositor, and having some assertions, for imposing some constraints on the ordering of elements (which means, that we are restricting the degree of openness achieved by xs:any using assertions) in the instance document:

XML document [1]:

  <Person>
    <fname>Mukul</fname>
    <lname>Gandhi</lname>
    <sex>M</sex>
    <address>
      <street1>xyz</street1>
      <street2>street</street2>
      <street2>gurgaon</street2>
    </address>
  </Person>

XSD 1.1 Schema [2]:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="Person">
      <xs:complexType>
        <xs:all>
          <xs:element name="fname" type="xs:string" />
          <xs:element name="lname" type="xs:string" />
          <xs:element name="sex" type="xs:string" />
          <xs:any processContents="lax" />
        </xs:all>
        <xs:assert test="(*[1]/local-name() = ('fname', 'lname')) and 
                         (*[2]/local-name() = ('fname', 'lname'))" />
      </xs:complexType>
    </xs:element>
  
  </xs:schema>

In the above Schema [2], if the assertions are not present, then xs:all compositor would mean, that it's contents can be present in any order (including the xs:any wild-card, which is newly introduced in XSD 1.1 within xs:all).

The assertion in the Schema above [2], constrains the first two child elements of, "Person" element to be "fname" or "lname".

Xerces-J's XSD 1.1 processor, seems to implement these syntax fine.

I hope that this post is useful.

XQuery 1.0: Full axis feature

2010-01-20T22:16:00.003+05:30

I was just reading through the XQuery 1.0 language (I'm reading, Priscilla Walmsley's excellent book on XQuery), and found an interesting point specified in XQuery 1.0.

The XQuery 1.0 spec says (Ref):
<quote>
[Definition: The following axes are designated as optional axes: ancestor, ancestor-or-self, following, following-sibling, preceding, and preceding-sibling.]

[Definition: A conforming XQuery implementation that supports the Full Axis Feature MUST support all the optional axes.]
</quote>

Giving a little thought about this, I felt that making certain XPath axes (and many of the useful ones, as specified above) as optional in XQuery language is most likely not correct, and would certainly trouble users, who want to uses these XPath axes out-of-the-box from an XQuery engine.

Interestingly, the latest working draft of the XQuery language (the, 1.1 version) has fixed this design mistake (I think, this was a design mistake!), and doesn't specify such a constraint in XQuery language.

Trang: DTD to XSD conversion

2010-01-17T10:32:00.005+05:30

This morning, I was playing with James Clark's utility which can do conversion between different XML Schema languages.

The utility is, Trang and it's described at:
http://www.thaiopensource.com/relaxng/trang.html

Here's a little example I tried with Trang (a DTD to XSD conversion).

DTD input:

  <!ELEMENT note (to,from,heading,body)>
  <!ELEMENT to (#PCDATA)>
  <!ELEMENT from (#PCDATA)>
  <!ELEMENT heading (#PCDATA)>
  <!ELEMENT body (#PCDATA)>

Here's the XSD output I got from, Trang:

  <?xml version="1.0" encoding="UTF-8"?>
  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified">
    <xs:element name="note">
      <xs:complexType>
        <xs:sequence>
          <xs:element ref="to"/>
          <xs:element ref="from"/>
          <xs:element ref="heading"/>
          <xs:element ref="body"/>
        </xs:sequence>
      </xs:complexType>
    </xs:element>
    <xs:element name="to" type="xs:string"/>
    <xs:element name="from" type="xs:string"/>
    <xs:element name="heading" type="xs:string"/>
    <xs:element name="body" type="xs:string"/>
  </xs:schema>

As per Trang documentation, following combination of conversions are possible:
RELAX NG (both XML and compact syntax) & DTD
to
RELAX NG (both XML and compact syntax), DTD & XSD

I found this good, and can recommend this.

XSD 1.1: Open contents, with Xerces-J

2010-01-16T17:24:00.003+05:30

Here's some further update to XSD 1.1 feature implementation in Xerces-J.

Another significant feature, that's been added in XSD language (in, 1.1 version) is Open contents. It's defined on XSD complex types, and also at the schema level. A nice explanation about XSD 1.1 Open contents, is available in an article at, http://www.ibm.com/developerworks/xml/library/x-xml11pt3/index.html#N1034D.

The wild-card instruction xs:any in XSD 1.0 was a way to implement XSD Open contents. XSD 1.1 introduces two new instructions, for implementing open contents namely, <xs:openContent> and <xs:defaultOpenContent>. These new XSD instructions have more options for the XSD schema authors, to write open schemas.

I've been able to test couple of Open content examples, with current Xerces-J SVN code base, and Xerces-J implements them, sufficiently fine.

XPath 2.0: PsychoPath processor update

2010-01-16T12:57:00.001+05:30

I just noticed release of the Eclipse, WTP (Web Tools Platform) 3.2M4 milestone (released on, December 11th, 2009). The WTP Source Editing release, in WTP 3.2M4 milestone, includes an enhanced PsychoPath XPath 2.0 engine.

Xerces-J users using the XSD 1.1 support (specifically XSD 1.1 assertions and CTA/type alternatives), can use the latest PsychoPath library from http://www.eclipse.org/webtools/. The latest development PsychoPath library can also be downloaded, from https://build.eclipse.org/hudson/view/WTP/job/cbi-wtp-wst.xsl.psychopath/.

Dave Carver, has been doing a great job in putting this all together.

Xerces-J: more XSD 1.1 tests; negative wild-cards

2010-01-09T17:28:00.018+05:30

I'm pretty satisfied with the XSD 1.1 assertions and CTA (type alternatives) implementation (as I've been writing few posts about them, earlier on this blog), in current Xerces-J SVN code base (though, a user feedback would be great, for the Xerces project. Instructions to report bugs in Xerces-J can be found at, http://xerces.apache.org/xerces2-j/jira.html).

I'm now beginning to test some of other XSD 1.1 features. To start with these new set of posts, following are few use cases for "Negative wildcards" (ref, http://www.ibm.com/developerworks/xml/library/x-xml11pt3/index.html#N101C9), which I've found to be working fine with Xerces-J.

XSD 1.0 had following XML representation of, xs:any wild-card Schema component:

  <any
    id = ID
    maxOccurs = (nonNegativeInteger | unbounded)  : 1
    minOccurs = nonNegativeInteger : 1
    namespace = ((##any | ##other) | List of (anyURI | (##targetNamespace | ##local)) )  : ##any
    processContents = (lax | skip | strict) : strict
    {any attributes with non-schema namespace . . .}>
      Content: (annotation?)
  </any>

("anyAttribute" is another wild-card Schema component)

XSD 1.1 enhances the xs:any wild-card definition to following:

  <any
    id = ID
    maxOccurs = (nonNegativeInteger | unbounded)  : 1
    minOccurs = nonNegativeInteger : 1
    namespace = ((##any | ##other) | List of (anyURI | (##targetNamespace | ##local)) ) 
    notNamespace = List of (anyURI | (##targetNamespace | ##local)) 
    notQName = List of (QName | (##defined | ##definedSibling)) 
    processContents = (lax | skip | strict) : strict
    {any attributes with non-schema namespace . . .}>
      Content: (annotation?)
  </any>

As we could notice, xs:any now allows (in XSD 1.1, which was not available in XSD 1.0) two additional specifiers in it's definition, namely "notNamespace" and "notQName".

Here's a fictitious example of usage of "notNamespace" specifier:

XML document, [1]:

  <Example xmlns="http://www.example.com/mySample">
    <a>hi there</a>
    <b>hi there ..</b>
    <c>hi there ...</c>
    <d xmlns="http://www.notallowed.com/sorry">hi there ....</d>
  </Example>

XSD 1.1 Schema, [2]:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
             targetNamespace="http://www.example.com/mySample"
             elementFormDefault="qualified">

    <xs:element name="Example">
      <xs:complexType>
        <xs:sequence>
          <xs:element name="a" type="xs:string" />
          <xs:element name="b" type="xs:string" />
          <xs:element name="c" type="xs:string" />
          <xs:any notNamespace="http://www.notallowed.com/sorry"
                  processContents="lax"/>
        </xs:sequence>
      </xs:complexType>
    </xs:element>
  
  </xs:schema>

The XSD schema, [2] defines an xs:any wild-card definition, which doesn't allow an XML instance document to have an element instance (allowed by the wild-card) to be in the namespace, "http://www.notallowed.com/sorry".

Therefore when the XML instance, [1] is validated by XSD document, [2] we get following error while performing validation with Xerces-J XSD 1.1 schema engine:
test.xml:5:46:cvc-complex-type.2.4.a: Invalid content was found starting with element 'd'. One of '{WC[##other:"http://www.notallowed.com/sorry"]}' is expected.

If we change the instance document, to specify element "d" to following:
<d>hi there ....</d>
or say,
<d xmlns="http://www.allowed.com">hi there ....</d>

the XSD validation, succeeds (as element "d" is now not in the namespace, "http://www.notallowed.com/sorry").

Here's an example for usage of "notQName" specifier:

XML document, [3]:

  <Example xmlns="http://www.example.com/mySample">
    <a>hi there</a>
    <b>hi there ..</b>
    <c>hi there ...</c>
    <XX>hi there ....</XX>
  </Example>

XSD 1.1 Schema, [4]:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
             xmlns:tns="http://www.example.com/mySample"
             targetNamespace="http://www.example.com/mySample"
             elementFormDefault="qualified">

    <xs:element name="Example">
      <xs:complexType>
        <xs:sequence>
          <xs:element name="a" type="xs:string" />
          <xs:element name="b" type="xs:string" />
          <xs:element name="c" type="xs:string" />
          <xs:any notQName="tns:XX"
                  processContents="lax"/>
        </xs:sequence>
      </xs:complexType>
    </xs:element>
  
  </xs:schema>

The XSD schema, [4] doesn't allow an instance document to have, an element "XX" (in namespace, "http://www.example.com/mySample"), where the xs:any wild-card allows an element content.

Therefore, when XML document [3] is validated by XSD schema, [4] we get following error message, with Xerces-J:
test.xml:5:7:cvc-complex-type.2.4.a: Invalid content was found starting with element 'XX'. One of '{WC[##any, notQName(tns:XX)]}' is expected.

So if we, replace the offending element, with:
<abc>hi there ....</abc>
or say,
<XX xmlns="http://www.allowed.com">hi there ....</XX>
(here, the local-name in XML instance document, is same as that specified in the "notQName" specifier, while the namespace of element instance is different, than specified by "notQName", which makes this element instance, valid)

the XML validation passes.

All these XSD language enhancements, in 1.1 version look cool (and, useful :)) to me, and they give some more XML validation capabilities to XSD schema, authors.

Out of my curiosity, I was thinking if we could write few of new XSD 1.1 wild-card capabilities, with assertions.

For e.g, some of the features of "notNamespace" attribute can be written with an assertion like following:

  <xs:assert test="not(namespace-uri(*[last()]) = (
                        'http://www.notallowed.com/sorry1',
                        'http://www.notallowed.com/sorry2',
                        'http://www.notallowed.com/sorry3')
                       )" />

But using assertions for this need, might have following disadvantages, or limitations [5]:
1. The XSD 1.1 engine, has to build a XPath tree to evaluate an assertion, which is a memory overhead.
2. It looks like, that by using assertions, we cannot implement following facilities of "notNamespace" attribute: we cannot specify namespace URIs with keywords, ##targetNamespace & ##local.
3. Using the, "notNamespace" attribute on xs:any wildcard, gives us optimization benefits of xs:any implementation (like, this doesn't have to build a XPath tree, which an assertion approach requires). Moreover, it's better to use a native facility of a construct (like, xs:any), which keeps the XSD schema's design more natural, and easy to understand.

And also, some of the features of "notQName" attribute can be written with an assertion like following:

  <xs:assert test="not(local-name(*[last()]) eq 'XX'
                       and
                      namespace-uri(*[last()]) eq 'http://www.example.com/mySample')" />

This approach would have similar issues, as specified above [5].

I hope, that this post was useful.

XSD 1.1: few more assertions use cases; assertion inheritance

2010-01-01T13:02:00.005+05:30

I'm continuing through, with writing uses cases for XSD 1.1 assertions, and testing them with Xerces-J. Here's the next ones in this series:

The uses cases below, demonstrate XSD assertions usage in a XSD schema type hierarchy.

Example 1

XML document [1]:

  <Example x_count="3">
    <x a="val1">2</x>
    <x a="val2">4</x>
    <x a="val3">6</x>
  </Example>

XSD 1.1 document [2]:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
    
    <xs:element name="Example">
      <xs:complexType>
        <xs:sequence>
          <xs:element name="x" type="x_Type" maxOccurs="unbounded" />
        </xs:sequence>
        <xs:attribute name="x_count" type="xs:nonNegativeInteger" use="required" />
        <xs:assert test="@x_count eq count(./x)" />
        <xs:assert test="every $x in x[position() lt last()] satisfies
                    number($x/@a/substring-after(.,'val')) lt
      number($x/following-sibling::x[1]/@a/substring-after(.,'val'))" />
      </xs:complexType>
    </xs:element>
  
    <xs:complexType name="x_Type">
      <xs:simpleContent>
        <xs:extension base="myInteger">
          <xs:attribute name="a" type="attrType" use="required" />
        </xs:extension>
      </xs:simpleContent>
    </xs:complexType>
  
    <xs:simpleType name="myInteger">
      <xs:restriction base="xs:positiveInteger">
        <xs:assertion test="$value mod 2 = 0" />
      </xs:restriction>
    </xs:simpleType>
  
    <xs:simpleType name="attrType">
      <xs:restriction base="xs:string">
         <xs:pattern value="val[1-9][0-9]*" />
         <xs:maxLength value="20" />
      </xs:restriction>
    </xs:simpleType>
  
  </xs:schema>

The purpose of the XML document [1], and the corresponding XSD schema [2] would be quite self-explanatory I believe.

I'll try to explain below, what the assertions in above XSD schema [2], are intended to accomplish:
a) The assertion on complex type (an anonymous type) of element, "Example" is an usual assertion on a XSD complex type, as I've explained in few earlier posts in this series. For the interest of readers, this assertion is ensuring that, value of attribute "a" of element(s) "x" have a suffix integer value, to string "val" is specified in a numerically ascending order.
b) The schema type, of element "x" is "x_Type". x_Type is a complex type (because, it specifies an attribute), and has simple content. The simple content definition of, element "x" is defined by the simple type, "myInteger". The type, myInteger specifies an assertion facet (which tests, that the integer value content, of element "x" is even). This demonstrates, that a schema type (x_Type here) inherits assertions from it's base types (the assertions, all the way up in type hierarchy are inherited -- if any of the assertions, in some of ancestor XSD types are specified).

Example 2

XML document [3]:

  <Example>
    <x a="val1">2</x>
    <x a="val2">4</x>
    <x a="val3">6</x>
  </Example>

XSD 1.1 document [4]:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
    
    <xs:element name="Example">
      <xs:complexType>
        <xs:sequence>
          <xs:element name="x" type="x_Type" maxOccurs="unbounded" />
        </xs:sequence>
        <xs:assert test="every $x in x[position() lt last()] satisfies
                       number($x/@a/substring-after(.,'val')) lt
         number($x/following-sibling::x[1]/@a/substring-after(.,'val'))" />
      </xs:complexType>
    </xs:element>
  
    <xs:complexType name="x_Type">
      <xs:simpleContent>
        <xs:restriction base="x_base">
          <xs:assert test="$value lt 100" /> 
        </xs:restriction>
      </xs:simpleContent>
    </xs:complexType>
  
    <xs:complexType name="x_base">
      <xs:simpleContent>
        <xs:extension base="myInteger">
          <xs:attribute name="a" type="attrType" use="required" />
        </xs:extension>
      </xs:simpleContent>
    </xs:complexType>
  
    <xs:simpleType name="myInteger">
      <xs:restriction base="xs:positiveInteger">
        <xs:assertion test="$value mod 2 = 0" />
      </xs:restriction>
    </xs:simpleType>
  
    <xs:simpleType name="attrType">
      <xs:restriction base="xs:string">
        <xs:pattern value="val[1-9][0-9]*" />
        <xs:maxLength value="20" />
      </xs:restriction>
    </xs:simpleType>
  
  </xs:schema>

Here's how the assertions in XSD schema [4], work on XML document, [3]:
There's nothing too complicated about the assertion rules here. The assertions on the complex type, "x_Type" consists of assertions within this type, and the assertions inherited from the base type.

The XSD examples, [2] and [4] look quite similar. The difference between XSD examples, [2] and [4] is that, the type "x_Type" in example [2] inherits a XSD simple type, while the type "x_Type" in example, [4] inherits a XSD complex type. The element, "Example" in XML document, [3] doesn't have an attribute, "x_count" (this is a cosmetic difference, between the two examples).

Both of the above, XSD examples demonstrate assertions inheritance from base XSD types (one of the examples demonstrates inheriting assertions from a simple type, while the other example demonstrates inheriting assertions, from a complex type).

I hope, that this post was useful.

Happy New Year, 2010

2010-01-01T10:49:00.000+05:30

As the new year dawns, here's a new year wish, to readers of this blog:

XSD 1.1: few more assertions and CTA use cases

2009-12-13T17:06:00.012+05:30

In my quest to test Xerces-J's XSD 1.1 implementation, I've come up with another example, using XSD 1.1 assertions and CTA (type alternatives) which I'll like to share here.

Here's a fictitious use-case and some discussions and analysis of the XSD technical options, for solving this use-case, later on in this post.

XML document [1]:

  <shapes>
    <polygon kind="square">
      <a>10</a>  
      <b>10</b>
      <c>10</c>
      <d>10</d>
    </polygon>
    <polygon kind="rectangle">
      <a>10</a>  
      <b>8</b>
      <c>10</c>
      <d>8</d>
    </polygon>
    <polygon kind="triangle">
      <a>5</a>  
      <b>10</b>
      <c>15</c>
    </polygon>
  </shapes>

XML document [2]:

  <shapes xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <polygon kind="square" xsi:type="Quadrilateral">
      <a>10</a>  
      <b>10</b>
      <c>10</c>
      <d>10</d>
    </polygon>
    <polygon kind="rectangle" xsi:type="Quadrilateral">
      <a>10</a>  
      <b>8</b>
      <c>10</c>
      <d>8</d>
    </polygon>
    <polygon kind="triangle">
      <a>5</a>  
      <b>10</b>
      <c>15</c>
    </polygon>
  </shapes>

XSD 1.1 Schema [3]:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="shapes">
       <xs:complexType>
         <xs:sequence>
           <xs:element name="polygon" type="Triangular" maxOccurs="unbounded">
             <xs:alternative test="@kind = ('square', 'rectangle')" type="Quadrilateral" />
           </xs:element>
         </xs:sequence>   
       </xs:complexType>    
    </xs:element>

    <xs:complexType name="Triangular">
       <xs:sequence>
         <xs:element name="a" type="xs:positiveInteger" />
         <xs:element name="b" type="xs:positiveInteger" />
         <xs:element name="c" type="xs:positiveInteger" />
       </xs:sequence>
       <xs:attribute name="kind" type="xs:string" use="required" />    
    </xs:complexType>

    <xs:complexType name="Quadrilateral">
       <xs:complexContent>
         <xs:extension base="Triangular">
           <xs:sequence>
             <xs:element name="d" type="xs:positiveInteger" />
           </xs:sequence>
           <xs:assert test="if (@kind = 'square') then (a = b and b = c and c = d) else true()" />
           <xs:assert test="if (@kind = 'rectangle') then (a = c and b = d) else true()" />
         </xs:extension>
       </xs:complexContent>
    </xs:complexType>

  </xs:schema>

XSD 1.1 Schema [4]:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="shapes">
       <xs:complexType>
         <xs:sequence>
           <xs:element name="polygon" type="Polygon" maxOccurs="unbounded" />
         </xs:sequence>   
       </xs:complexType>    
    </xs:element>

    <xs:complexType name="Polygon">
       <xs:sequence>
         <xs:element name="a" type="xs:positiveInteger" />
         <xs:element name="b" type="xs:positiveInteger" />
         <xs:element name="c" type="xs:positiveInteger" />
         <xs:element name="d" type="xs:positiveInteger" minOccurs="0" />
       </xs:sequence>
       <xs:attribute name="kind" type="xs:string" use="required" />
       <xs:assert test="if (@kind = 'triangle') then not(d) else true()" />
       <xs:assert test="if (@kind = 'square') then (a = b and b = c and c = d) else true()" />
       <xs:assert test="if (@kind = 'rectangle') then (a = c and b = d) else true()" />    
    </xs:complexType>

  </xs:schema>

XSD 1.1 Schema [5]:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="shapes">
       <xs:complexType>
         <xs:sequence>
           <xs:element name="polygon" type="Polygon" maxOccurs="unbounded" />
         </xs:sequence>   
       </xs:complexType>
    </xs:element>

    <xs:complexType name="Polygon">
       <xs:sequence>
         <xs:element name="a" type="xs:positiveInteger" />
         <xs:element name="b" type="xs:positiveInteger" />
         <xs:element name="c" type="xs:positiveInteger" />
         <xs:element name="d" type="xs:positiveInteger" minOccurs="0" />
       </xs:sequence>
       <xs:attribute name="kind" use="required">
          <xs:simpleType>
            <xs:restriction base="xs:string">
              <xs:enumeration value="square" />
              <xs:enumeration value="rectangle" />
              <xs:enumeration value="triangle" />
            </xs:restriction>
          </xs:simpleType>
       </xs:attribute>
    </xs:complexType>

  </xs:schema>

XSD 1.1 Schema [6]:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="shapes">
      <xs:complexType>
         <xs:sequence>
           <xs:element name="polygon" type="Polygon" maxOccurs="unbounded" />
         </xs:sequence>   
       </xs:complexType>    
    </xs:element>

    <xs:complexType name="Polygon">
       <xs:sequence>
         <xs:element name="a" type="xs:positiveInteger" />
         <xs:element name="b" type="xs:positiveInteger" />
         <xs:element name="c" type="xs:positiveInteger" />
       </xs:sequence>
       <xs:attribute name="kind" use="required">
         <xs:simpleType>
           <xs:restriction base="xs:string">
             <xs:enumeration value="square" />
             <xs:enumeration value="rectangle" />
             <xs:enumeration value="triangle" />
           </xs:restriction>
         </xs:simpleType>
        </xs:attribute>    
    </xs:complexType>
 
    <xs:complexType name="Quadrilateral">
       <xs:complexContent>
         <xs:extension base="Polygon">
           <xs:sequence>
             <xs:element name="d" type="xs:positiveInteger" />
           </xs:sequence>
         </xs:extension>
       </xs:complexContent>
    </xs:complexType>

  </xs:schema>

The goal of this use-case is following:
To define a XSD content model, for the XML document [1].

Solution of the use-case, and analysis:
The XSD 1.1 way of solving this would be schema's, [3] or [4] (These are possible solutions, that come to my mind. There could be other solutions too).

The Schema [3] uses both CTA and assertions. Whereas, Schema [4] uses only assertions. To solve this particular use-case, I might likely prefer Schema [4], because the content model defined in this schema is simpler/smaller, which is achieved by defining less of Schema types (only the 'Polygon' type here), and achieving further validation objectives, by defining assertions within this type.

Though, Schema [3] is also an useful solution to this problem, which according to me depicts better XSD type modularity, and also offers better possibilies to reuse the types, defined here in other contexts/use-cases.

But my gut feeling, is to go for Schema [4], for this use-case :)

I am next trying to think, how to solve this use-case in XSD 1.0 way. Here are the things, that come to my mind (with some of of my analysis):
1. Write a XSD Schema, as number [5] above. This is close to the desired solution of the use-case, described in this post. But this schema, doesn't solve this problem completely, as it doesn't strictly enforce the properties of a traingle (has 3 sides), square (has 4 sides, and all sides are equal) or a rectangle (has 4 sides, and opposite sides are equal). This is where, XSD assertions are really needed, if we want to specify XML validation entirely in the XSD layer (I think specifying much of XML validation in XSD layer is good, from application design point of view, as constraints specified with assertions, are entirely declarative and can be easily specified/modified by people, responsible for maintaining business rules, and without requiring to write say procedural code for these kind of validations, in imperative/OO languages like Java).
2. Modify the XML instance, to something like [2]. i.e, make use of XSD 1.0 construct xsi:type (which needs to be specified in the XML instance document) in some way, and validate it with a Schema like, [6]. This solution again, doesn't (and I think with XSD 1.0, we cannot do so) enforce properties of different kind of polygons (as specified in point 1, above), and this also makes the XML document XSD language specific (because it contains the instruction, xsi:type from XSD namespace), making it inconvenient to use such an XML document in environments, where XSD is not available, or where XSD processing is not needed.

The solutions presented in this post, are some of the possible ways, in which the given problem description here might be solved. But I can imagine, that there could be few other possibilities too (from XSD, syntax point of view), to solve such a use-case.

That's all about, I wanted to write at the moment :)

I hope that this post was useful!

Xerces-J: XSD 1.1 assertions implementation updates

2009-12-10T22:25:00.013+05:30

There have been some improvements lately, to the XSD 1.1 assertions support in Xerces-J.

Here are the summary of recent assertion implementation changes, in Xerces-J:

1) XPath 2 expressions, in assertion facets should not access the XPath 2 context, because XPath context is "undefined" during assert facet evaluation

This implies that, the right way to invoke assertion facets, is as follows:

  <xs:simpleType>
    <xs:restriction base="xs:int">
      <xs:assertion test="$value mod 2 = 0" />
    </xs:restriction>
  </xs:simpleType>

(i.e, we need to use the XPath "dynamic context" variable, $value to access the XSD simple type value.)

If an attempt is made to access the XPath context in above XPath expression, like say as follows (using the expression, "." here):
<xs:assertion test=". mod 2 = 0" />

Xerces returns an error message like, following:
test.xml:4:21:cvc-assertion.4.3.15.3: Assertion evaluation ('. mod 2 = 0') for element 'x (attribute => a)' with type '#anonymous' did not succeed (undefined context).

Or an XPath expression, like following:
./@a mod 2 = 0

Would result in a similar error.

A special error message, was constructed (designating, "undefined context" to the user) in Xerces, for this use case.

2) Ability to evaluate assertions, on XML attributes

If attributes in XML document use user-defined XSD simple types, then assertions would also apply to attributes, as they do for XML elements.

Following is a little example for this, use case.

XML document:

  <Example>
    <x a="210">101</x>
  </Example>

Corresponding XSD 1.1 schema:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
    
    <xs:element name="Example">
       <xs:complexType>
         <xs:sequence>
           <xs:element name="x" type="X_Type" maxOccurs="unbounded" />
         </xs:sequence>
       </xs:complexType>
    </xs:element>
 
    <xs:complexType name="X_Type">
       <xs:simpleContent>
          <xs:extension base="xs:int">
             <xs:attribute name="a">
                <xs:simpleType>
                   <xs:restriction base="xs:int">
                     <xs:assertion test="$value mod 2 = 0" />
                   </xs:restriction>
                </xs:simpleType>
             </xs:attribute>
          </xs:extension>
       </xs:simpleContent>
     </xs:complexType>

  </xs:schema>

Please note, that how we specify a XSD user-defined simple type for attribute "a" above, and an assertion facet on the simple type (there could by 0-n assertion facets here, as we have been looking at earlier).

The assertion facet XPath expression, $value mod 2 = 0 would operate on the context variable, $value (which is the attribute's value) and such an assert facet doesn't have access to the XPath context (a "context undefined" error would be flagged by Xerces, if an attempt is made to access the XPath context).

I hope, that this post was useful.

XPath 2.0: PsychoPath XPath processor update

2009-12-05T18:36:00.008+05:30

I've just run all the PsychoPath XPath 2 processor (an Eclipse Web Tools, Source Editing sub-project) W3C test-suite tests, and here are the results for them:

Tests: 8143
Errors: 0
Failures: 0

So it seems, PsychoPath XPath engine passes, 100% of the W3C XPath 2.0 test suite, and some of it's own tests.

This should be a moment of cheer, and wow!

It also looks, like that the upcoming Xerces-J release, 2.10.0 (ref, http://wiki.apache.org/xerces/November2009) would be getting almost a compliant XPath 2.0, engine for XSD 1.1 assertions and CTA.

Ref: An earlier post about PsychoPath status: http://mukulgandhi.blogspot.com/2009/09/psychopath-xpath-20-processor-update.html.

Xerces-J: XSD 1.1 assertions on simple types

2009-11-28T10:56:00.022+05:30

I'm trying to put up a post here, with few examples for assertions on XSD simple types, and also for complex types with simple contents, and testing them with Xerces-J XSD 1.1 implementation. The previous couple of posts on this blog, described assertions on XSD complex types having complex content (i.e, elements having "element only" or mixed content, and/or attributes).

1) Here's an example, taken from Roger L. Costello's collections of XSD 1.1 examples, which he's published on his web site:

XML document [1]:

  <Example>
    <even-integer>100</even-integer>        
  </Example>

XSD 1.1 document [2]:

  <schema xmlns="http://www.w3.org/2001/XMLSchema"
          elementFormDefault="qualified">

    <element name="Example">
       <complexType>
          <sequence>
             <element name="even-integer">
                <simpleType>
                  <restriction base="integer">
                     <assertion test="$value mod 2 = 0" />
                  </restriction>
                </simpleType>
             </element>
          </sequence>
       </complexType>
    </element>

  </schema>

The above XSD 1.1 schema [2] constrains the XSD integer values, to only even ones (this works fine with Xerces!). XSD 1.1 defines a new facet named, assertion on XSD built in simple types, which the above example describes.

Please note that, "assertion" facet (applicable both to XSD simple types, and complex types with simple contents) is conceptually different than "assert" constraint on complex types (some of the explanation, about this is also given below as well).

The XSD 1.1 spec mentions, that the assertions XPath 2 "dynamic context" get's augmented with a variable, $value. The XSD type of variable, $value is that of the base simple type (in this example, the type of $value is xs:integer). The detailed rules, for using variable $value in XSD 1.1 schemas are described, here.

It looks to me, that the ability to have an assertion facet on simple types, significantly enhances the XSD author's capability to provide many new constraints on simple type values, which were not possible in XSD 1.0 (for e.g, an ability to constrain integer values to be even, was not possible in XSD 1.0).

For the above example, we could specify assertions to something like below, as well:
<assertion test="$value mod 2 = 0" />
<assertion test="$value lt 500" />
(i.e, a set of two assertion facet instances)

Or perhaps, specifying only one assertion facet instance as following, <assertion test="($value mod 2 = 0) and ($value lt 500)" /> if user wishes, which realizes the same objective.

This enforces that the simple type value should be even, and also should be less than 500. Also, there are no limits to the number of assertion facet instances that can be specified. To my opinion, an ability to specify unlimited number of assertion facets (and also the assert constraints on complex types), makes assertions a tremendously useful XSD validation constructs.

Notes: Interestingly, the following facet definition achieves the same results as met by the 2nd assertion facet instance, that's described above:
<maxExclusive value="500" />
(this was available in, XSD 1.0 as well)

2) Complex types with simple contents, using assertions:
XML document [3]:

  <root>
    <x label="a">2</x>
    <x label="b">4</x>
  </root>

Here, the element "x" should have an attribute "label" with type xs:string. But the content of element "x" is simple (of type, xs:int for this example).
Additional we also want, that the simple content value of "x", should be an even number.

The XSD document for these validation constraints, is as follows [4]:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
 
   <xs:element name="root">
     <xs:complexType>
       <xs:sequence>
         <xs:element name="x" maxOccurs="unbounded" type="X_Type" />
       </xs:sequence>
     </xs:complexType>
   </xs:element>
   
   <xs:complexType name="X_Type">
     <xs:simpleContent>
        <xs:extension base="xs:int">    
          <xs:attribute name="label" type="xs:string" />
          <xs:assert test="$value mod 2 = 0" />
        </xs:extension>
     </xs:simpleContent>
   </xs:complexType>
  
  </xs:schema>

The use of xs:assert instruction is stressed in this example.

It's interesting to see, that if we change value of one of "x" elements as follows:
<x label="a">21</x>
(I changed the first "x")

Xerces fails the validation of XML instance, and returns following error message to the user:
test.xml:2:22:cvc-assertion.3.13.4.1: Assertion evaluation ('$value mod 2 = 0') for element 'x' with type 'X_Type' did not succeed.

Here, the XML validation did not succeed, because the value 21 is not an even number.

3) The last example of this post is following:
This describes the scenario of Complex types with simple contents. But here, the simple content get's its value by "restriction of a complex type". The previous example described Complex types with simple contents, using derivation by extension.

The XML file remains same [3], while the new XSD document is following [5]:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
 
   <xs:element name="root">
     <xs:complexType>
       <xs:sequence>
         <xs:element name="x" maxOccurs="unbounded" type="X_Type" />
       </xs:sequence>
     </xs:complexType>
   </xs:element>
   
   <xs:complexType name="X_Type">
     <xs:simpleContent>
        <xs:restriction base="x_base">      
           <xs:assertion test="$value mod 2 = 0" />
           <xs:assert test="@label = ('a','b')" />
        </xs:restriction>
     </xs:simpleContent>
   </xs:complexType>
   
   <xs:complexType name="x_base">
     <xs:simpleContent>
        <xs:extension base="xs:int">    
          <xs:attribute name="label" type="xs:string" />
        </xs:extension>
     </xs:simpleContent>
   </xs:complexType>
  
 </xs:schema>

Please notice, how assertions are specified on the complex type, "X_Type" (shown with bold emphasis). Here, we have two assertion instructions (xs:assertion and xs:assert). In this example, xs:assertion is a facet for the atomic value, of the complex type (the value of complex type is simple in this case!). While xs:assert is the assertions instruction on the complex type (which has access to the element tree).

The complexType -> simpleContent -> restriction, type definition can specify assertions with following grammar:
... assertion*, ..., assert* (i.e, 0-n xs:assertion components can be followed by 0-n xs:assert components (this ordering is significant, otherwise the XSD 1.1 processor will flag an error).
There could be other constructs as well, before xs:assertion here (and some after it. But anything after xs:assertion*, needs to be before the trailing xs:assert's). This is described in the relevant XSD 1.1 grammar at, http://www.w3.org/TR/2009/CR-xmlschema11-1-20090430/#dcl.ctd.ctsc.

Notes: The XML Schema WG decided to have two different names for assertion instructions (xs:assertion and xs:assert), for this particular scenario, so that the XSD Schema authors could decide, whether they are writing assertions as a facet for simple values, or assertions for complex types (which have access to the element tree). If this naming distinction was not made in XSD 1.1 assertions, then specification of asserts in XSD documents, in this case would have caused ambiguity (i.e, the XSD 1.1 processor could not tell, which assertion is a facet, and which is an assertion for the complex type).

Acknowledgements:
I must mention that XSD 1.1 examples shared by Roger L. Costello, helped us fix quite a bit of bugs in Xerces assertions implementation. Our sincere thanks are due, to Roger.

References:
1. Reader's could also find this article useful, http://www.ibm.com/developerworks/library/x-xml11pt2/ about XSD 1.1 co-occurence constraints, which describes XSD 1.1 assertions facility in detail.

I hope that this post was useful.

XSD 1.1: another assertions example with Xerces-J !

2009-11-27T04:21:00.007+05:30

Here's another XSD 1.1 assertions example, which I came up with today :)

An XML document is something like below:

  <person_db>
    <person id="1">
      <fname>john</fname>
      <lname>backus</lname>
      <dob>1995-12-10</dob>
    </person>
    <person id="2">
      <fname>rick</fname>
      <lname>palmer</lname>
      <dob>2001-11-09</dob>
    </person>
    <person id="3">
      <fname>neil</fname>
      <lname>cooks</lname>
      <dob>1998-11-10</dob>
    </person>
  </person_db>

Other than constraining the XML document to a structure like above, the XSD schema should specify following additional validation constraints, as well:
1) Each person's dob field should specify a date, which must be later than or equal to the date, 1900-01-01.
2) Each "person" element, should be sorted numerically according to "id" attribute, in an ascending fashion.

I wanted to achieve these validation objectives, completely with XSD 1.1 assertions. Here's the XSD 1.1 document, which I find that works fine, with Xerces-J:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
 
     <xs:element name="person_db">
       <xs:complexType>
          <xs:sequence>
            <xs:element name="person" maxOccurs="unbounded" type="Person" />
          </xs:sequence>
          <xs:assert test="every $p in person[position() lt last()] satisfies
                            ($p/@id lt $p/following-sibling::person[1]/@id)" />
       </xs:complexType>
     </xs:element>
   
     <xs:complexType name="Person">
        <xs:sequence>
          <xs:element name="fname" type="xs:string" />
          <xs:element name="lname" type="xs:string" />
          <xs:element name="dob" type="xs:date" />
        </xs:sequence>
        <xs:attribute name="id" type="xs:int" use="required" />
        <xs:assert test="dob ge xs:date('1900-01-01')" />
     </xs:complexType>
  
   </xs:schema>

Notes: It also seems, that above XSD validation requirements could be met, with following changes as well:
1. Remove assertion from the complex type, "Person".
2. Have an additional assertion on the element, "person_db" which will now look something like following:
<xs:assert test="every $p in person[position() lt last()] satisfies
($p/@id lt $p/following-sibling::person[1]/@id)" />
<xs:assert test="every $p in person satisfies ($p/dob ge xs:date('1900-01-01'))" />

i.e, we'll now have two assertions on the element, "person_db" (which are actually specified on the element's schema type).

Though, I seem to like the first solution as it seems elegant to me, and more logically in place.

I am happy, that this particular example worked fine as I expected, with Xerces.

I hope that this post was useful.

XSD 1.1: some CTA samples with Xerces-J

2009-11-20T21:51:00.020+05:30

I've been trying to write few XSD 1.1 Conditional Type Assignment (CTA) samples, and trying them to run with the current Xerces-J schema development SVN code.

To start with, here's the first example (a very simple one indeed) that I find, which runs fine with Xerces-J:

XML document [1]:

  <root>
    <x>hello</x>
    <x kind="int">10</x>
  </root>

XSD 1.1 document [2]:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

     <xs:element name="root">
       <xs:complexType>
         <xs:sequence>
           <xs:element name="x" type="xs:anyType" maxOccurs="unbounded">
             <xs:alternative test="@kind='int'" type="xInt_Type" />
             <xs:alternative type="xString_Type" />
           </xs:element>
         </xs:sequence>
       </xs:complexType>
     </xs:element>

     <xs:complexType name="xInt_Type">
       <xs:simpleContent>
         <xs:extension base="xs:int">
           <xs:attribute name="kind" type="xs:string" />
         </xs:extension>
       </xs:simpleContent>
     </xs:complexType>

     <xs:complexType name="xString_Type">
       <xs:simpleContent>
         <xs:extension base="xs:string">
           <xs:attribute name="kind" type="xs:string" />
         </xs:extension>
       </xs:simpleContent>
     </xs:complexType>

  </xs:schema>

Please note the presence of XSD 1.1 instruction, xs:alternative (which is newly introduced in XSD 1.1, and makes this XSD Schema, a type alternative scenario), within the declaration for element, "x" in above schema [2]. If the value of "kind" attribute on element "x" is 'int', then a schema type "xInt_Type" will be assigned to element "x". If the attribute "kind" is not present on element, "x" or if it's present, and it's value if not 'int', the schema type xString_Type get's assigned to element, "x".

Xerces-J successfully validates the above XML document [1] with the given XSD 1.1 Schema [2].

If we introduce the following change to the XML document:
<x kind="int">not an int</x>

Xerces-J would display following error messages:
cvc-datatype-valid.1.2.1: 'not an int' is not a valid value for 'integer'.

The above error message is correct, because the value 'not an int' in the XML document is not of type, xs:int.

Notes:
The schema types specified on xs:alternative instructions, need to validly derive (also referred to as, "type substitutable" in XSD 1.1 spec) from the default type specified on the element (which is, xs:anyType in this example), or the type on xs:alternative could be xs:error (this is a new schema type defined in XSD 1.1 spec, and is particularly useful with XSD type alternatives. The schema type xs:error has an empty lexical and value space, and any XML element or attribute which has this type, will always be invalid).

So for example, if we write an element declaration like following (demonstrating type substitutability/derivation of XSD types, specified on xs:alternative instructions):

  <xs:element name="x" type="xs:string" maxOccurs="unbounded">
    <xs:alternative test="@kind='int'" type="xInt_Type" />
  ...

Xerces-J would return following error message:
e-props-correct.7: Type alternative 'xInt_Type' is not xs:error or is not validly derived from the type definition, 'string', of element 'x'.

Making use of type xs:error, in CTAs:
Let's assume, that XML document remains same as document [1], and declaration of element "x" is now written like following:

  <xs:element name="x" type="xs:anyType" maxOccurs="unbounded">
    <xs:alternative test="@kind='int'" type="xInt_Type" />
    <xs:alternative type="xs:error" />
  </xs:element>

Now Xerces returns an error message like following:
cvc-datatype-valid.1.2.1: 'hello' is not a valid value for 'error'.

For this particular example, this error would occur if attribute "kind" is not present, or if the attribute "kind" is present, and it's value is not 'int'.

Xerces-J CTA implementation, using PsychoPath XPath 2 engine:
The XSD 1.1 spec, defines a small XPath 2 language subset, to be used by XSD 1.1 CTA instructions. Xerces-J has a native implementation of this XPath 2 subset (implemented by Hiranya Jayathilaka, a fellow Xerces-J committer), which get's selected by Xerces as a default XPath 2 processor, if CTA XPath 2 expressions conform to this XPath 2 subset (this was designed into Xerces, to make efficient XPath 2 evaluations using the CTA XPath 2 subset, since evaluating every XPath 2 expression with PsychoPath engine could have been computationally expensive).

But if, the XSD CTA XPath 2 expressions cannot be compiled by the native Xerces-J CTA XPath 2 subset, Xerces will attempt to use the PsychoPath XPath engine to evaluate CTA XPath expressions, as a fall back option (and also to enable users to use the full XPath 2 language with Xerces CTA implementation, if they want to).

To test, that PsychoPath engine does work with Xerces CTA implementation, I modified the type alternative instruction for the XSD example [2] above, to following:
<xs:alternative test="@kind='int' and (tokenize('xxx xx', '\s+')[1] eq 'xxx')" type="xInt_Type" />
I added a dummy XPath "and" clause, which can only succeed with Xerces, if PsychoPath engine would evaluate this XPath expression. This additional "and" clause doesn't make any difference to the validity of the XML document [1], as in this example it would always evaluate to a boolean "true". If we try to introduce any error into the above XPath expression like say, to following:
tokenize('xxx xx', '\s+')[1] eq 'xx' (please note the change from eq 'xxx' to eq 'xx', which will cause this XPath expression to evaluate to a boolean "false"), Xerces would report a XML validity error, which is really expected of the Xerces CTA implementation.

I hope that this post was useful.

XSD 1.1: some XSD 1.1 samples running with Xerces-J

2009-11-18T16:45:00.025+05:30

I was thinking lately to functionally stress test, the upcoming Xerces-J XSD 1.1 preview release (using the SVN code we have now, and later using the public binaries which will be provided by the Xerces project). I'm just curious to know, if there are any non-compliant parts in Xerces-J XSD 1.1 implementation, that I can find, which could probably serve as inputs to improving Xerces-J XSD 1.1 code base. To start with, I'll try to write few XSD 1.1 schemas, using the XSD 1.1 assertions and "Conditional Type Assignment (CTA)/type alternative" instructions.

Assertions examples

Example 1
Sample XML [1]

  <x a="xyz">
    <foo>5</foo>
    <bar>10</bar>
  </x>

XSD 1.1 Schema [2]
(Use Case: "the value of the foo element must be less than or equal to the value of the bar element")

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
 
    <xs:element name="x">
      <xs:complexType>
         <xs:sequence>
           <xs:element name="foo" type="xs:int" />
           <xs:element name="bar" type="xs:int" />
         </xs:sequence>
         <xs:attribute name="a" type="xs:string" use="required" />
         <xs:assert test="foo le bar" />
      </xs:complexType>
    </xs:element>
  
  </xs:schema>

Using Xerces-J XSD 1.1 validator, the XML document [1] above validates fine with the given XSD document [2].

If the assertion is written as follows (which is a false assertions. this is just to check for false assertions, and the error messages):
<xs:assert test="(foo + 10) le bar" />

Then that would make the XML instance document ([1] above) invalid, and following error message is returned by Xerces:
test.xml:4:5:cvc-assertion.3.13.4.1: Assertion evaluation ('(foo + 10) le bar') for element 'x' with type '#anonymous' did not succeed.

Use Case: "if the value of the attribute "a" is xyz, then the bar and baz elements are required, but otherwise they are optional".

This would require following assertion definition:
<xs:assert test="if (@a eq 'xyz') then (foo and bar) else true()" />

This works fine with Xerces-J.

Acknowledgements: Thanks to Douglass A Glidden for contributing these use cases, on xml-dev list.

Example 2
Sample XML [3]

  <Example>
    <x>hi</x>
    <y>there</y>
    <ASomeNameSuffix/>
  </Example>

XSD 1.1 Schema [4]

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
    
    <xs:element name="Example" type="myType" />
 
    <xs:complexType name="myType">
      <xs:sequence>
        <xs:element name="x" type="xs:string" />
        <xs:element name="y" type="xs:string" />
        <xs:any processContents="lax" />
      </xs:sequence>
      <xs:assert test="starts-with(local-name(*[3]), 'A')" />
    </xs:complexType>

  </xs:schema>

In this particular example (Example 2), the immediate sibling element, of element "y" is defined via the XSD wild-card instruction, <xs:any/>. The assertion in XSD Schema [4] enforces, that name of the sibling element, that appears after element "y" must start with letter "A". I think, this could not have been accomplished (i.e, defining a constraint on an element name, in xs:any wild-card instruction) with XSD 1.0.

Example 3
Sample XML [5]

  <record>
    <wins>20</wins>
    <losses>15</losses>
    <ties>8</ties>
    <!--
      0 to n no's of well-formed elements, allowed here
      by XSD wild-card instruction, <xs:any />
    -->
  </record>

XSD 1.1 Schema [6]

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:complexType name="Record">
      <xs:sequence>
        <xs:element name="wins" type="xs:nonNegativeInteger"/>
        <xs:element name="losses" type="xs:nonNegativeInteger"/>
        <xs:element name="ties" type="xs:nonNegativeInteger" minOccurs="0"/>
        <xs:any minOccurs="0" maxOccurs="unbounded" namespace="##any" processContents="lax"/>   
      </xs:sequence>
      <xs:assert test="every $x in ties/following-sibling::* satisfies
                     not(empty(index-of(('x','y','z'), local-name($x))))" />
    </xs:complexType>

    <xs:element name="record" type="Record"/>

  </xs:schema>

The XSD schema, [6] validates the XML document [5]. The <xs:any ../> instruction in this schema ([6]) allows, 0-n number of well-formed XML elements after element, "ties". This facility was available in XSD 1.0 as well (for the interest of readers, XSD 1.1 has a weakened wild-card support, which makes the above XSD schema [6] valid -- in XSD 1.0 this schema was invalid, due to enforcement of UPA (unique particle attribution) constraint. An example of this is given in an article here, http://www.ibm.com/developerworks/xml/library/x-xml11pt3/index.html#N10122.).

The assertion in this schema ([6]) enforces that, any element after element, "ties" which is allowed by the xs:any wild-card, should have a name (i.e, a name without namespace prefix -- a XML local-name) among this list, ('x', 'y', 'z'). Something like this, was not possible with XSD 1.0, and to my opinion this is nice :)

PS: more examples to follow, in the next few posts :)

References:
XSD 1.1 Part 1: Structures
XSD 1.1 Part 2: Datatypes

I must acknowledge (a long enough acknowledgement. but I must do it anyway :)), that Xerces assertions is really powered by the PsychoPath XPath 2 engine, and the credit for bringing PsychoPath engine to almost 100% compliance to W3C XPath 2.0 test suite (as of now, PsychoPath is 99% + compliant to the W3C XPath 2.0 test suite) should largely go to Dave Carver and Jesper Steen Møller. I was fortunate enough to contribute somewhat to PsychoPath XPath implementation (the freedom given to me as a Eclipse Source Editing project committer -- thanks to Dave Carver for this, helped me to drive Xerces assertions development quickly). Needless to mention the original PsychoPath code contribution by Andrea Bittau and his team, to Eclipse Foundation. I must also mention the numerous reviews, and improvements suggested by Khaled Noaman and general design advice by Michael Glavassevich (both are Xerces committers) helped tremendously while developing Xerces assertions. I must also mention Ken Cai's contribution, who wrote the original Xerces-PsychoPath interface, and also an initial implementation of that interface.

Xerces-J XSD 1.1 update: bug fixes and enhancements

2009-11-14T20:04:00.015+05:30

The Xerces-J team did few enhancements to the XSD 1.1 implementation, which solves few important XSD namespace URI issues, which affected Xerces assertions and Conditional Type Alternatives (CTA) implementation. These changes went into the Xerces-J SVN repository today.

Here are the summary of these improvements:
1) There is now an ability with Xerces-J XSD 1.1 implementation, to pass on the XSD language namespace prefix (which is declared on the XSD <schema> element), along with the XSD language URI as a prefix-URI binding pair to PsychoPath XPath 2.0 engine. This enhancement allows, the XSD language prefix declared on the "XSD 1.1 Schema instance" 's <schema> element to be used in the assertions and CTA XPath 2.0 expressions, for example as following:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" ...>
    ...
    <xs:assert test="xs:string(test) eq 'xxx'" />
    ...
  </xs:schema>

OR say,

  <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" ...>
    ...
    <xsd:assert test="xsd:string(test) eq 'xxx'" />
    ...
  </xsd:schema>

The earlier code in Xerces SVN (before the today's commit), hardcoded the XML Schema prefix to string, "xs" while communicating to the PsychoPath XPath 2 engine interface. That didn't allow the XPath 2 expressions in assertions and CTA to evaluate correctly (the Xerces code before this fix, always returned false for assertions, due to the presence of this bug), which used any other XSD prefix, like say "xsd" (even if the prefix "xsd" was bound to the XSD namespace, on the XSD root element, <schema>).

This was a significant Xerces assertions and CTA bug, which got solved today, and the fix for this is now available on the Xerces-J XSD 1.1 development SVN repository.

2) Another enhancement which went into Xerces-J SVN repository today, is the ability to specify the XPath 2.0 F&O namespace declaration on the XSD document root element, <schema>.

This enhancement makes possible something like, the following XSD 1.1 Schema to become valid:

  <xs:schema xmlns:xs="" xmlns:fn="http://www.w3.org/2005/xpath-functions" ...>
    ...
     <xs:assert test="xs:string(test) eq fn:string('xxx')" />
    ...
  </xs:schema>

Here the XML Schema author can, qualify the XPath 2 function calls in assertions XPath expressions, with the XPath 2 F&O namespace prefix, like fn:string('xxx') above. The F&O namespace prefix must be bound to the F&O namespace URI, "http://www.w3.org/2005/xpath-functions" for such a XSD Schema to be valid.

Even the following XSD 1.1 Schema is also valid (this happened to work correctly, earlier also before this Xerces SVN commit):

  <xs:schema xmlns:xs="" ...>
    ...
     <xs:assert test="xs:string(test) eq string('xxx')" />
    ...
  </xs:schema>

Here the XML Schema author, can use XPath 2 functions in Xerces assertions without specifying any prefix, for example like string('xxx') in the above example. The XPath 2 function calls without specifying the XPath 2 F&O prefix, would work correctly for all the XPath 2.0 built in functions, in Xerces assertions XPath 2 expressions.

World community grid

2009-11-14T10:31:00.003+05:30

There seems to be a nice initiative, "world community grid". I think, IBM sponsors this community computing grid. I have been participating on this grid, since quite a few days now, and it really works! and I believe, it does make a difference to community good.

This grid is composed by, computers which could be normal public personal computers at home, or office or any kind of computers that all can connect to the web. When a grid client is connected to the web, enabled by user authentication, the client computer participates in numerous public computing projects. Joining the grid, helps us to donate our computer's processing power to computations needed by these public projects, normally those that require massive computing simulations in short time.

Joining the grid, doesn't disrupt the normal user activity on client computers, and the grid client intelligently utilizes memory (a very less amount of memory is needed by the grid client, while it works, which is normally as less as 5-10 MB) and the CPU, without disrupting anything for user's personal activities. It is also possible to configure the user's grid activity, about how to use one's CPU. Somebody may want to work in the default mode, or can give more CPU usage to the grid project tasks. The default mode works, well for me.

All these details, and much more are available on the "world community grid", web page.

XML spec and XSD

2009-11-13T21:58:00.000+05:30

A few days ago, I started of a pretty length discussion on xml-dev mailing list about the following topic,

"Should the W3C XML specification specify XML Schema (a.k.a XSD) also as a XML validation language, as it specifies DTD (Document Type Definition)."

The XML spec seems to convey, that an XML document is valid, *only* if it's valid according to a DTD. I had a contention to this point, and started of a debate on xml-dev list related to this question. I argued, that since there are now newer XML validation languages like XSD, RelaxNG, Schematron etc, the XML spec now can modify the XML validation definition to refer to other XML Schema languages as well, rather than saying, that XML document is valid *only* if DTD is associated with the XML document.

Unfortunately, may people who spoke on xml-dev, who have been working with XML for long, did not agree to this idea. But alas, I still feel I had/have a valid point about this :(

I am referring to this threaded discussion again here, for records of this blog. Please follow this link, if anybody wants to read this whole discussion.

XSLT 1.0: Regular expression string tokenization, and Xalan-J

2009-11-01T10:22:00.024+05:30

Some time ago, XSLT folks were debating on xsl-list (ref, http://www.biglist.com/lists/lists.mulberrytech.com/xsl-list/archives/200910/msg00365.html) about how to implement string tokenizer functionality in XSLT. XPath 2.0 (and therefore, XSLT 2.0) has a built in function for this need (ref, fn:tokenize). XPath 2.0 string tokenizer method, 'fn:tokenize' takes a string and a tokenizing regular expression pattern as arguments. This is something, which cannot be done natively in XSLT 1.0. To do this, with XSLT 1.0 we need to write a recursive tokenizing "named XSLT template". But a "named XSLT template" using XSLT 1.0, for string tokenization has limitation, that it cannot accept natively an arbitrary regular expression, as a tokenizing delimiter.

I got motivated enough, to write a Java extension mechanism for regular expression based, string tokenization facility for XSLT 1.0 stylesheets, using the Xalan-J XSLT 1.0 engine.

Here's Java code and a sample XSLT stylesheet for this particular, functionality:

String tokenizer Xalan-J Java extension:

package org.apache.xalan.xslt.ext;

import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import org.apache.xpath.NodeSet;
import org.w3c.dom.Document;

public class XalanUtil {
    public static NodeSet tokenize(String str, String regExp) throws ParserConfigurationException {
      String[] tokens = str.split(regExp);
      NodeSet nodeSet = new NodeSet();
       
      DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
      DocumentBuilder docBuilder = dbf.newDocumentBuilder();
      Document document = docBuilder.newDocument();
       
      for (int nodeCount = 0; nodeCount < tokens.length; nodeCount++) {
        nodeSet.addElement(document.createTextNode(tokens[nodeCount]));   
      }
       
      return nodeSet;
    }
}

Sample XSLT stylesheet, using the above Java extension (named, test.xsl):

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                version="1.0"                                                    
                xmlns:java="http://xml.apache.org/xalan/java"
                exclude-result-prefixes="java">
                 
   <xsl:output method="xml" indent="yes" />
   
   <xsl:param name="str" />
   
   <xsl:template match="/">
     <words>
       <xsl:for-each select="java:org.apache.xalan.xslt.ext.XalanUtil.tokenize($str, '\s+')">
         <word>
           <xsl:value-of select="." />
         </word>
       </xsl:for-each>
     </words>
   </xsl:template>
   
 </xsl:stylesheet>

Now for e.g, when the above stylesheet is run with Xalan as follows:java -classpath <path to the extension java class> org.apache.xalan.xslt.Process -in test.xsl -xsl test.xsl -PARAM str "hello world", following output is produced:

<?xml version="1.0" encoding="UTF-8"?>
<words>
 <word>hello</word>
 <word>world</word>
</words>

This illustrates, that regular expression based string tokenization was applied as designed above, for XSLT 1.0 environment.

The above Java extension, should be running fine with a min JRE level of, 1.4 as it relies on the JDK method, java.lang.String.split(String regex) which is available since JDK 1.4.

PS: For easy reading and verboseness, the package name in the above Java extension class may be omitted, which will cause the corresponding XSLT instruction to be written like following:
xsl:for-each select="java:XalanUtil.tokenize(... I would personally prefer this coding style, for production Java XSLT extensions. Though, this should not matter and to my opinion, decision to handle this can be left to individual XSLT developers.

I hope, that this was useful.

Mozilla firefox and XPath namespace axis

2009-10-25T05:37:00.005+05:30

C. M. Sperberg-McQueen shared with us, that Mozilla Firefox browser doesn't implement the XPath namespace axis (ref, http://cmsmcq.com/mib/?p=757). CMSMcQ has encouraged us to cast a vote on Mozilla forum, to push Firefox team, to implement XPath namespace axis. I agree with CMSMcQ, and also find that XPath namespace axis is quite a critical functionality for XPath data model. This is certainly true for XPath 1.0, where namespace axis is very critical (and Mozilla, implements XPath 1.0). In XPath 2.0, namespace axis is deprecated but namespace nodes still is a core part of, XPath 2.0 data model as well.

I have already casted my vote for this with my support at, https://bugzilla.mozilla.org/show_bug.cgi?id=94270.

Other's might follow, please.

Martin Fowler: UML Distilled, 3rd Edition

2009-10-24T09:57:00.001+05:30

I have been reading Martin Fowler's book, "UML Distilled, 3rd Edition" since last few months (my book reading has been very slow, keeping in mind the time I spend on web these days, to do most of my learnings).

This is a great UML book (and has only, 175 pages but very good), and I recommend it to anybody wanting to know about UML (Unified Modeling Language).

How XPath compare values in prediates

2009-10-24T07:20:00.001+05:30

A user asked question similar to following, on IBM developerWorks XQuery and XPath forum:

What does A = B and A != B mean in XPath expressions?

Michael Kay provided a very nice explanation to this:
The operators "=" and "!=" in XPath use "implicit existential quantification". So A=B is shorthand for "some $a in A, $b in B satisfies $a eq $b" (the longhand form is legal in XPath 2.0), while A!=B is shorthand for "some $a in A, $b in B satisfies $a ne $b".

So, not(A=B) is true if there is no pair of items from A and B that are equal, while (A!=B) is true if there is a pair of values that are not equal. In practice, you nearly always want not(A=B).

OO multiple inheritance, and Java

2009-09-27T21:33:00.004+05:30

I have been thinking again, about multiple inheritance and why Java doesn't support it. I wrote a bit about this topic, some time ago.

There are so, so many resources on web about this, and it's actually very easy to find the answer to this, via a simple web search. Here is an article from where I started to know an answer to this, http://www.javaworld.com/javaqa/2002-07/02-qa-0719-multinheritance.html, which pointed me to this white paper by James Gosling and Henry McGilton. Really, I did not read this white paper by Java creators earlier (it never came across my eyes :)), in spite being familiar and working with Java since long time. Sometimes, we find gems on web in an unexpected ways (I mean, this paper is a gem for me :)). I'll try to read this paper (hopefully fully, and being able to understand it) over the next few days.

And here is a link in this white paper, which explains why Java doesn't support multiple inheritance. The following white paper link is also interesting, which gives a complete overview of C and C++ features, that were omitted in Java language (Java has been influenced from C and C++).

My personal opinion is, that if we must need to use multiple inheritance, we should just try to write programs in C++. On the contrary, my experience in using Java for about a decade, convinces me, that Java is suitable to solve almost any business application problem, and absence of multiple inheritance in Java, is not an hindrance to design good programming abstractions for problem domain. The advantages like Java's byte code portability and web friendliness far outweigh, any disadvantages caused by absence of multiple inheritance. On numerous occasions, I have created Java byte code on Windows, and used it without modification on Unix based systems (and vice versa). This is something which is built into the Java language, and it is cool!

Xerces-J: XML Schema 1.1 Conditional Type Alternatives (CTA), enhancements

2009-09-20T19:08:00.012+05:30

Apache Xerces-J team has made enhancements, to the way XML Schema 1.1, Conditional Type Alternatives (CTA) are evaluated.

Following are summary of the recent XML Schema 1.1 CTA enhancements, being done in Xerces-J:

1) The XML Schema 1.1 spec, allows the implementations to use a smaller XPath 2.0 subset for CTA, and can also provide the full XPath 2.0 language, for CTA evaluations.

Xerces-J had the smaller XPath 2.0 susbset already implemented, for CTA since quite some time (which was contributed by Hiranya Jayathilaka), and that was the only XPath 2.0 support which, Xerces-J CTA implementation had earlier. Xerces-J team recently added full XPath 2.0 support as well, for XML Schema 1.1 CTA, using the PsychoPath XPath 2.0 engine (which is used, by Xerces-J assertions facility as well).

If the user, writes XPath 2.0 expressions adhering to the XPath 2.0 subset for CTA, then the native XPath 2.0 implementation in Xerces-J shall process those XPath expressions. But if the XPath expression parsing fails by the native Xerces-J XPath 2.0 processor, Xerces-J falls back to the PsychoPath processor for XPath evaluation, allowing users to use the full XPath 2.0 language, for CTA.

2) Some time ago, Xerces-J implemented the XSD 1.1 data type, xs:error as well, which is useful in XML Schema 1.1 CTA.

It's been quite a pleasure, working on some of these patches.

Xerces-J: XML Schema 1.1 assertions enhancements

2009-09-20T13:26:00.013+05:30

The XML Schema 1.1 language, defines an assertions facility (xs:assert and xs:assertion), which constrain the XML Schema simple and complex types.

Apache Xerces-J implements XML Schema 1.1 assertions. As described in the XSD 1.1 spec, assertions typically have following XML representation:

  <assert
    id = ID
    test = an XPath expression
    xpathDefaultNamespace = (anyURI | (##defaultNamespace | ##targetNamespace | ##local)) 
{any attributes with non-schema namespace . . .}>
    Content: (annotation?)
 </assert>

For XML Schema simple type facets, the assertions are named, xs:assertion (as opposed to the xs:assert instruction for complex types) and rest of assertion contents are same.

Some background about XML Schema 1.1 assertions, and it's implementation in Xerces-J, could be referred at following blog post, which I wrote some time ago.

During the last week, we enhanced Xerces-J assertions to support the assertions attribute, 'xpathDefaultNamespace'. I did contribute this patch to Xerces-J, and it's now available on Xerces-J SVN repository.

The following XML Schema 1.1, specification description, describes how an assertions attribute 'xpathDefaultNamespace' works (please see the section, "XML Mapping Summary for XPath Expression Property Record {default namespace}").

Here's a simple example, about how 'xpathDefaultNamespace' functions in XML Schema 1.1 assertions:

XML document [1]:

  <X xmlns="http://xyz">
    <message>hello</message>
  </X>

XSD 1.1 document [2]:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
             targetNamespace="http://xyz"
             elementFormDefault="qualified">

     <xs:element name="X">
       <xs:complexType>
          <xs:sequence>
             <xs:element name="message" type="xs:string" />
          </xs:sequence>
          <xs:assert test="message = 'hello'" xpathDefaultNamespace="##targetNamespace" />
       </xs:complexType>
     </xs:element>

  </xs:schema>

In the XML document [1] above, since element "message" belongs to the namespace, "http://xyz" (by virtue of the default namespace declaration, xmlns="http://xyz" on element "X"), therefore the XPath (2.0) expression message = 'hello', on xs:assert instruction would return a boolean value "true", only if an element reference "message" in the XPath expression belongs to the namespace, "http://xyz". This namespace information needs to be provided to the XPath engine, via the 'xpathDefaultNamespace' attribute, on xs:assert instruction. If for the above XML instance document [1], 'xpathDefaultNamespace' attribute is not provided on the xs:assert instruction, then the XPath expression, message = 'hello' would return false (as then, the element "message" would be considered in no/null namespace, by the XPath engine), and the element instance at runtime, would become invalid according to such a XSD 1.1 Schema.

Allowing the 'xpathDefaultNamespace' attribute to be working on XML Schema 1.1 assertions, further increases the usefulness of XML Schema 'assertions' instruction, because now the XPath expressions, on assertions can be XML namespace aware.

The implementation of 'xpathDefaultNamespace' attribute on assertions, required enhancing PsychoPath XPath 2.0 engine as well. The updated PsychoPath library, JAR has been copied to Xerces-J SVN repository as well.

Xerces-J: XML Schema 1.1 inheritable attributes and CTA

2009-09-12T20:16:00.008+05:30

I just had my XML Schema 1.1 inheritable attributes patch, and it's integration with XML Schema 1.1 Conditional Type Assignment (CTA) facility, committed to Xerces-J XML Schema 1.1 SVN dev stream. I had earlier written about this, on this blog.

Credits for Xerces-J inheritable attributes work, should also go to Khaled Noaman (a fellow Xerces-J committer), who reviewed my patch and suggested many improvements. The Xerces-J inheritable attributes work, adds to an earlier work for XML Schema 1.1 Conditional Type Assignment (CTA), by Hiranya Jayathilaka (also a fellow Xerces-J committer).

PsychoPath XPath 2.0 processor update

2009-09-08T09:22:00.011+05:30

Some time ago, I wrote a progress update, about PsychoPath XPath 2.0 processor's (an Eclipse, Web Tools Source Editing subproject) compliance with, W3C XPath 2.0 test suite.

As of today, following are the test suite success rate, for PsychoPath engine:

Tests: 8137
Failures: 163
Errors: 5

This reflects a success rate of: 97.9%.

I think, these recent PsychoPath improvements with W3C XPath 2.0 test suite, reflect a much increased quality of PsychoPath product.

I am hoping, we would reach 100% test suite compliance, with PsychoPath in a near future!

Credits, for most of the recent PsychoPath improvements, should go to Dave Carver and Jesper Steen Møller.

Update on 2009-09-18: Took an update today from the Eclipse CVS servers, for PsychoPath source code, and following are the W3C XPath 2.0 test results:
Tests: 8137
Failures: 132
Errors: 5

Update on 2009-10-17: As of today, following are the W3C XPath 2.0 test results:
Tests: 8137
Failures: 81
Errors: 1

This reflects the W3C XPath 2.0 test suite pass percentage, for PsychoPath engine of about 98.9%. It seems, we are moving closer to the 100% test success rate, for PsychoPath. Dave Carver has been working a lot, on these improvements during last few days.

Roger L. Costello: About XML Schema 1.1

2009-09-05T22:53:00.011+05:30

Roger L. Costello a brilliant thinker, about XML and related technologies, has recently posted extensive study materials related to XML Schema 1.1, on his web site (ref, http://www.xfront.com/xml-schema-1-1/).

Roger mentions, Saxon 9.2 as one of the products that implements XML Schema 1.1 features.

From the information I have from Apache Xerces-J developer's forum, I am happy to share with the community, that Apache Xerces-J is planning to release, a preview of XML Schema 1.1 implementation, by around the end of this year (i.e, Dec, 2009). A latest snapshot of Xerces-J XML Schema 1.1 source code repository and related dependencies, is located at, https://svn.apache.org/repos/asf/xerces/java/branches/xml-schema-1.1-dev/. User's are welcome to build binaries from the sources, available here and try XML Schema 1.1 features, currently implemented in Apache Xerces-J. Much of the XML Schema 1.1 features, are implemented at this Xerces-J snapshot URL.

References:
1) XML Schema Definition Language (XSD) 1.1 Part 1: Structures
2) XML Schema Definition Language (XSD) 1.1 Part 2: Datatypes

XML document validation, while parsing with Java DOM API

2009-08-22T22:14:00.011+05:30

I spent few hours, discovering this while working with the DOM XML parsing API, and using it with Xerces-J, in a Java program.

I wanted to parse an XML document in Java using a plain DOM parser, along with doing validation, using either W3C XML Schema or a DTD.

Following is a sequence of instructions which needs to be written for this:

DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
dbf.setNamespaceAware(true);
dbf.setValidating(true);
SchemaFactory schemaFactory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
Schema schema = schemaFactory.newSchema ...
dbf.setSchema(schema);
DocumentBuilder docBuilder = dbf.newDocumentBuilder();
docBuilder.parse(..

These statements, are all that are necessary to accomplish this task. But there, are few catches here, which I wish to share.

1) If dbf.setValidating(true) is specified, then a DTD is mandatory. Even if W3C XML Schema is provided with dbf.setSchema .., parsing would fail, since dbf.setValidating(true) was specified, and if a DTD is absent.

2) If we only want to do validation with W3C XML Schema, then we shouldn't specify dbf.setValidating(true), which is required only for DTD validation.

I spent a few hours discovering this, and thought that somebody might benefit from this post.

XML Schema 1.1: inheritable attributes, and it's implementation in Apache Xerces-J

2009-08-08T21:56:00.022+05:30

The XML Schema 1.1, language has defined a new facility to define attributes as inheritable.

The XML Schema, attribute definition(s) can now specify an additional property (in 1.1 version of the XML Schema language), inheritable (having a schema type, xs:boolean), which will indicate that all the descendant elements to the element (which specifies an inheritable attribute), can access the inheritable attribute by it's name.

It could first appear to the reader of the XML Schema 1.1 spec, that inheritable attributes are something, which can physically be present (i.e., a copy of it) on descendant elements. But this is not the correct interpretation of the inheritable attributes concept. I'll try to illustrate this point with few examples in this post.

Please consider the following XML Schema 1.1, fragment:

  <xs:element name="X">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="Y" type="xs:string" />
      </xs:sequence>
      <xs:attribute name="attr" type="xs:int" inheritable="true" />
    </xs:complexType>
  </xs:element>

This corresponds to, an XML structure like following:

  <X attr="1">
    <Y>hello</Y>
  </X>

The above XML Schema 1.1 fragment, indicates that attribute, "attr" is inheritable. The word inheritable seems to convey, that the following XML fragment could be valid as well, for the above XML Schema 1.1 fragment:

  <X attr="1">
    <Y attr="1">hello</Y>
  </X>

But this interpration of inheritable attributes is not correct. The inheritable attributes, cannot be physically copied to the descendant elements. In the above examples, the Schema type of element, "Y" is a simple type (i.e., xs:string). So how, could Y have an attribute, "attr" (since by definition, elements with simple types cannot have attributes)? Only XML Schema "complex types", can specify attributes. XML Schema 1.1, inheritable attributes do not change the nature of XML Schema simple types, and simple contents. The presence of attributes on any XML element, is governed only by the attribute declarations on the complex type definition of the element. This meaning for attributes with respect to XSD complex types is preserved, in XML Schema 1.1 as well.

Then it's interesting to think, that what could be the use of specifying the attribute as inheritable (when it cannot be physically present in the descendant elements)?

Inheritable attributes, are useful in a XML Schema 1.1 facility, like Conditional Type Assignment (CTA) / type alternatives.

Please consider the following XML Schema 1.1 example, defining an XML element and it's Schema type, using CTA and inheritable attributes:

  <xs:element name="X">
     <xs:complexType>
       <xs:sequence>
         <xs:element name="Y" type="xs:anyType">
           <xs:alternative test="@attr = 'INT'" type="xs:int" />
           <xs:alternative type="xs:error" />
         </xs:element>
       </xs:sequence>
       <xs:attribute name="attr" type="xs:int" inheritable="true" />
     </xs:complexType>
   </xs:element>

As per the above Schema (using type alternatives), the following XML instance is valid:

  <X attr="INT">
    <Y>100</Y>
  </X>

But the following XML instance would be invalid:

  <X attr="INT">
    <Y>hello</Y>
  </X>

The inheritable attribute is also particularly useful, to define the attribute xml:lang as inheritable in XML elements.

I got to know these facts, after raising a query last week, to W3C XML Schema comments forum.

I am thankful to following gentlemen, for answering my queries, on the W3C XML Schema forum:

C. M. Sperberg-McQueen
Noah Mendelsohn
Michael Kay

The fact, which I really wanted to share on this blog post (other than, sharing what the XML Schema, inheritable attributes are used for), was that I've written an implementation of inheritable attributes, for Apache Xerces-J's XML Schema 1.1 validator. I've submitted a patch for this, to Apache Xerces-J JIRA issue tracking system.

This patch currently, has a full implementation of attribute syntax changes (i.e, the presence of inheritable attribute itself, and it's binding with the XML Schema type, xs:boolean).

I'm in a process to, enhance the Xerces-J implementation of Conditional Type Assignment (CTA) facility, to be able to use inheritable attributes. I hope to complete the CTA changes in Xerces-J, for inheritable attributes in near future.

After all necessary reviews are done for this patch, by Xerces-J committers, I hope to have the inheritable attributes implementation, go to Xerces-J SVN repository, which will in most likelihood subsequently become part of an official future release, of Xerces-J.

2009-08-14: Today, I submitted all the Conditional Type Assignment (CTA) related changes, for inheritable attributes, to Apache Xerces-J JIRA issue tracking system. I would say, the XML Schema 1.1 inheritable attributes, and it's integration with CTA is completed, for Xerces-J. I'm feeling good about it :)

Grady Booch: about Linux, and TTY interfaces

2009-07-30T14:06:00.003+05:30

Reading through Grady Booch's latest blog post, I found that Grady has shared interesting information about TTY interfaces in Linux, and of UNIX based systems.

I read the article, which Grady pointed, almost completely and found it a great read.

Something interesting to share, I thought!

Niklaus Wirth: On current state of software development, and future

2009-07-18T20:39:00.003+05:30

Navigating from Dr. Niklaus Wirth's wikipedia web page, I could find a very interesting interview conversation, Dr. Wirth had, on the following web site (ref, http://www.eptacom.net/pubblicazioni/pub_eng/wirth.html).

This interview is dated, in 1997. I found Dr. Wirth's views in this interview, quite good to read.

Something interesting to share, I thought!

Niklaus Wirth: On recursive algorithms

2009-07-12T14:05:00.019+05:30

I have started reading the computer science, classic collection "ALGORITHMS + DATA STRUCTURES = PROGRAMS" by Niklaus Wirth. Dr. Wirth wrote this text in 1975. It's a great book.

Though "recursive algorithms" are widely known to computer science community since long time, I still could find some good advice in Dr. Wirth's book on usage of recursive algorithms.

Dr. Wirth mentions:
"An object is said to be recursive if it partially consists or is defined in terms of itself. Recursion is a particularly powerful means in mathematical definitions. The power of recursion evidently lies in the possibility of defining an infinite set of objects by a finite statement. In the same manner, an infinite number of computations can be described by a finite recursive program, even if this program contains no explicit repetitions."

We all know, what recursive algorithms are. It's a widely known programming technique. But I found particularly the advice, "When not to use recursion" in Dr. Wirth's book very worth while to apply.

Dr. Wirth further mentions:
"Recursive algorithms are particularly appropriate when the underlying problem or the data to be treated are defined in recursive terms. This does not mean, however, that such recursive definitions guarantee that a recursive algorithm is the best way to solve the problem.

Programs in which the use of algorithmic recursion is to be avoided can be characterized by a schema which exhibits the pattern of their composition. Such schema's can described as following:

[1]
P => if B then (S; P)

or, equivalently

P => (S; if B then P)
"

Dr. Wirth illustrates this principle with a well known, recursive definition of the factorial computation (mentioned below):

F0 = 1
F(i+1) = (i + 1) * f(i)

Dr. Wirth maps the factorial problem with the recursive anti-pattern he defines ([1] above):

[2]
P => if I < n then (I := I + 1; F := I * F; P)
I := 0; F := 1; P

In the above definition [2], S (ref, [1]) refers to,
I := I + 1; F := I * F

Dr. Wirth in the book, illustrates a following, iterative definition of factorial computation:


I := 0;
F := 1;
while I < n do
  begin I := I + 1; F := I * F
  end

Dr. Wirth says, "The lesson to draw is to avoid the use of recursion when there is an obvious solution by iteration.
This, however, should not lead to shying away from recursion at any price. The fact that implementations of recursive procedures on essentially non-recursive machines exists proves that for practical purposes every recursive program can be transformed into a purely iterative one. This, however, involves the explicit handling of a recursion stack, and these operations will often obscure the essence of a program to such an extent that it becomes most difficult to comprehend. The lesson is that algorithms which by their nature are recursive rather than iterative should be formulated as recursive procedures."

Just thought of sharing a bit of text, from this nice book and encouraging readers to read the book!

PsychoPath XPath 2.0 processor update

2009-07-04T12:16:00.013+05:30

Dave Carver and I have been trying to improve the Eclipse XPath 2.0 processor (a.k.a PsychoPath) during last couple of weeks. My motivation to keep working on PsychoPath engine has been a desire, to help Eclipse and Apache (Apache Xerces-J uses PsychoPath engine for XML Schema 1.1 processing) communities to be able to have a highly compliant XPath 2.0 engine.

Dave has written today, a progress update of PsychoPath development on this blog. I feel, we now have a pretty good XPath 2.0 implementation with PsychoPath. We are continuing to work on remaining non-compliant items, with PsychoPath. The remaining non compliance cases, to my opinion are near edge cases which users don't use too often. But we'll continue to solve them, with each future day and weeks.

Update on 2009-07-11: During last couple of days, Dave Carver has made quite a few useful improvements to PsychoPath, and the W3C XPath 2.0 test suite within Eclipse. I took an update today of the latest PsychoPath sources, and the XPath 2.0 test suite, and following are the latest test results:

Total tests: 8137
Failures: 811
Errors: 48

This reflects, the test pass success rate of about 89.5%. I think, this is quite good. Lot of credit of these improvements should go to Dave Carver. Dave has single handedly, created a JUnit version of the full W3C XPath 2.0 test suite, which is in itself a great feat! Having JUnit tests, helps us tremendously to run the XPath 2.0 tests, from within Eclipse.

Update on 2009-08-09: Following are the current PsychoPath test data:
Total tests: 8137
Failures: 386
Errors: 24
This reflects a test suite pass percentage of about, 95% which looks very impressive. The test suite code coverage, is about 75-80%.
Lot of credit for the latest PsychoPath improvements should go to, "Jesper S Møller" who has recently volunteered to help improve PsychoPath with the XPath 2.0 test suite. Dave Carver is also putting in his time, on PsychoPath improvements.

Multiple inheritance in Java

2009-06-26T06:33:00.004+05:30

I have always missed true multiple inheritance in Java (like, in C++). For e.g., we are not able to define a class as follows in Java:

class X extends A, B, C {

}

Though, I do not see any inheritance use case which cannot be solved by the current Java facilities, but I would love to have this facility in Java. I think, the most latest Java version (1.7) doesn't have this feature.

One workaround I can see, for multiple inheritance, is to define a class like following:

class X {
A a;
B b;
C c;
}

i.e., we could create private class members inside X (whose functionality we want to use in class X).

Though this might serve purpose for some of the cases, but it's not true multiple inheritance! This I think, is actually aggregation pattern.

Of course, Java has multiple inheritance of interfaces. But that is inheritance of method signatures, and not of implementation.

I guess, keeping the number of base classes to one, Java is much simpler syntactically, and has a simpler compiler implementation. Though I agree, that having a simple syntax (as the current Java inheritance facilities) which is powerful enough, and can solve many use cases is better, than having a complex syntactical facility, which might serve even more uses cases, but could also lead to semantically difficult programs, which may be difficult to maintain and debug, as complexity of the problem domain increases.

Primitive long a subtype of float

2009-06-21T09:56:00.005+05:30

The Java language specification defines, that primitive "long" is a subtype of primitive "float" (ref, http://java.sun.com/docs/books/jls/third_edition/html/typesValues.html#4.10.1).

But XML Schema Datatypes spec, shows no relationship between xs:float and xs:long (
ref: XML Schema 1.0 data types, XML Schema 1.1 data types).

I'm a little confused, that which concept is correct (Java's definition of this data-type inheritance, or XML Schema). I seem to be in favor of XML Schema definition. But perhaps, XML Schema type system is for XML oriented data, and Java type system is for a wider class of applications. But I'm not sure, if this is the reason for the differences of definitions in Java language spec and XML Schema.

Became Eclipse WTP committer

2009-06-20T14:47:00.006+05:30

I was nominated as Eclipse WTP project committer, for the WTP Source Editing subproject.

As per the voting process, for becoming an Eclipse project committer, the Eclipse WTP Source Editing team, granted me project committership on 18, Jun 2009.

My contributions to PsychoPath XPath 2.0 engine (which is one of the components in WTP Source Editing tooling), helped me become a committer to this project.

I am happy to be included in the Eclipse WTP team. I look forward to contribute more to PsychoPath, and other WTP components. Apart from PsychoPath, I look forward to work on WTP XSL components in near future.

Running first XSLT 2.0 stylesheet with IBM XSLT 2.0 engine

2009-06-15T18:17:00.007+05:30

I could run my first XSLT 2.0 stylesheet with IBM XSLT 2.0 engine (ref, WAS XML Feature Pack Open Beta).

I tried the following XSLT 2.0 stylesheet, using xsl:for-each-group instruction, which worked well with the IBM XSLT engine.


<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                version="2.0">

  <xsl:output method="xml" indent="yes" />

  <xsl:template match="books">
     <books>
       <xsl:for-each-group select="book" group-by="author">
   <author name="{current-grouping-key()}">
      <xsl:for-each select="current-group()">
        <book>
          <xsl:copy-of select="name" />
   <xsl:copy-of select="publisher" />
        </book>
      </xsl:for-each>
   </author>
       </xsl:for-each-group>
     </books>
  </xsl:template>

</xsl:stylesheet>

XML Schema validation with Xerces-J

2009-06-13T20:57:00.006+05:30

Hiranya Jayathilaka raised an interesting discussion some time ago on xerces-dev list, that how we could validate an XML Schema 1.0 document using Xerces-J. Hiranya was looking for a solution using a Java API with Xerces-J. We were looking for verifying the correctness of the Schema document, and not doing an XML instance document validation.

I'm providing a summary of the discussion we had on the list, and the conclusions we made.

There are basically three ways of doing this:

1. Using a JAXP SchemaFactory
Using this technique, we do something like below:

SchemaFactory sf = SchemaFactory.newInstance ..
sf.setErrorHandler ..
Schema s = sf.newSchema(new StreamSource(schemapath));

The 'SchemaFactory.newSchema' call would not succeed if XML Schema has a grammar error.

2. Using XSLoader
Using this technique, we do something like below:

XSLoaderImpl xsLoader = new XSLoaderImpl();
XSModel xsModel = xsLoader.loadURI(xsdUri);

Michael Glavassevich suggested, how we could add an error handler to this mechanism:

DOMErrorHandler myErrorHandler = ...;

XSImplementation xsImpl = (XSImplementation) registry.getDOMImplementation("XS-Loader");
XSLoader xsLoader = xsImpl.createXSLoader(null);

DOMConfiguration config = xsLoader.getConfig();
config.setParameter("error-handler", myErrorHandler); // <-- set the error handler

3. Using XMLGrammarPreparser
Using this technique, we do something like below (thanks to Hiranya Jayathilaka for sharing this code):

XMLGrammarPreparser preparser = new XMLGrammarPreparser();
preparser.registerPreparser(XMLGrammarDescription.XML_SCHEMA, null);
preparser.setFeature("http://xml.org/sax/features/namespaces", true);
preparser.setFeature("http://xml.org/sax/features/validation", true);
preparser.setFeature("http://apache.org/xml/features/validation/schema", true);
preparser.setErrorHandler(new MyErrorHandler());
Grammar g = preparser.preparseGrammar(XMLGrammarDescription.XML_SCHEMA, new XMLInputSource(null, xsdUrl, null));

Michael Glavassevich provided a nice comparison of these three approaches:
SchemaFactory - it is an entry point into the JAXP Validation API for loading schemas for validation. If it was a user asking I'd recommend SchemaFactory of the three choices since it's in Java 5+ and would work in environments where Xerces isn't available.
XSLoader- it is an entry point into the XML Schema API for obtaining an XSModel for analysis/processing of the component model.
XMLGrammarPreparser - it provides API for preparsing schemas and DTDs for use in grammar caching (i.e. a lower-level alternative to SchemaFactory).

Function parameters vs global variables

2009-05-30T11:22:00.001+05:30

I bumped upon this problem, and thought of sharing my experiences here.

There are occasions, where I have to write an XSLT function and simply use it. For e.g. (this is just an illustration. we could have more function parameters, and a different return type),


<xsl:function name="fn:somefunction" as="xs:boolean">
  <xsl:param name="pName" as="xs:string" />

  <!-- use $pName and the variable, $someList (defined below) -->
  <xsl:sequence select="something.." />
</xsl:function>

The evaluation of this function also depends on some data/information other than the parameters being passed. This external information on which the function depends, could be a global variable. Say for e.g.,


<xsl:variable name="someList" as="element()+">
  <x>a</x>
  <x>b</x>
  ..
  ..
</xsl:variable>

In my case, this is a fairly static data (the variable, $someList), and is needed for the evaluation of above function.

I could see two option, on how the function may use an external data ($someList in this case):
1) Have an external data as a global variable (as illustrated above)
2) Supply this data as additional function parameter

I was in a sort of dilemma recently, where I had to decide whether I should go for option 1) or 2).

In my case, I opted for option 1) i.e., the global variable.

I can think of few pros and cons of both of the above options:
1. Having a global variable: This is good, if the external information is fairly static and perhaps has big chunk of data. Having global variable could be also useful, if the data is shared between multiple functions.
2. Having a parameter for the data: This option looks good from the point of view of the principle of composability. Functional programming advocates like this idea. I think, in classical computer science theory, a function (a callable module) is an abstraction which takes some input and produces some output. I think, the notion of functions accessing data which exists outside it's body is a mechanism devised by specific programming languages, and not as such defined by computer science theory. So from the point of view of this idea, having parameter for data is a good option. In fact I would also support this option, as far as possible.

In my case, I was working with XSLT. But I guess, these concepts would apply to many of other programming languages as well.

This topic could turn into a discussion, about how we must write good computer programs.

Any ideas are welcome please.

PsychoPath XPath 2.0 processor update

2009-05-26T02:31:00.001+05:30

We recently implemented quite a few built in XSD numeric data types in PsychoPath XPath 2.0 processor (ref, http://www.w3.org/TR/xmlschema-2/#built-in-datatypes). Now all (I mean, really all of xs:decimal ones :)) the data types in the xs:decimal hierarchy are available in PsychoPath, and these should be available in Eclipse WTP 3.2 M1 (which should be released sometime soon after the Eclipse Galileo release, at around June 09' end).

Now almost all the major built in Schema types are available in PsychoPath, except for few subtypes of xs:string (like xs:normalizedString, xs:token etc.). These shouldn't be much difficult to add.

Dave Carver reported, that the improvements we have done recently in PsychoPath have significantly improved it's compliance to the W3C XPath 2.0 test suite.

PsychoPath processor version is now enhanced from 1.0 to 1.1.

Xerces-J XSD 1.1 assertions and PsychoPath XPath 2.0 processor update

2009-05-17T12:03:00.006+05:30

I recently contributed few patches to the Eclipse PsychoPath XPath 2.0 engine, to support Schema aware XPath (2.0) expressions. These patches would enhance Schema aware support in PsychoPath XPath2 engine, for element and attribute nodes, for the XML Schema primitive types.

These enhancements in PsychoPath engine would make XPath expressions like following possible, to be evaluated by PsychoPath engine:

person/@dob eq xs:date('2006-12-10') // if dob is an attribute, and of schema type xs:date

person/dob eq xs:date('2006-12-10') // if dob is an element, and of schema type xs:date

@max ge @min // this would work if 'max' and 'min' have say schema types, xs:int

The patch in my local environment already exhibits these improvements. As promised by Dave Carver (the PsychoPath engine project lead), users would likely get these improvements in Eclipse WTP (Web Tools Project) 3.2.
2009-05-24: These changes are now committed to the Eclipse CVS server, and the improvements are flagged to be delivered in Eclipse WTP 3.2 M1, which should be quite sooner. Thanks to Dave Carver for testing all my patches, and committing them to the server.

PsychoPath engine already has a framework (thanks to Andrea Bittau and his team) for supporting Schema awareness (based on the Xerces-J XSD schema model). I just added in small pieces of code in attribute and element node implementations (particularly, improving the "typed value" of attribute and element nodes for built in XSD schema types), to enhance schema aware support.

I think, we are gradually moving to a more mature schema aware support in PsychoPath.

I'm also using these new PsychoPath processor capabilities, to implement schema aware XPath 2.0 evaluations in Xerces-J XSD assertions support.

I'm currently working on to construct a typed XPath data model instance, for XSD 1.1 assertions evaluations. Having this capability, would allow users to write XPath expressions like, following:

@max ge @min

or

person/@dob eq xs:date('2006-12-10')

In the absence of this (i.e, typed XDM nodes), currently users have to make explicit cast operations, like following:

xs:int(@max) ge xs:int(@min)

or

xs:date(person/@dob) eq xs:date('2006-12-10')

The XML Schema 1.1 assertions spec recommends a typed XDM instance.

We hope to provide this capability within Xerces-J, inline with the XML Schema 1.1 assertions specification.

2009-05-23: These improvements are now implemented, and I've submitted the code improvements to the Apache Xerces-J JIRA server. I'm hoping, we'll have these improvements committed on the Xerces-J SVN server some time soon.

Became Apache Xerces-J committer

2009-05-08T23:10:00.003+05:30

As per the voting process for becoming an Apache project committer, the Apache Xerces-J team granted me the committer status for the Xerces-J project on May 5, 2009.

This gives me an opportunity to contribute to the Xerces-J codebase, in a more direct way.

It's indeed a privilege for me to be part of the core Xerces team. Starting from being an Xerces user (since long time ago :)), to becoming project committer has been a rewarding journey in numerous ways.

Apache Xerces-J assertions implementation and PsychoPath XPath 2.0 processor

2009-05-03T14:45:00.001+05:30

I shared sometime back on this blog, on the work I am doing regarding XML Schema 1.1 assertions support in Xerces-J. The XML Schema 1.1 assertions processing requires a XPath 2.0 processor for performing Schema validation.

The Xerces-J team has opted to use the open source XPath 2.0 processor, PsychoPath. PsychoPath was developed by Andrea Bittau and his team. The PsychoPath team donated the PsychoPath code base to Eclipse community, where it is now formally used in the Eclipse, Web Tools Platform project. Future enhancements to PsychoPath are now taking place at Eclipse.

Since Xerces-J is using PsychoPath XPath 2.0 engine, we wish that PsychoPath be ideally 100% compliant to the XPath 2.0 specification, so Xerces-J users can use much of the failities of the XPath 2.0 language while using XML Schema 1.1 assertions.

After looking at the PsychoPath source code and using it quite a bit, my personal observation is, that PsychoPath has a pretty good XPath 2.0 implementation. Please refer to this documentation for knowing more about PsychoPath and the current compliance status.

The Eclipse WTP team is working actively to solve any remaining non-compliant items in PsychoPath. Incidentally, I have been working recently to help improve PsychoPath's compliance to the XPath 2.0 spec, and have contributed few patches to Eclipse.

We are also planning to run the W3C XPath 2.0 test suite on PsychoPath, and targetting PsychoPath to pass the test suite, with 100% coverage. This should give the PsychoPath adopters more confidence while using it.

WAS XML Feature Pack Open Beta

2009-04-23T12:07:00.001+05:30

There was an annoucement recently from IBM (http://webspherecommunity.blogspot.com/2009/04/was-open-xml-feature-pack-beta.html), about availability of "WAS XML Feature Pack Open Beta" supporting XPath 2.0, XSLT 2.0 and XQuery 1.0. It was good to know this.

Therefore, users would be able to use XPath 2.0, XSLT 2.0 and XQuery 1.0 in a WAS environment, using IBM's own processors for these languages.

This is an early preview release, with more enhancements expecting to come later.

I'm looking forward to try these language processors myself.

Xerces-J: XML Schema 1.1 assertions support

2009-04-21T22:14:00.006+05:30

This post is related to my earlier blog post, http://mukulgandhi.blogspot.com/2008/07/assertions.html about the XML Schema 1.1 assertions implementation into Xerces-J.

Today, I reached an important milestone with all the development finished for assertions in Xerces-J, and submitted an Apache JIRA issue for review.

Here is a small example of what XML Schema 1.1 assertions means:

<xs:complexType name="book">
    <xs:sequence>
      <xs:element name="name" type="xs:string" />
      <xs:element name="author" type="xs:string" />
      <xs:element name="price" type="xs:string" />
      <xs:element name="publisher" type="xs:string" />
      <xs:element name="pub-date" type="xs:date" /> 
    </xs:sequence>
    <xs:assert test="ends-with(price, 'USD')" />
    <xs:assert test="pub-date > xs:date('2007-12-31')" />
  </xs:complexType>

With this XML Schema 1.1 fragment, the user wants to have a validation constraint that, the price string should end with literal 'USD' and pub-date should be greater than the date 2007-12-31. This is a very simple example, but it does signify the usefulness of assertions syntax. We could have unlimited (0-n) numbers of xs:assert elements in a XSD schema type (which could be a simple type or a complex type. Though the assertions facet name in simple types is named xs:assertion). The value of 'test' attribute in assertions is an XPath 2.0 expression. All the assertions have to evaluate to boolean, "true" for an element to be locally valid.

There could be many other scenarios (and some of them quite complex, like for e.g., assertions present in a Schema type hierarchy) for writing assertions in XML Schema 1.1. It's difficult to specify all of them here. I'd ask the reader, to read the article [2] below, for learning about many of other, XML Schema 1.1 assertions scenarios.

With assertions in XML Schema 1.1 language, we could express much more involved XML validation constraints, that were almost impossible to specify in XML Schema 1.0. Using assertions, we can specify relationships between elements (like element names, contents etc), between elements and attributes, between attributes, and perhaps much more.

The assertions processing in XML Schema 1.1 works as follows:
When a XML Schema (1.1) processor encounters an element in the XML instance document, it must validate the element (if the user has requested validation) with it's associated type in the Schema (which could be a simple type or a complex type). The element's type declaration could be anonymous, or it could be a named type (which has a "name" attribute, and they are globally defined in the schema) declaration in the Schema. The XML Schema processor builds a XPath data model (XDM) tree rooted at this element (with Xerces, a XDM tree is built only if any assertions (which could be, 1-n in numbers) are associated with an element's type. If schema types of XML attributes have assertion facets, then these assertion facets work upon the attribute's value, and no XDM tree is constructed in this case). The XDM tree consists of the root element, it's attributes and all it's descendants. When an element validation is going on within Xerces, assertions evaluation also takes place as part of the validation process. Each assertion is evaluated on the XDM tree rooted at a given context element. Therefore, also any attempt by the assert XPath expression to access any node outside this element tree will not succeed.

We also have a wiki page for Xerces assertions implementation, http://wiki.apache.org/xerces/XML_Schema_1.1_Assertions. It describes a bit of implementation details of assertions in Xerces.

I'm happy to share that we expect Xerces-J to support the whole of assertions implementation in a near future release. And of course, Xerces would support lot of other XML Schema 1.1 features as well.

Following are few nice articles related to XML Schema 1.1, which are worth reading:
1. Overview of XML Schema 1.1 language
2. XML Schema 1.1 co-occurence constraints using XPath 2.0

Data modeling approaches

2009-03-20T22:15:00.000+05:30

With reference to my earliest blog post on "Relational/XML hybrid database design",
http://mukulgandhi.blogspot.com/2009/01/relationalxml-hybrid-database-design.html,

I have some more thoughts on this topic.

I think, an information model of an enterprise should consider XML as a first class data storage mechanism, along side the relational mechanism.

When XML was not there, using relational storage (and some other forms of EIS data) was the only choice. Given the benefits of XML (some of which are mentioned in my previous blog post cited above), it should be considered equally important as the relational mechanism, for data persistence needs.

There should be a mixture of XML and relational data in the overall data model of an enterprise. This would bring best of the worlds of relational and XML storage, to the enterprise's data.

I have seen that, today XML is used a lot for various kinds of property/configuration files, and for sending messages. But I feel, that wherever it makes good business sense, the core business entities should be modeled as XML along side the relational models. With the current hybrid XML/Relational databases (like, DB2 with pureXML), we can have the relational and XML data in the same transactional context.

Ptarithmetic: logic in computer science

2009-02-28T17:11:00.000+05:30

A post on comp.theory newsgroup referred to this paper, by Giorgi Japaridze. I read a bit of this paper, and believe the thoughts presented here are quite promising (at least, I found the goal of this study quite ambitious). I guess, people with interest in computer logic might find this interesting.

This paper introduces ptarithmetic (short for "polynomial time arithmetic") - a formal number theory similar to the well known Peano arithmetic, but based on the recent, computability logic instead of classical logic.

The arithmetic primitives defined in this theory, compute in polynomial time (as against classical logic, operations in which can compute in exponential or combinatorial time).

An ideal compiler architecture

2009-02-28T11:57:00.000+05:30

I found the compiler architecture description here, http://lambda.uta.edu/cse5317/notes/node5.html to be the most appealing from all the definitions I have read upto now.

Particularly I like the following idea:

"Suppose that you want to build compilers for n programming languages (eg, FORTRAN, C, C++, Java, BASIC, etc) and you want these compilers to run on m different architectures (eg, MIPS, SPARC, Intel, alpha, etc). If you do that naively, you need to write n*m compilers, one for each language-architecture combination.

The holly grail of portability in compilers is to do the same thing by writing n + m programs only. How? You use a universal Intermediate Representation (IR) and you make the compiler a two-phase compiler. An IR is typically a tree-like data structure that captures the basic features of most computer architectures. The first phase of this compilation scheme, called the front-end, maps the source code into IR, and the second phase, called the back-end, maps IR into machine code. That way, for each programming language you want to compile, you write one front-end only, and for each computer architecture, you write one back-end. So, totally you have n + m components."

Though the above idea of compiler construction is quite ideal, the author cautions with these statements:

"But the above ideal separation of compilation into two phases does not work very well for real programming languages and architectures. Ideally, you must encode all knowledge about the source programming language in the front end, you must handle all machine architecture features in the back end, and you must design your IRs in such a way that all language and machine features are captured properly."

I guess, this is something interesting to think about. How good it will be, if all major programming languages translate to the same intermediate representation.

Some thoughts about IPv6

2009-02-21T09:56:00.001+05:30

I read an article at IANA about the current state of IPv6 implementation on the world wide internet. I just thought of sharing few points here, as mentioned in the IANA IPv6 report, to help spread some awareness about this important issue. The article presents following facts:

1. IPv4 (the current traditional sceme of IP address allocation) theoretically has address space of about 4-5 billion IP addresses. When this scheme was designed, when the modern internet came into existence, it was imagined that about 4-5 billion devices will be a good maximum limit to the number of devices which can be connected to the internet.

But with the growing popularity of the internet across the world, the IPv4 scheme has proved to be insufficient to scale the internet using new devices. By Oct 2007, only 17% IP addresses were available for allocation under IPv4 scheme. Somewhere in 2010-11, all the IPv4 addresses will be exhausted. So after that time, it will be inevitable for the world to shift to IPv6 scheme to add more IP devices to the internet.

Apart from computer machines needing to connect to internet, many different kind of devices are now being designed to have IP capability (like many home appliances for example). To be able to assign all these new devices the globally unique IP addresses, adoption of IPv6 is bound to take place.

2. During the transition period from IPv4 to IPv6, both IPv4 and IPv6 hosts can continue to coexist and interoperate. But interoperation of these two schemes would require investment of time and money from hardware vendors, and network service providers.
The IPv4 scheme is expected to continue in operation for couple of years from now (I guess 5-10 years).

3. The IPv6 support in the softwares seems not to be a problem. Major OS vendors already have IPv6 support built into the OSs. The challenging aspect is the IPv6 support from the hardware and network equipment providers. I am sure that hardware and network vendors who intend to be be in the internet business for a long time in future, will commit the time and money to bring this vital expansion of the internet.

Interestingly, JDK 1.4 and above has provided support for IPv6 based IP addresses. So writing IPv6 enabled client applications is already possible with Java JDK.

Google is also serious about IPv6 adoption. The google search service is now available for IPv6 enabled devices. Please refer, http://googleblog.blogspot.com/2008/05/looking-towards-ipv6.html for more information about Google's efforts to promote IPv6 scheme. http://ipv6.google.com/ is the web URL to do google search from IPv6 machines. If you do a google search from an IPv4 machine (as I do, from the IP address 192.168.1.8) to this host, you'll get an "unknown host error". The machine should have an IPv6 based IP address to access this search service from Google.

DOM vs JAXB

2009-02-11T22:24:00.000+05:30

The DOM API and JAXB (Java Architecture for XML Binding) look a bit similar to me, for the task of mapping Java objects to XML documents (the concept popularly known as, XML data binding).

JAXB essentially does [1],

1. Marshalling of Java objects to XML.

2. Unmarshalling of the XML data to Java objects.

The DOM API also does [2],

1. Transformation of DOM tree to XML (known as the XML serialization process).

2. Transforming of XML documents to a DOM object tree.

At a broad level, the approaches [1] and [2] look similar.

But according to me, following are the differences between the two approaches:

1. The JAXB transformation is driven by a Schema. The JAXB binding compiler generates Java type definitions for the Schema components. With JAXB, the object model (generated by the JAXB framework) maps closely to the concepts in the problem domain (for e.g, there will be a Java class PurchaseOrder corresponding to a Schema type PurchaseOrder).

Whereas with DOM, the Schema does not drive the XML and Java mapping. And, all the elements in an XML document map to the DOM class, org.w3c.dom.Element. The name of the element, the data inside it and other properties of the element are accessible using the methods provided by the DOM API.

2. There are perhaps some performance differences between JAXB and DOM. A study about this has been done by Santiago Pericas-Geertsen, as shared in these blog posts.

http://weblogs.java.net/blog/spericas/archive/2005/12/dom_vs_jaxb_per.html
http://weblogs.java.net/blog/spericas/archive/2005/12/dom_vs_jaxb_per_1.html

Under certain circumstances (to my opinion), the XML data bindings needs can be met by the DOM API (where a Schema does not drive the data binding process, and a raw mapping of XML and Java is required).

Relational/XML hybrid database design

2009-01-26T17:21:00.000+05:30

Lately, I have been reading about the database design (particularly, for relational and XML hybrid storage).

I felt that a column design (for complex types, like say purchase order) having XML data (like the native SQL type, 'XML') is quite useful, as compared to distributing the whole schema (say, the purchase order - PO schema) across multiple relational tables and columns (with simple types).

Following are some of the advantages for using the SQL type, 'XML' in RDBMS tables:

1. The database and nearly the whole application becomes shielded from the Schema changes. If say something in the PO Schema changes, we only change the PO XML Schema. The RDBMS data model need not change (the columns of the table would still be same, say a, b, po where the column po is of type 'XML'). The SQL statements in the application programs (say, the JDBC statements) also need not change.

The impact of the Schema change on the application will be quite less, as compared to the situation, if the data were kept in simple type columns.

2. Designing a complex type (say, a purchase order) in a 'XML' column makes good design sense. The complex type most of the times has hierarchical structure, as is XML. So it is a good design decision to design complex types as XML. This also has advantage during validation of the data during data inserts and updates. The XML data as a whole get's written to the database, and an internal schema in the database validates the incoming XML data. This is much more simpler and natural than validating the data, had it been stored in multiple tables and columns.

3. The XML data in the relational/XML hybrid database can be accessed using XQuery, and also updated using XQuery update facility.

I found DB2 with pureXML having an excellent support for all these features.

Normalizing unnecessary whitespace text nodes during XSLT transformation

2009-01-16T21:53:00.000+05:30

Let's say that my input XML is following,


<test>
  <a/>
  <b/>
  <c/>
  <d/>
  <e/>
  <f>some data ..</f>
</test>

I need to write an XSLT transformation, which just removes elements, 'c' and 'd' and keeps rest of the structure same.

The result of the transformation should be following [1]:


<test>
  <a/>
  <b/>
  <e/>
  <f>some data ..</f>
</test>

The obvious solution to this problem is, to write a modified identity transformation logic.

i.e.,


<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                version="1.0">
   
   <xsl:output method="xml" indent="yes" />

   <xsl:template match="node() | @*">
     <xsl:copy>
       <xsl:apply-templates select="node() | @*" />
     </xsl:copy>
   </xsl:template>
   
   <xsl:template match="c | d" />
   
</xsl:stylesheet>

But there is a subtle flaw in this logic. The actual output produced by the above stylesheet is,


<?xml version="1.0" encoding="UTF-8"?>
<test>
  <a/>
  <b/>


  <e/>
  <f>some data ..</f>
</test>

There are a kind of two whitepace holes in the output (created by the elements which are removed). This makes the output not 100% same as the desired output [1].

The whitespace holes in the output above can be very well explained. They are actually the newline whitespaces (near the elements 'c' and 'd') present in the original document, which are preserved in the generated output.

Adding a little bit of extra logic in the stylesheet can fix this problem.

The right solution will be following,


<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                version="1.0">
   
   <xsl:output method="xml" indent="yes" />

   <xsl:template match="node() | @*">
     <xsl:copy>
       <xsl:apply-templates select="node() | @*" />
     </xsl:copy>
   </xsl:template>
   
   <xsl:template match="c | d" />
   
   <xsl:template match="text()[(normalize-space() = '') and (preceding-sibling::node()[1]/self::c or preceding-sibling::node()[1]/self::d)]" />
   
</xsl:stylesheet>

Please note the last template in this stylesheet, which fixed the whitespaces problem for me.

XSLT functions returning void

2009-01-10T21:48:00.000+05:30

I just thought, whether XSLT 2.0 functions can return something like a void value.

for e.g. as we could do,

public void myfunction() {

}

in Java.

I do not think there is any syntax in XSLT which allows this declaration.

i.e., can we do something like following in XSLT.


<xsl:function name="my:testfunction" as="a-void-type">
  <!-- something here -->
</xsl:function>

I think there is no way of specifying a void type in XSLT. A function must return something. The best we could do is, that we specify the return type as, as="xs:string?". i.e., the function may return a xs:string value, or it may return nothing (i.e., an empty sequence: ()).

But if we want to return something like void from a function, we can instead implement a named template for this. i.e., something like following,


<xsl:template name="testtemplate">
  <!-- something here -->
</xsl:template>

The named template is conceptually similar to xsl:function (both are callable modules), but there are some subtle differences as well, between them. xsl:function is a lot more loosely coupled module than the named template. The named template inherits the context from the caller, whereas the function has no access to the context information of the caller (though any piece of the context can be passed to the function as parameters).

XSLT: sorting data by duration

2009-01-04T23:11:00.002+05:30

I've this input text file (test.txt):

A started at 03:12:10
A ended at 03:20:20
B started at 03:20:25
B ended at 03:22:21
C started at 03:22:23
C ended at 03:22:55
D started at 03:22:57
D ended at 03:23:21
E started at 03:23:25
E ended at 03:24:40

Here A, B, C etc. are some events, and they start at a particular time and end at another time.

I need to produce an output like following:

D : 0-0-24
C : 0-0-32
E : 0-1-15
B : 0-1-56
A : 0-8-10

i.e., events sorted by the time they took (in ascending order of durations). The duration format in the output is, hr-min-sec.

The following XSLT 2.0 stylesheet worked well for this problem,

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:xs="http://www.w3.org/2001/XMLSchema"
                exclude-result-prefixes="xs"
                version="2.0">

<xsl:output method="text" />

<xsl:variable name="time-data" select="tokenize(unparsed-text('test.txt', 'UTF-8'), '\r?\n')" />

<xsl:template match="/">
  <xsl:variable name="temp-data">
    <xsl:for-each select="$time-data">
      <xsl:variable name="val" select="normalize-space(.)" />
      <xsl:variable name="pos" select="position()" />
      <xsl:if test="position() mod 2 = 1">
        <data key="{tokenize($val, '\s')[1]}">
          <xsl:value-of select="xs:time(tokenize($time-data[$pos + 1], '\s')[last()]) -xs:time(tokenize($val, '\s')[last()])" />
        </data>
      </xsl:if>
    </xsl:for-each>
  </xsl:variable>
  <xsl:for-each select="$temp-data/*">
    <xsl:sort select="xs:dayTimeDuration(.)" />
    <xsl:variable name="hr" select="hours-from-duration(.)" />
    <xsl:variable name="min" select="minutes-from-duration(.)" />
    <xsl:variable name="sec" select="seconds-from-duration(.)" />
    <xsl:value-of select="@key" /> : <xsl:value-of select="concat($hr, '-', $min, '-', $sec)" /> <xsl:text>
</xsl:text>
  </xsl:for-each>
</xsl:template>

</xsl:stylesheet>

The following XSLT sort instruction worked, well for this use case:
<xsl:sort select="xs:dayTimeDuration(.)" order="descending" />

I've used Saxon to solve this.

If anybody bumps by this post, and could think of a better solution (particularly written in a more functional style, and avoiding the temporary tree), I would be very glad to know that.

A static code quality tool, for XSLT code

2008-12-28T16:59:00.001+05:30

I wrote a little tool to measure and enhance the code quality, of XSLT programs/scripts. It's available at, http://gandhimukul.tripod.com/xslt/xslquality.html.

I hope that the XSLT community might find this useful.

Any comments/suggestions about this tool would be most welcome.

Michael Kay on application processing

2008-12-13T23:13:00.000+05:30

Roger L. Costello summarized an interesting discussion we had on xml-dev list, citing Michael Kay's thoughts (just thought of sharing with the readers here):

Question:

What language(s) do you recommend using for application processing?

Michael Kay:

... do everything using XML-based processing languages (XSLT and XQuery).

The idea I'm pushing is that you write the application end-to-end using XML-based languages. That means that you never convert the data into C++ or Java data types. If you only go half way, by calling XQuery from Java and then converting the results to Java values, you lose half the benefits.

Question:

By "write the application end-to-end using XML-based languages" do you mean XML end-to-end or PSVI end-to-end?

Michael Kay:

I'm not too fussed how the XML is represented, the important thing is to avoid converting it unnecessarily into non-XML models such as Java objects or SQL tables. Where "unnecessarily" means "except as required by the need to interface with other applications".

Question:

Isn't XSLT a special-purpose programming language?

Michael Kay:

It's true that XSLT is a specialized language rather than a general-purpose language. But it's capable of a lot more than some people imagine. Some people never really get below the surface to discover its depth.

Question:

What's wrong with using a mix of XML-based processing languages and imperative languages (Java, C++, etc)?

Michael Kay:

[If you use a mix of languages then you end up] spending 75% of your programming effort converting your data between one type system and another.

Writing your application in a single language, if you can do it, will always give you a substantial reduction in complexity versus using multiple languages.

And writing in a language that is well adapted to the data it is required to process will save you a lot of effort (human and machine effort) in doing data conversion.

Question:

Aren't there some applications for which an imperative language is needed? Aren't some problems so complex that they need an imperative language?

Michael Kay:

There are no applications that NEED imperative programming (functional programming has provably the same computational power). Whether things are more EASILY done that way is in large measure a matter of your skills and experience. (I remember working with programmers from an older generation who claimed coding was easier if they used GOTO statements.) There are one or two things I still find easier in imperative languages - notably some graph-walking applications - but they are few and far between.

Many problems that appear to be so complex that you need an imperative language turn out, on examination, to be complex only BECAUSE you are using an imperative language.

Question:

I agree that functional languages like XSLT are quite useful. But many of the underlying hardware architectures are von Neumann based and von Neumann machines work best with imperative programs (languages).

Michael Kay:

If we believed that we would all still be writing in Assembler, or at any rate using GOTO statements.

Question:

Isn't application processing faster using an imperative language? Aren't imperative languages easier for programmers to use?

Michael Kay:

... on both ease-of-use and performance I would go for XSLT or XQuery in preference to lower-level languages every time.

If you're receiving lexical XML from a web service, the time taken to process it in XSLT or XQuery is usually less than the time taken to parse and validate it. I would take a lot of convincing that a data binding approach is likely to be faster, given the cost of marshalling and unmarshalling the data.

And on ease of use, I've seen programmers struggling with regenerating all their Java classes when the schema changes and it's horrendous. (Worse, I've seen people refuse to change the schema because it has become too expensive to contemplate!) Having two different models of the same data, understanding how they relate, and organizing yourself to keep them in sync is simply complexity that you don't need.

Question:

XQuery can be usable when you need to access a small subset of an XML document. However, when one needs to access most of the data, or, worse, access the same data many times, data binding will have speed/memory advantage.

Michael Kay:

Evidence please! I don't see any reason why it should.

(There are still people who maintain that coding in assembler is faster than in C, or that coding in C is faster than in Java. All the evidence is that there are very few people with the skills in the lower-level language to beat the optimizers for the higher-level language. It can be done if you try hard enough, but not by the average programmer.)

Question:

For most web service applications the Java (or other programming language) representation is actually primary, and XML is just being used for interchange. So surely application processing should be done in an imperative language, right?

Michael Kay:

I don't think it matters which is primary, the complexity comes from having two representations and keeping them aligned. But I would have thought the format used for data interchange is primary in the sense that it needs to be agreed with other parties, whereas the Java representation is under local control. (Unless of course you're running the same code at both ends, in which case I'm not sure why you're using XML at all.)

Question:

Suppose I have a Web page that contains a form that users fill in. When the submit button is pressed the form data is sent to a server. Don't we need something like Java to receive the data? Doesn't this break the "all-XML" approach you advocate?

Michael Kay:

It doesn't really matter too much if you have a bit of Java glue to accept the input and fire off the [XSLT] transformation, so long as it doesn't try to manipulate the data. But why use Java for this? There are plenty of higher-level frameworks that will do the job - since you're using forms, Orbeon does the job nicely.

Question:

Why is there such variance in data binding tools with respect to mapping XML Schema structures into data structures in an imperative language?

Michael Kay:

XSD is used for other things [in addition to validation], notably data binding. To a large extent data binding is outside the scope of the XSD specification itself (XSD doesn't tell you how its types map to Java or C++) and this probably explains why there is wide variation between products in this area.

Question:

I would be interested to hear about any non-trivial and practical application other than "XML in, XML out", format changing kind, that are written exclusively in XSLT/XQuery.

Michael Kay:

How about an application for managing the creation, processing, and review of capital spending proposals in a large international corporation. The proposals are entered by form-filling using XForms, and each proposal is an XML document in an XML database. The rules defining the approval process (based on the nature of the capital project) are defined in a business rules document, also XML, (for example "anything over $1m requires CEO approval") and the abstract roles defined in that document (such as "CEO", or "finance controller, Taiwan") are mapped to real people in another XML document generated by a transformation of an XML dump of the LDAP directory. There's
also an XML document that defines the corporate reporting structure, or actually two structures one functional and one geographic. These rules are used to construct an approval schedule for the proposal, which is also stored in the XML database, and emails are sent to the relevant people at the relevant times, with clickable URLs that they can use to access the application and approve or deny requests, or ask for more information. There's a full query/reporting system built using the same technology: you define a query by form-filling using XForms, and this generates an XQuery to get the data from the database and an XSLT stylesheet to format the query results.

I believe the only parts of this application which are not written in XSLT, XQuery, XForms, or the Orbeon XML pipeline language (a precursor to XProc) are one or two simple extension functions to do things like sending an email or translating Base64 data from the LDAP directory.

Feasibility of "do all application coding in the XML languages"?

2008-12-06T09:35:00.000+05:30

Roger L. Costello initiated an interesting discussion (on, 01-Dec-08) on xml-dev mailing list, about following concept:

I am exploring the idea of "do all application coding in the XML languages."

Here is a response from a colleague:

"... in general XSLT is cool but limited. If your transform requires any "higher math" or advanced functionality or external code libraries (such as geometry coordinate system libraries), you almost always have to go back to a higher level language (such as Java) at some point."

Does my colleague make a true or false statement?

XML professionals on the list expressed interesting opinions about this. Following is the link to this threaded discussion, http://lists.xml.org/archives/xml-dev/200812/msg00007.html.

Just thought of sharing this with readers of this blog (if you got a chance to visit here).

Are multiple XPath predicates same as boolean "and" operator

2008-11-22T09:34:00.001+05:30

I had a doubt about this concept, and asked following question on xsl-list.

Supposing I write the following XPath expressions,

1) X[c1][c2] or specifying generically, X[c1][c2][]...[cn]

2) X[c1 and c2] or specifying generically, X[c1 and c2 ... and cn]

where c1, c2 etc. are boolean expressions.

are the two forms (1 & 2) above exactly equivalent (i.e., will they return the same nodeset/sequence)? I think yes ... but just wanted to confirm with the list.

if 1 & 2 are exactly equivalent, then what could be the rule of thumb for using which form in certain scenarios?

There was a good discussion on the list about this, and list members shared some useful thoughts.

Below is a summary of the points we discussed on the list.

1. David Carlisle

> are the two forms (1 & 2) above exactly equivalent

No

compare

X[position()=2][position()=2]

and

[position()=2 and position()=2]

the first one is

()

the second is

X[2]

David further wrote,

context position (position()) and size (last()) do change. so basically repeated filters are equivalent to and unless any of them depend on position() or last(), including the special case of [integer] being equivalent to [position()=integer]
this last case is what makes it tricky to do a static rewrite of this.

If you have

X[... foo ..][... bar ...]

you can only rewrite that to

X[(... foo ..) and (... bar ...)]

if you know that neither expression will evaluate to a number at run time.

2. Vasu Chakkera

If that were true, then the condition

myelement[@myattribute][1] should be same as
myelement[1][@myattribute], which is not true...

The predicate order is important

in a typical "and"

[a and b] = [b and a]

3. Andrew Welch

Only / will change the context node, so I would've thought one predicate after the other is pretty much equivalent apart from cases that rely on size of the selection (which is the only thing that changes after each predicate).

Mukul: I asked a related question in continuation to this.

for real world XSLT/XPath programs, upto how many predicates can we typically see?

I haven't seen programs using 3, 4 or more predicates.

X[..][..][..][..]

I have used only one or two predicates upto now.

are excessively large number or predicates really useful? (though, the syntax allows that)

I think perhaps, for complex 'and' conditions, using multiple predicates are useful.

David shared an interesting observation about this:

He has been using some stylesheets having upto 11 predicates.

He wrote:

> are excessively large number or predicates really useful? (though, the syntax allows that)

isn't that like asking if complicated expressions are useful? they are useful if you need them, otherwise they are not.

3 or 4 predicates is totally routine but the most common reason for having larger numbers is to filter attributes

[not(@purpose='iemode')]
[not(@purpose='artifact')]
[not(@purpose='w-dimension')]

is equivalent to

[not(@purpose='iemode') and
[not(@purpose='artifact') and
[not(@purpose='w-dimension')]

but I'd almost always use the first form in XSLT 1.0 because it's easier to indent and easier to refactor, but if starting from the beginning in XSLT 2.0 I'd write it as

[not(@purpose=('iemode','artifact','w-dimension'))]

Mukul: This was a nice discussion I believe, and I have learnt few useful concepts.

fn:contains -> multiple strings to compare with

2008-11-04T11:52:00.000+05:30

An XSLT user asked following question on xsl-list:

I want to have something that does this: contains('$d/ris:organ/text()', 'Hamburg' or 'Koblenz' or 'xxx'...) ===> Compare 1 String with multpile strings.

instead of: contains('$d/ris:organ/text()','Hamburg') or contains('$d/ris:organ/text()','Koblenz')...

Andrew Welch suggested following answer:

some $x in ('Hamburg', 'Koblenz', 'xxx') satisfies
contains($d/ris:organ/text(), $x)

This uses the XPath 2.0 quantified expression, "some".

This is cool.

I was prompted to share Andrew's answer here, because I thought of a lengthy and perhaps inefficient solution for this (I feel a bit stupid, actually :) ):

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:xs="http://www.w3.org/2001/XMLSchema"
                xmlns:my="http://my-functions"
                version="2.0">

 <xsl:output indent="yes" omit-xml-declaration="yes" />

 <xsl:template match="/">
  <xsl:variable name="str" select="'hello xxx dd'" />
  <xsl:variable name="list" select="('Hamburg','Koblenz','xxx')" />

  <xsl:if test="my:contains($str, $list)">
     matches
  </xsl:if>

 </xsl:template>

 <!-- a custom 'contains' implementation -->
 <xsl:function name="my:contains" as="xs:boolean">
  <xsl:param name="str" as="xs:string" />
  <xsl:param name="list" as="xs:string+" />

  <xsl:variable name="temp" as="xs:boolean*">
    <xsl:for-each select="$list">
      <xsl:if test="contains($str, .)">
        <xsl:sequence select="xs:boolean('true')" />
      </xsl:if>
    </xsl:for-each>
  </xsl:variable>

  <xsl:sequence select="if ($temp[1] = xs:boolean('true')) then
xs:boolean('true') else xs:boolean('false')" />

 </xsl:function>

</xsl:stylesheet>

boolean value of a RTF, in XSLT 1.0

2008-09-27T16:58:00.000+05:30

A recent discussion on xsl-list taught me something about a feature of XSLT 1.0 language.

Let's say there is a variable reference like following, in a XSLT 1.0 code:

<xsl:variable name="temp">
  <!-- anything could be here -->
</xsl:variable>

According to the XSLT 1.0 specification, the contents of the variable temp is called a RTF (Result Tree Fragment). Now what will be the result of the following function call, boolean($temp)

This will always return boolean value true.

Because, as defined in XSLT 1.0 specification,
The boolean value of a node set, with at least one node, is true. The RTF always has a root node.

It seems to me, that the function call, boolean($rtfVariable) is not very useful to applications, as it's always true.

This post applies to XSLT 1.0. I haven't cross checked what are the rules for this in XSLT 2.0.

Should we include
2008-09-12T19:44:00.000+05:30
We all know that the following declaration statement appears at the beginning of most of the XML documents.

[1]
<?xml version="1.0" encoding="UTF-8"?>

But the details of the above statement is not included in a DOM tree, and neither it is part of the XPath (2.0) data model. To my opinion, this statement is used by the XML parser to initialize certain behaviors (or to enable certain properties).

I just had a weird thought, that should we not include this information as part of XPath data model, perhaps as properties of the document node; and in case of DOM, part of the DOM tree? If this declaration is not present in the XML document, then the relevant properties can be empty sequences.

The XML declaration is available in the XML infoset [2], but it's not included in a DOM, or the XPath data model.

I think, perhaps the XML declaration information is not useful to end user applications, and is only useful to the XML parser.

[2] http://www.w3.org/TR/xml-infoset/

Michael Glavassevich on xml-dev list corrected me, about DOM:

Information from the XML declaration is already stored in the DOM [3][4][5] (since DOM Level 3). Within the API the values have an effect on serialization, in-memory well-formedness checking and in-memory validation.

[3] http://www.w3.org/TR/2004/REC-DOM-Level-3-Core-20040407/core.html#Document3-version
[4] http://www.w3.org/TR/2004/REC-DOM-Level-3-Core-20040407/core.html#Document3-encoding
[5] http://www.w3.org/TR/2004/REC-DOM-Level-3-Core-20040407/core.html#Document3-standalone

Schema aware processing with XSLT 1.0

2008-08-30T20:24:00.001+05:30

I wrote following article, about implementing some of the Schema aware stylesheet ideas (as defined in the XSLT 2.0 spec.) using a XSLT 1.0 processor and Java extensions, and a suitable validating XML parser. A requirement from a XSLT user on this blog motivated me to work on this idea.

The article on my web site showcases just how few of the XSLT 2.0 Schema aware facilities like, validating the output tree (or even a result tree fragment) prior to serialization can be done. This covers one of the very important aspects of Schema aware XSLT 2.0 stylesheet design.

I think it's not possible to implement following Schema aware XSLT 2.0 facilities, using XSLT 1.0 and extensions alone:

1) Pass the validated XML instance tree to the XSLT processor, and the stylesheet is able to access Schema annotated XPath 2.0 data model tree. The XSLT 2.0 and XPath 2.0 languages define various Schema related instructions and expressions (for e.g., element(*, typeName) etc.), which cannot be simulated with XSLT 1.0 and extensions.

2) Since we cannot access Schema annotated XPath 2.0 data model tree in a XSLT 1.0 stylesheet, we cannot access XML Schema type names in a stylesheet, which prohibits the enhanced static typing features in XSLT stylesheets.

Using a complete Schema aware XSLT 2.0 system allows very rich static typing in XSLT stylesheets out of the box.

Nice use case for xsl:analyze-string instruction

2008-08-22T18:43:00.000+05:30

I thought that this was interesting to share.

Recently an XSLT user discussed a problem on xsl-list, which was solved by Jeni Tennison using XSLT 1.0 long time ago.

I presented a XSLT 2.0 solution for the same problem. The 2.0 solution is lot shorter as compared to the 1.0 solution, and utilizes the XSLT 2.0 instruction, xsl:analyze-string.

The link to this thread is at, http://www.biglist.com/lists/lists.mulberrytech.com/xsl-list/archives/200808/msg00383.html.

This problem could be a nice use case for xsl:analyze-string instruction.

Transforming tree structure from one format into another

2008-08-14T22:30:00.000+05:30

An interesting question was asked on xsl-list,

The input XML file is something as:


<Objs>
   <obj name="a" child="b"/>
   <obj name="b" child="c"/>
   <obj name="b" child="d"/>
   <obj name="c" child="e"/>
</Objs>

Let's say that XML file has only one root node.

The output XML file would be:


<Obj name="a">
  <Obj name="b">
      <Obj name="c">
         <Obj name="e"/>
      </Obj>
      <Obj name="d"/>
  </Obj>
</Obj>

A tree structure is defined in input XML, by the 'name' and 'child' attributes. The output represents a true logical tree. We should be able to cater to unlimited number of tree nodes.

We need to write a XSLT stylesheet for this.

At first thought, I imagined that this could be a tough problem. But a little bit of patience helped me to write the stylesheet for this. The solution is presented below.


<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                version="1.0">

<xsl:output method="xml" indent="yes" />

<xsl:template match="Objs">
  <xsl:variable name="start" select="obj[not(@name = ../obj/@child)]" />
  <xsl:variable name="startName" select="$start[1]/@name" />
  <Obj name="{$startName}">
    <xsl:for-each select="obj[(@name = $startName) and not(../obj/@name = @child)]">
      <Obj name="{@child}" />
    </xsl:for-each>
    <xsl:call-template name="makeTree">
      <xsl:with-param name="list" select="obj[@name = $start/@child]" />
    </xsl:call-template>
  </Obj>
</xsl:template>

<xsl:template name="makeTree">
 <xsl:param name="list" />

 <Obj name="{$list[1]/@name}">
   <xsl:for-each select="$list">
     <xsl:variable name="child" select="@child" />
     <xsl:choose>
       <xsl:when test="not(../obj[@name = $child])">
         <Obj name="{$child}" />
       </xsl:when>
       <xsl:otherwise>
         <xsl:call-template name="makeTree">
           <xsl:with-param name="list" select="../obj[@name = $child]" />
         </xsl:call-template>
       </xsl:otherwise>
     </xsl:choose>
   </xsl:for-each>
 </Obj>
</xsl:template>

</xsl:stylesheet>

At first thought, I felt that XSLT 2.0 constructs will be required to solve this problem. But the problem can be solved completely with a XSLT 1.0 stylesheet.

My belief that XSLT is a wonderful language for processing XML data, became stronger after solving this problem.

Multiple values for XSLT keys

2008-08-03T20:15:00.000+05:30

An XSLT user, asked following question, on xsl-list.

How do I use multiple key values?

Declaration:

<xsl:key name="keyname" match="subroot" use="ccc"/>

During the usage, I want to specify multiple values:

<xsl:variable name="keyname" select="key('keyname', '11' or '22')"/> ==> Here I want to use multiple values 11 and 22.

xsl-list members suggested useful options,

1. David Carlisle

<xsl:variable name="keyname" select="key('keyname', '22')|key('keyname', '11')"/>

2. Michael Kay

In XSLT 2.0, you can supply a sequence:

key('keyname', ('111', '222'))

In 1.0, you can supply a node-set with one value per node - but of course it's hard to set that up, you need the xx:node-set() function.

I worked upon Mike's idea for a XSLT 1.0 solution,

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:exslt="http://exslt.org/common"
exclude-result-prefixes="exslt"
version="1.0">

<xsl:output method="xml" indent="yes" />

<xsl:key name="x" match="subroot" use="ccc"/>

<xsl:variable name="x-values">
<v>11</v>
<v>22</v>
</xsl:variable>

<xsl:template match="/root">
<result>
<xsl:for-each select="key('x', exslt:node-set($x-values)/v)">
<value>
<xsl:value-of select="eee" />
</value>
</xsl:for-each>
</result>
</xsl:template>

</xsl:stylesheet>

Mukul: I think, this could be better than David's suggestion, because if we want to have quite large number of different values to search by the key, we just have to change following code fragment,

<xsl:variable name="x-values">
<v>11</v>
<v>22</v>

</xsl:variable>

G. Ken Holman responded to my post, and provided a brilliant idea,

The node-set extension can be avoided to achieve what you want.

<xsl:for-each select="key('x', exslt:node-set($x-values)/v)">

The above can be replaced with standard XSLT 1.0 to read the stylesheet file as a source node tree.

<xsl:for-each
select="key('x',document('')/*/xsl:variable[@name='x-values']/v)">

Ken further wrote,

I grant, though, that if your stylesheet is large then putting this into a small included or imported fragment would keep any overhead of building the tree small.

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
version="1.0">

<xsl:output method="xml" indent="yes" />

<xsl:key name="x" match="subroot" use="ccc"/>

<xsl:include href="ken2values.xsl"/>

<xsl:template match="/root">
<result>
<xsl:for-each select="key('x',$x-values)">
<value>
<xsl:value-of select="eee" />
</value>
</xsl:for-each>
</result>
</xsl:template>

</xsl:stylesheet>

ken2values.xsl
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
version="1.0">

<xsl:variable name="x-values-data">
<v>11</v>
<v>22</v>
</xsl:variable>

<xsl:variable name="x-values"
select="document('')/*/xsl:variable[@name='x-values-data']/v"/>

</xsl:stylesheet>

I think, Ken's idea of having an included stylesheet (ken2values.xsl, above) is brilliant, as it is memory efficient, and we are able to avoid the node-set extension (as mentioned earlier).

XML Schema 1.1 assertions implementation in Xerces-J

2008-07-27T22:43:00.005+05:30

One of the interesting enhancements, that are happening recently to Xerces code base, is XML Schema 1.1 implementation.

The Xerces-J team motivated me to implement some of the XML Schema 1.1 features into Xerces. I've started with the implementation of XML Schema 1.1 facility, "assertions".

I've completed quite a bit of work regarding this, and am hoping that the assertions support I'm writing would be available in Xerces-J in near future.

2008-11-04: Xerces team approved my work so far for assertions implementation, and have committed my patch to the Xerces code base. In the coming weeks, I would be working on integrating XPath 2.0 processing for assertions.