Sunday, April 25, 2010

XSD 1.1: negative "pattern" facets and assertions

While exploring more of XSD 1.1 assertions, I've been pretty convinced that much of the limitations of XSD "pattern" facet can be overcome with assertions (and of-course one of real benefits of XSD 1.1 assertions is the ability to specify co-occurrence constraints, in XML Schema documents -- here's a nice article explaining XML Schema 1.1 co-occurrence constraints).

I think, one of the things which might get quite difficult to express in XML Schema 1.0, is specifying a negative word list.

For example, if we have this simple XML document:
  <fruit>apple</fruit>

And we want that, the XML element "fruit" must not contain say the words "cherry" or "guava". Although, this looks a pretty straight-forward regex use-case, but unfortunately it might get quite cumbersome to express this seemingly straightforward regex pattern, with the available XSD 1.0 regular-expression syntax.

My quick try to express this with XSD 1.0, was something like following:
<xs:pattern value="^(cherry|guava)" />

But unfortunately, the above pattern facet and quite a few similar regexes, can't accomplish this seemingly common use-case easily (I think, this is doable with XSD 1.0 regex's but certainly, it would be quite tedious to come to the right regex pattern -- of-course regex experts/gurus could do this easily, but not me at this moment!).

And now, I try to express these validation constraints with XSD 1.1 assertions. Here's a sample XSD 1.1 schema [1], using assertions to solve this, and few of similar use-cases:

  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

     <xs:element name="Example" type="Fruits1" />
  
     <xs:complexType name="Fruits1">
       <xs:sequence>
         <xs:element name="fruit" type="xs:string" />
         <xs:element name="exclude" type="xs:string" />
       </xs:sequence>
       <xs:assert test="not(fruit = tokenize(exclude,','))" />
     </xs:complexType>
  
     <xs:complexType name="Fruits2">
       <xs:sequence>
         <xs:element name="fruit" type="xs:string" />
         <xs:element name="exclude" type="xs:string" />
       </xs:sequence>
       <xs:assert test="not(fruit = (for $x in tokenize(exclude,',') return 
                                              normalize-space($x)))" />
     </xs:complexType>
  
     <xs:complexType name="Fruits3">
       <xs:sequence>
         <xs:element name="fruit" type="xs:string" />
         <xs:element name="exclude" type="xs:string" />
       </xs:sequence>
       <xs:assert test="not(fruit = (for $x in tokenize(exclude,',') return 
                                (string-join(tokenize($x,' '),''))))" />
     </xs:complexType>
  
   </xs:schema>

A sample XML instance document [2], that we'll validate with the above schema, is following:
  <Example>
    <fruit>apple</fruit>
    <exclude>cherry,guava</exclude>
  </Example>

As stated in the original requirements above, we want that the word in element "fruit" must not contain any of words, from the comma-separated list in the "exclude" element.

In the above XSD schema [1], the complex type "Fruits1" can successfully validate the above XML instance document [2].

The complex type "Fruits2" can validate an exclude list, where there could be white-spaces before and after the 'comma separator'. For example, the list "cherry, guava" (please note, an extra white-space after the 'comma') would be considered an appropriate exclusion list for this example. Whereas, this list variant cannot be validated by the schema type, "Fruits1".

And the complex type "Fruits3" can validate an exclude list of kind, "cherry, g u a v a" (i.e, there could be white-space characters, within a word) -- this is a figment of my imagination :). But certainly there could possibly be such lexical constraints in instance documents.

PS: All the examples in this post were tested with, Xerces-J.

I hope, that this post is useful.

Saturday, April 17, 2010

XSD 1.1: xs:precisionDecimal, assertions and Xerces-J updates

Section 1
Recently, I went through in sufficient detail about the XSD primitive data-type, xs:precisionDecimal (newly introduced in, XSD 1.1), and was trying to use XSD 1.1 assertions to simulate xs:precisionDecimal (just to vent my curiosity and exploring more of, XSD assertions) as a user-defined (as a restriction of xs:decimal data-type) XSD Simple Type (though I believe, a native implementation of xs:precisionDecimal should also exist in an XSD 1.1 implementation, or in language systems which may use the XSD type system -- for example, a stand-alone XPath (2.x) implementation which uses an XSD type system).

Here's an XSD 1.1 schema example, illustrating these concepts:
[1]
   <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

      <xs:element name="example" type="myPrecisionDecimal" />
  
      <xs:simpleType name="myPrecisionDecimal">
        <xs:restriction base="xs:decimal" xmlns:xerces="http://xerces.apache.org">
           <xs:totalDigits value="6" />
           <xs:fractionDigits value="4" />
           <xs:assertion test="string-length(substring-after(string($value), '.')) ge 2" 
                  xerces:message="minScale of this decimal number should be 2" />
        </xs:restriction>
      </xs:simpleType>
  
   </xs:schema>

The XSD type, "myPrecisionDecimal" defined above has following correspondences with the type, xs:decimal:
a) The facet specification, xs:totalDigits in "myPrecisionDecimal" is equivalent to the facet xs:totalDigits in xs:decimal.
b) The facet specification, xs:fractionDigits in "myPrecisionDecimal" is equivalent to the facet "maxScale" for, xs:decimal.
c) The assertion facet in, "myPrecisionDecimal" is equivalent (an user-defined attempt to equalize!) to the facet "minScale" for, xs:decimal.

When the above schema document [1], is used to validate the following XML instance:
<example>44.4</example>
The following error message is produced:
[Error] test.xml:1:24: cvc-assertion.failure: Assertion failure. minScale of this decimal number should be 2.

It's also worth noting that, the above user-defined type "myPrecisionDecimal" cannot be considered a true equivalent of XSD type, xs:precisionDecimal as defined in XSD 1.1 spec, because xs:precisionDecimal also includes values for positive and negative infinity and for "not a number", and it differentiates between "positive zero" and "negative zero" (these aspects, are not defined for xs:decimal). The above example, for "myPrecisionDecimal" only demostrates, simulating the "minScale" facet (which is not available in the type, xs:decimal) of xs:precisionDecimal.

Section 2
(Xerces-J, assertions implementation update)

Xerces-J recently implemented, an extension attribute "message" (specified in a namespace, http://xerces.apache.org, for Xerces-J XSD 1.1 implementation) on XSD 1.1 assertion instructions. The value of this attribute, needs to be an error message that will be reported by an XSD 1.1 engine upon assertions failure.

An example of this is illustrated, in the schema document above [1].

In the absence of the "message" attribute on assertions (or if it's present, but it doesn't contain any significant non-whitespace characters), the following default error message is produced by Xerces:
[Error] test.xml:1:24: cvc-assertion.3.13.4.1: Assertion evaluation ('string-l
ength(substring-after(string($value), '.')) ge 2') for element 'example' with type 'myPrecisionDecimal' did not succeed.


We could see the benefit of, the "message" attribute on assertions, which to my opinion are following:
a) For complex (& particularly, lengthy) XPath expressions in assertions, the default error messages produced by Xerces, could be quite verbose which the user's may not find convenient to view & debug. The user experience, with default assertions error messages, may be further trouble-some if there are numerous assertion evaluations for XML documents -- we could imagine the user-experience, for say maxOccurs="unbounded" specification on XML elements on which assertions apply OR let's say, there may be of the order of "> 10" different assertions.
b) We could specify, domain specific error messages with the assertions "message" attribute.

Though, the advantage of the default assertion error messages produced by Xerces is that, it prints to the user, the name of XSD type and the element/attribute involved in a particular assertions validation episode.

PS: There's been a recent issue raised with the XSD WG, which proposes addition of a "message" attribute on assertions in the XSD 1.1 language itself. The Xerces implementation of assertions "message" attribute may change in future, depending on a recommendation related to this, from the XSD WG.

I hope, that this post is useful.

Sunday, March 21, 2010

playing again with XSD 1.1 assertions

Some time ago, XSLT folks (including me!) were discussing on XSL-List the design of an XML schema, describing a product catalog. This post has nothing to do with XSLT, except that an earlier discussion on XSL-List enkindled me with yet another XSD schema use-case, to try out the Xerces-J XSD 1.1 assertions implementation. I wrote the following XSD 1.1 schema use-case, with a desire to find out if Xerces-J XSD 1.1 assertion implementation would succeed, for this example (and to cause no surprise to readers, I'm pleased to say, that Xerces passes this example!).

So here goes this example.

XML document:
  <?xml version="1.0" encoding="UTF-8" ?>
  <product id="100">
    <shortname>Sun Press, Java Book</shortname>
    <description>Java Language: Design and Programming</description>
    <author>James Gosling</author>
    <price>
      <value effective="2000-10-10" format="hard cover">25</value>
      <value effective="2005-10-10" format="hard cover">20</value>
      <value effective="2009-10-10" format="pdf" freeware="true">0</value>
    </price>
  </product>
An XSD 1.1 schema validating the above XML document:
  <?xml version="1.0" encoding="UTF-8" ?>
  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

   <xs:complexType name="Product">
     <xs:sequence>
       <xs:element name="shortName">
         <xs:simpleType>
           <xs:restriction base="xs:string">
             <xs:maxLength value="20"/>
           </xs:restriction>
         </xs:simpleType>
       </xs:element>
       <xs:element name="description" type="xs:string"/>
       <xs:element name="author" type="xs:string"/>
       <xs:element name="price">
         <xs:complexType>
           <xs:sequence>
             <xs:element name="value" maxOccurs="unbounded">
                <xs:complexType>
                  <xs:simpleContent>
                    <xs:extension base="xs:double">
                      <xs:attribute name="effective" type="xs:date" use="required"/>
                      <xs:attribute name="freeware" type="xs:boolean"/>
                      <xs:attribute name="format" use="required">
                        <xs:simpleType>
                          <xs:restriction base="xs:string">
                            <xs:enumeration value="hard cover"/>
                            <xs:enumeration value="pdf"/>
                          </xs:restriction>
                        </xs:simpleType>
                      </xs:attribute>
                      <xs:assert test="@effective lt current-date()" />
                      <xs:assert test="if (@freeware eq true()) then (@format eq 'pdf' and . eq 0)
                                       else true()" />                    
                    </xs:extension>
                  </xs:simpleContent>
                </xs:complexType>
             </xs:element>
           </xs:sequence>
           <xs:assert test="every $vl in value[position() lt last()] satisfies
                            ($vl gt $vl/following-sibling::value[1]) and 
                            ($vl/@effective lt $vl/following-sibling::value[1]/@effective)" />
         </xs:complexType>
       </xs:element>
     </xs:sequence>
     <xs:attribute name="ID" type="xs:positiveInteger" use="required"/>
   </xs:complexType>
   
   <xs:element name="product" type="Product"/>

  </xs:schema>

I don't wish to explain in detail the problem domain behind the above XML & XSD documents (I believe, readers familiar with XSD language & XML could easily understand the intent of the above example). In very shortest description, this example "illustrates a simple product catalog, describing a single product".
Here's a short explanation, about what the assertions -- highlighted with a different color (starting from assertion at top, to assertion at bottom) in above schema document are, intending to do:
1. The first assertion is checking, that the value of attribute "effective" (with a schema type, xs:date) is prior to today's date.
2. The second assertion is checking, that if value of attribute "freeware" is a boolean 'true', then value of attribute "format" must be 'pdf' & the numeric value of price should be 0.
3. The third assertion is checking, that price always reduces in future, & the effective date of the price is prior to the next price revision.

I enjoyed writing this example, and I'm glad that this worked with Xerces. The Eclipse/PsychoPath XPath 2.0 implementation (which is the underlying XPath 2 implementation, used by Xerces-J XSD 1.1 assertions implementation) also looks pretty compliant to the XPath 2 language.

I hope, that this post is useful.

Sunday, March 7, 2010

Xerces-J: XSModel serialization

There's a new API sample contributed to the Xerces-J code-base (in the schema-dev XSD 1.1, branch), which allows us to serialize a Xerces-J XSModel. This should be available in the upcoming Xerces-J release, 2.10.0.

This could be invoked by using the Java class, xs.XSSerializer.

Here's one of the use-case for this, as asked by colleagues in the community:
http://mail-archives.apache.org/mod_mbox/xerces-j-users/200611.mbox/%3c4557B05A.6000808@gael.fr%3e

Wednesday, February 24, 2010

XSD 1.1: some more assertions fun

Here are some more XSD 1.1 assertions examples (interesting one's I guess), that I tried running with Xerces-J XSD 1.1 implementation (these ones run fine, with Xerces!):

Example 1 [1]:

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="test" type="X" />
   
    <xs:complexType name="X">
      <xs:group ref="List1" />
      <xs:assert test="a and b and d" />
    </xs:complexType>
   
    <xs:complexType name="Y">
      <xs:group ref="List1" />
      <xs:assert test="a and b and c and d" />
    </xs:complexType>
   
    <xs:group name="List1">
       <xs:sequence>
         <xs:element name="a" type="xs:string" minOccurs="0"/>
         <xs:element name="b" type="xs:string" minOccurs="0"/>
         <xs:element name="c" type="xs:string" minOccurs="0"/>
         <xs:element name="d" type="xs:string" minOccurs="0"/>
       </xs:sequence>
    </xs:group>
           
  </xs:schema>

The corresponding XML instance, document is:
<test>
    <a>hello</a>
    <b>world</b>
    <!--<c>hello..</c>-->
    <d>world..</d>
  </test>

Here's the rationale/goal, that motived me to write this XSD sample:
I wanted to define a pair of XSD complex types (something like, X & Y above), such that one of the types could reuse the element particles, from the other type. If this problem could have been solved with XSD type derivation (which I attempted initially), I wanted that only one of the elements in the derived type could become optional -- element, "c" in this example (i.e, with minOccurs = 0 & maxOccurs = 1), while the other elements from the base type should have the same occurrence indicator (i.e, a mandatory indicator -- which is, minOccurs = maxOccurs = 1).

Interestingly, this problem is unsolvable with XSD type derivation (either complex type extension, or restriction mechanism).

For this schema use-case, I came up with the XSD sample above [1], which meets my goal to be able to re-use the element particles in the XSD types. The Schema above [1], defines a global group which contains a sequence of XML element definitions. All of the elements in the group, are marked as optional. Within the complex types (X & Y), the cardinality of elements (0-1 or 1-1) is enforced with XSD assertions. Defining all elements in the group, as optional allows us to reuse this list in different XSD types easily, as we can constrain the elements (say controlling the cardinality of elements, or even the contents of elements/attributes) in different contexts/types say using, assertions.

Using the above schema example [1], therefore if one wants to use a XSD type, where element "c" is optional, one would use the type, "X". While if, one wants to use a XSD type, where all elements are mandatory, one would use the type, "Y".

After having solved the use-case I had in mind (explained above), so just for fun, I wrote another schema using some more assertions.

Here's the 2nd XSD schema:

Example 2 [2]:

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

     <xs:element name="test" type="X" />
   
     <xs:complexType name="X">
       <xs:group ref="List1" />
       <xs:assert test="a and b and d" />
     </xs:complexType>
   
     <xs:complexType name="Y">
       <xs:group ref="List1" />
       <xs:assert test="a and b and c and d" />
     </xs:complexType>
   
     <xs:group name="List1">
        <xs:sequence>
           <xs:element name="a" minOccurs="0">
             <xs:complexType>
               <xs:sequence>
                 <xs:element name="a1" type="xs:string" maxOccurs="unbounded" />
               </xs:sequence>
               <xs:attribute name="aCount" type="xs:nonNegativeInteger" />
               <xs:assert test="count(a1) eq @aCount" />
             </xs:complexType>
           </xs:element>
           <xs:element name="b" type="xs:string" minOccurs="0"/>
           <xs:element name="c" type="xs:string" minOccurs="0"/>
           <xs:element name="d" type="xs:string" minOccurs="0"/>
        </xs:sequence>
     </xs:group>
           
  </xs:schema>

The schema [2] is conceptually similar, to schema [1]. The only difference between the two schemas is, that in schema [2], element "a" has complex content, while in schema [1], element "a" is defined to have simple content (which is, xs:string). In schema, [2]'s complex type we define another assertion (which enforces the constraint that, value of attribute "aCount" is equal to the number of, "a1" children of element, "a"). The assertion definition in the complex type of element, "a" in the 2nd schema, is written only to visually increase the complexity of the element a's definition (of-course, this also does increase the functional complexity of element, "a" and subsequently the complexity of contents of the global group definition, in the 2nd schema).

The 2nd schema illustrates, that a more functionally complex list of particles (a, b, c & d here) get more benefit by the schema component re-use technique (accomplished with a XSD group, and assertions) illustrated in this post.

I hope, that this post is useful.

Sunday, February 14, 2010

Xerces-J, XSD 1.1 assertions: complexType -> simpleContent -> restriction

XSD 1.1 complex types are specified by the grammar given here, in the XSD 1.1 spec:
http://www.w3.org/TR/xmlschema11-1/#declare-type

XSD complex type definitions are essentially composed of three mutually exclusive definitions, as follows:
  <complexType ...
    simpleContent |
    complexContent |
    openContent?, (group | all | choice | sequence)?, ((attribute | attributeGroup)*, anyAttribute?), assert*))
  </complexType>

The assertions specification in complexType -> simpleContent -> restriction is a bit different, that all other assertions cases on complex types (as this consists of assertion facets, as well as/or assertions on the complex type).

This is specified by the following XSD 1.1 grammar:
  <simpleContent
    id = ID
    {any attributes with non-schema namespace . . .}>
    Content: (annotation?, (restriction | extension))
  </simpleContent>

  <restriction
    base = QName
    id = ID
    {any attributes with non-schema namespace . . .}>
    Content: (annotation?, (simpleType?, (minExclusive | minInclusive | maxExclusive | maxInclusive | totalDigits | fractionDigits | maxScale | minScale | length | minLength | maxLength | enumeration | whiteSpace | pattern | assertion | {any with namespace: ##other})*)?, ((attribute | attributeGroup)*, anyAttribute?), assert*)
  </restriction>

The XSD definition for xs:restriction above specifies assertions something like following:
assertion*, ..., assert*

Here, xs:assertion (with cardinality, 0-n) is a facet for the simple type value (specified by, complexType -> simpleContent). Whereas, xs:assert (with cardinality, 0-n) is an assertion definition on the complex type (which has access to the element tree, like the XML element itself, and it's attributes if there are any). xs:assertion definitions on, complexType -> simpleContent -> restriction do not have access to the element tree (on which the complex type is applicable), and can only access the simple type value (using, the implicit assertion variable $value, having a XSD type specified by the definition, <xs:restriction base = QName ...) of the element in the context.

Here's a small fictitious examples, illustrating these concepts:

XML document [1]:
  <A a="15">Example A</A>

XSD 1.1, Schema [2]:
  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="A">
      <xs:complexType>
        <xs:simpleContent>
          <xs:restriction base="myBase">    
            <xs:assertion test="contains($value, 'Example')" />
            <xs:assert test="@a mod 5 = 0" />    
          </xs:restriction>
        </xs:simpleContent>
      </xs:complexType>
    </xs:element>
  
    <xs:complexType name="myBase">
      <xs:simpleContent>
         <xs:extension base="xs:string"> 
           <xs:attribute name="a" type="xs:int" />  
         </xs:extension>
      </xs:simpleContent> 
    </xs:complexType>

  </xs:schema>

In the Schema above [2], there are two assertions (shown with bold emphasis) specified on the XSD type. One of assertions is a facet for the simple content, and the other is an assertion on the complex type.

I believe, the above Schema is simple enough and self-explanatory, to illustrate the points I've tried to explain in this post.

Actually, what prompted me to write this post, was that there was a minor bug in complexType -> simpleContent -> restriction facet processing in Xerces-J XSD 1.1 SVN code, which we could fix today, and the fix is now available in Xerces-J SVN repository.

Interestingly, this fix was there in Xerces-J SVN during some past Xerces SVN version. But going forward with assertions development, this bug got introduced, and now has been fixed again.

Saturday, February 6, 2010

PsychoPath XPath2 processor update: fn:name() function fix

While writing following blog post, http://mukulgandhi.blogspot.com/2010/01/xsd-11-wild-cards-in-compositor-and.html (dated, Jan 31, 2010) [1], I actually unearthed a bug in PsychoPath XPath 2 processor, whereby the XPath2 fn:name() function didn't evaluate properly with zero arity (it raised a "context undefined" exception, even if a context item existed).

This bug led me to use the, fn:local-name() (whose implementation was correct) function instead, for the above mentioned blog post [1].

The good news is, that now this bug with fn:name() function is fixed (ref, https://bugs.eclipse.org/bugs/show_bug.cgi?id=301539).

For the example given in the blog post [1], the given XSD 1.1 assertion could now be written like following, as well:
  <xs:assert test="(*[1]/name() = ('fname', 'lname')) and 
                 (*[2]/name() = ('fname', 'lname'))" />

(instead, of the "local-name" function as used in the mentioned blog post [1])