Showing posts with label eclipse. Show all posts
Showing posts with label eclipse. Show all posts

Sunday, September 5, 2010

XSD 1.1: Xerces-J implementation updates

Over the past one or two months, there have been few interesting changes happening at Xerces-J XML Schema 1.1 implementation. I feel obliged to share these enhancements with the XML Schema community, and also with folks at Eclipse WTP (where we enhanced few "schema aware" components of PsychoPath XPath 2.0 engine, to support these recent Xerces enhancements -- I think we improved the design of typed values of XML element and attribute XDM nodes in PsychoPath XPath2 engine, in case the XDM node has a type annotation of kind XML Schema simpleType, with varieties list or union).

Here's a summary of XML Schema 1.1 implementation changes that have recently been completed with Xerces (available at Xerces SVN repos as of now), which are planned to be part of the Xerces-J 2.11.0 release, planned to take please during November 2010 time frame.

1. Xerces-J now has a complete implementation of XML Schema 1.1 conditional inclusion functionality. The Xerces-J 2.10.0 release had implementation of XML Schema 1.1 conditional inclusion vc:minVersion and vc:maxVersion attributes. Xerces-J now supports all of "conditional inclusion" attributes as specified by the XML Schema 1.1 spec. The "conditional inclusion" attributes that are now newly supported in Xerces-J are: vc:typeAvailable, vc:typeUnavailable, vc:facetAvailable and vc:facetUnavailable. All of XML Schema 1.1 built-in types and facets are now supported by Xerces-J related to XML Schema 1.1 "conditional inclusion" components.

2. There are few interesting changes that have happened to Xerces-J XML Schema 1.1 assertions implementation as well, that are planned to be part of Xerces-J 2.11.0 release. Xerces now has an improved assertions evaluation processing on XML Schema (1.1) simple types, with varieties 'list' and 'union'.

2.1 Enhancements to assertions evaluation on simpleType -> list:

Here's an example of XML Schema 1.1 assertions on an xs:list schema component:
[XML Schema 1]
   <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

      <xs:element name="Example" type="EXAMPLE_LIST" />
   
      <xs:simpleType name="EXAMPLE_LIST">
         <xs:list>
            <xs:simpleType>
               <xs:restriction base="xs:integer">
                  <xs:assertion test="$value mod 2 = 0" />
               </xs:restriction>
            </xs:simpleType>
         </xs:list>
      </xs:simpleType>
   
   </xs:schema> 

If an XML instance document has a structure something like following:
[XML 1]
<Example>1 2 3</Example>

And if this XML instance document ([XML 1]) is validated by the above XML schema ([XML Schema 1]), Xerces-J would report error messages like following (assuming the name of XML document was, test.xml):
[Error] test.xml:1:25: cvc-assertion.3.13.4.1: Assertion evaluation ('$value mod 2 = 0') for element 'Example' with type '#anonymous' did not succeed. Assertion failed for an xs:list member value '1'.
[Error] test.xml:1:25: cvc-assertion.3.13.4.1: Assertion evaluation ('$value mod 2 = 0') for element 'Example' with type '#anonymous' did not succeed. Assertion failed for an xs:list member value '3'.


An assertion must evaluate on every 'simpleType -> list' item (which is validated by the itemType of xs:list) in an XML instance document. Xerces now does this, and needed error messages are displayed in case of schema assertion failures.

2.2 Enhancements to assertions evaluation on simpleType -> union:

Here's an example of XML Schema 1.1 assertions on an xs:union schema component:
[XML Schema 2]
   <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
   
      <xs:element name="Example">
         <xs:simpleType>
            <xs:union memberTypes="MYDATE xs:integer" />
         </xs:simpleType>
      </xs:element>
   
      <xs:simpleType name="MYDATE">
         <xs:restriction base="xs:date">
            <xs:assertion test="$value lt current-date()" />
         </xs:restriction>
      </xs:simpleType>

   </xs:schema>

If an XML instance document has a structure something like following:
[XML 2]
<Example>2010-12-05</Example>

And this instance document is validated by the schema document, [XML Schema 2] the following error message is displayed by Xerces:
[Error] temp.xml:1:30: cvc-assertion.union.3.13.4.1: Element 'Example' with value '2010-12-05' is not locally valid. One or more of the assertion facets on an element's schema type, with variety union, have failed.

Xerces tried to validate an atomic value '2010-12-05' both with schema types xs:integer and MYDATE. Since none of these types could successfully validate this atomic value, and an assertion failed in the process of these validation checks, the relevant assertion failure was reported by Xerces.

If the XML schema, [XML Schema 2] tries to validate the XML instance document:
<example>10</Example>

no validation failures are reported in this case, since an atomic value '10' conforms to the schema type xs:integer, which results in an overall validation success of the atomic value with an 'union' schema type.

I'm ending this blog post now. Stay tuned for more news here :)

And I hope, that this post was useful.

Saturday, June 19, 2010

Xerces-J 2.10.0 released!

I'm pleased to extend the announcement of Xerces-J 2.10.0 release (released today. ref, http://markmail.org/message/m73xwkrmyacppu3l).

Xerces-J 2.10.0 is now available on the Xerces site, http://xerces.apache.org/xerces2-j/ to be used by community. This is a release after nearly two and a half years of the previous Xerces-J release, 2.9.1. This is a significant milestone at Xerces, with lot's of new features & bug fixes/enhancements.

Xerces-J 2.10.0 provides two versions of the Xerces distributables:
1. Xerces2 Java 2.10.0 - this is a maintainence release for 2.9.1 release (having bug fixes & enhancements), with essentially the same parsing & API support as 2.9.1 (except I think addition of support for StAX 1.0 event API & Element Traversal API -- more details are available in Xerces release notes).
2. Xerces2 Java 2.10.0 (XML Schema 1.1) (Beta) - this release supports a partial experimental implementation of the XML Schema 1.1 Structures and Datatypes Working Drafts (December 2009), along with all the changes available in "Xerces2 Java 2.10.0" release (point 1).

PS: I would also like to extend special thanks to the Eclipse/PsychoPath XPath 2.0 team members, Dave Carver & Jesper S Møller for helping producing an excellent XPath 2.0 engine, which underlies Xerces XML Schema 1.1 assertions and type-alternatives implementation. Special thanks also to Andrea Bittau, who originally authored the PsychoPath engine.

I hope the community likes the Xerces 2.10.0 release!

Please feel free to discuss about Xerces-J at:

Sunday, March 21, 2010

playing again with XSD 1.1 assertions

Some time ago, XSLT folks (including me!) were discussing on XSL-List the design of an XML schema, describing a product catalog. This post has nothing to do with XSLT, except that an earlier discussion on XSL-List enkindled me with yet another XSD schema use-case, to try out the Xerces-J XSD 1.1 assertions implementation. I wrote the following XSD 1.1 schema use-case, with a desire to find out if Xerces-J XSD 1.1 assertion implementation would succeed, for this example (and to cause no surprise to readers, I'm pleased to say, that Xerces passes this example!).

So here goes this example.

XML document:
  <?xml version="1.0" encoding="UTF-8" ?>
  <product id="100">
    <shortname>Sun Press, Java Book</shortname>
    <description>Java Language: Design and Programming</description>
    <author>James Gosling</author>
    <price>
      <value effective="2000-10-10" format="hard cover">25</value>
      <value effective="2005-10-10" format="hard cover">20</value>
      <value effective="2009-10-10" format="pdf" freeware="true">0</value>
    </price>
  </product>
An XSD 1.1 schema validating the above XML document:
  <?xml version="1.0" encoding="UTF-8" ?>
  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

   <xs:complexType name="Product">
     <xs:sequence>
       <xs:element name="shortName">
         <xs:simpleType>
           <xs:restriction base="xs:string">
             <xs:maxLength value="20"/>
           </xs:restriction>
         </xs:simpleType>
       </xs:element>
       <xs:element name="description" type="xs:string"/>
       <xs:element name="author" type="xs:string"/>
       <xs:element name="price">
         <xs:complexType>
           <xs:sequence>
             <xs:element name="value" maxOccurs="unbounded">
                <xs:complexType>
                  <xs:simpleContent>
                    <xs:extension base="xs:double">
                      <xs:attribute name="effective" type="xs:date" use="required"/>
                      <xs:attribute name="freeware" type="xs:boolean"/>
                      <xs:attribute name="format" use="required">
                        <xs:simpleType>
                          <xs:restriction base="xs:string">
                            <xs:enumeration value="hard cover"/>
                            <xs:enumeration value="pdf"/>
                          </xs:restriction>
                        </xs:simpleType>
                      </xs:attribute>
                      <xs:assert test="@effective lt current-date()" />
                      <xs:assert test="if (@freeware eq true()) then (@format eq 'pdf' and . eq 0)
                                       else true()" />                    
                    </xs:extension>
                  </xs:simpleContent>
                </xs:complexType>
             </xs:element>
           </xs:sequence>
           <xs:assert test="every $vl in value[position() lt last()] satisfies
                            ($vl gt $vl/following-sibling::value[1]) and 
                            ($vl/@effective lt $vl/following-sibling::value[1]/@effective)" />
         </xs:complexType>
       </xs:element>
     </xs:sequence>
     <xs:attribute name="ID" type="xs:positiveInteger" use="required"/>
   </xs:complexType>
   
   <xs:element name="product" type="Product"/>

  </xs:schema>

I don't wish to explain in detail the problem domain behind the above XML & XSD documents (I believe, readers familiar with XSD language & XML could easily understand the intent of the above example). In very shortest description, this example "illustrates a simple product catalog, describing a single product".
Here's a short explanation, about what the assertions -- highlighted with a different color (starting from assertion at top, to assertion at bottom) in above schema document are, intending to do:
1. The first assertion is checking, that the value of attribute "effective" (with a schema type, xs:date) is prior to today's date.
2. The second assertion is checking, that if value of attribute "freeware" is a boolean 'true', then value of attribute "format" must be 'pdf' & the numeric value of price should be 0.
3. The third assertion is checking, that price always reduces in future, & the effective date of the price is prior to the next price revision.

I enjoyed writing this example, and I'm glad that this worked with Xerces. The Eclipse/PsychoPath XPath 2.0 implementation (which is the underlying XPath 2 implementation, used by Xerces-J XSD 1.1 assertions implementation) also looks pretty compliant to the XPath 2 language.

I hope, that this post is useful.

Saturday, February 6, 2010

PsychoPath XPath2 processor update: fn:name() function fix

While writing following blog post, http://mukulgandhi.blogspot.com/2010/01/xsd-11-wild-cards-in-compositor-and.html (dated, Jan 31, 2010) [1], I actually unearthed a bug in PsychoPath XPath 2 processor, whereby the XPath2 fn:name() function didn't evaluate properly with zero arity (it raised a "context undefined" exception, even if a context item existed).

This bug led me to use the, fn:local-name() (whose implementation was correct) function instead, for the above mentioned blog post [1].

The good news is, that now this bug with fn:name() function is fixed (ref, https://bugs.eclipse.org/bugs/show_bug.cgi?id=301539).

For the example given in the blog post [1], the given XSD 1.1 assertion could now be written like following, as well:
  <xs:assert test="(*[1]/name() = ('fname', 'lname')) and 
                 (*[2]/name() = ('fname', 'lname'))" />

(instead, of the "local-name" function as used in the mentioned blog post [1])

Saturday, January 16, 2010

XPath 2.0: PsychoPath processor update

I just noticed release of the Eclipse, WTP (Web Tools Platform) 3.2M4 milestone (released on, December 11th, 2009). The WTP Source Editing release, in WTP 3.2M4 milestone, includes an enhanced PsychoPath XPath 2.0 engine.

Xerces-J users using the XSD 1.1 support (specifically XSD 1.1 assertions and CTA/type alternatives), can use the latest PsychoPath library from http://www.eclipse.org/webtools/. The latest development PsychoPath library can also be downloaded, from https://build.eclipse.org/hudson/view/WTP/job/cbi-wtp-wst.xsl.psychopath/.

Dave Carver, has been doing a great job in putting this all together.

Saturday, December 5, 2009

XPath 2.0: PsychoPath XPath processor update

I've just run all the PsychoPath XPath 2 processor (an Eclipse Web Tools, Source Editing sub-project) W3C test-suite tests, and here are the results for them:

Tests: 8143
Errors: 0
Failures: 0

So it seems, PsychoPath XPath engine passes, 100% of the W3C XPath 2.0 test suite, and some of it's own tests.

This should be a moment of cheer, and wow!

It also looks, like that the upcoming Xerces-J release, 2.10.0 (ref, http://wiki.apache.org/xerces/November2009) would be getting almost a compliant XPath 2.0, engine for XSD 1.1 assertions and CTA.

Ref: An earlier post about PsychoPath status: http://mukulgandhi.blogspot.com/2009/09/psychopath-xpath-20-processor-update.html.

Tuesday, September 8, 2009

PsychoPath XPath 2.0 processor update

Some time ago, I wrote a progress update, about PsychoPath XPath 2.0 processor's (an Eclipse, Web Tools Source Editing subproject) compliance with, W3C XPath 2.0 test suite.

As of today, following are the test suite success rate, for PsychoPath engine:

Tests: 8137
Failures: 163
Errors: 5

This reflects a success rate of: 97.9%.

I think, these recent PsychoPath improvements with W3C XPath 2.0 test suite, reflect a much increased quality of PsychoPath product.

I am hoping, we would reach 100% test suite compliance, with PsychoPath in a near future!

Credits, for most of the recent PsychoPath improvements, should go to Dave Carver and Jesper Steen Møller.

Update on 2009-09-18: Took an update today from the Eclipse CVS servers, for PsychoPath source code, and following are the W3C XPath 2.0 test results:
Tests: 8137
Failures: 132
Errors: 5

Update on 2009-10-17: As of today, following are the W3C XPath 2.0 test results:
Tests: 8137
Failures: 81
Errors: 1

This reflects the W3C XPath 2.0 test suite pass percentage, for PsychoPath engine of about 98.9%. It seems, we are moving closer to the 100% test success rate, for PsychoPath. Dave Carver has been working a lot, on these improvements during last few days.

Saturday, July 4, 2009

PsychoPath XPath 2.0 processor update

Dave Carver and I have been trying to improve the Eclipse XPath 2.0 processor (a.k.a PsychoPath) during last couple of weeks. My motivation to keep working on PsychoPath engine has been a desire, to help Eclipse and Apache (Apache Xerces-J uses PsychoPath engine for XML Schema 1.1 processing) communities to be able to have a highly compliant XPath 2.0 engine.

Dave has written today, a progress update of PsychoPath development on this blog. I feel, we now have a pretty good XPath 2.0 implementation with PsychoPath. We are continuing to work on remaining non-compliant items, with PsychoPath. The remaining non compliance cases, to my opinion are near edge cases which users don't use too often. But we'll continue to solve them, with each future day and weeks.

Update on 2009-07-11: During last couple of days, Dave Carver has made quite a few useful improvements to PsychoPath, and the W3C XPath 2.0 test suite within Eclipse. I took an update today of the latest PsychoPath sources, and the XPath 2.0 test suite, and following are the latest test results:

Total tests: 8137
Failures: 811
Errors: 48

This reflects, the test pass success rate of about 89.5%. I think, this is quite good. Lot of credit of these improvements should go to Dave Carver. Dave has single handedly, created a JUnit version of the full W3C XPath 2.0 test suite, which is in itself a great feat! Having JUnit tests, helps us tremendously to run the XPath 2.0 tests, from within Eclipse.

Update on 2009-08-09: Following are the current PsychoPath test data:
Total tests: 8137
Failures: 386
Errors: 24
This reflects a test suite pass percentage of about, 95% which looks very impressive. The test suite code coverage, is about 75-80%.
Lot of credit for the latest PsychoPath improvements should go to, "Jesper S Møller" who has recently volunteered to help improve PsychoPath with the XPath 2.0 test suite. Dave Carver is also putting in his time, on PsychoPath improvements.

Saturday, June 20, 2009

Became Eclipse WTP committer

I was nominated as Eclipse WTP project committer, for the WTP Source Editing subproject.

As per the voting process, for becoming an Eclipse project committer, the Eclipse WTP Source Editing team, granted me project committership on 18, Jun 2009.

My contributions to PsychoPath XPath 2.0 engine (which is one of the components in WTP Source Editing tooling), helped me become a committer to this project.

I am happy to be included in the Eclipse WTP team. I look forward to contribute more to PsychoPath, and other WTP components. Apart from PsychoPath, I look forward to work on WTP XSL components in near future.

Tuesday, May 26, 2009

PsychoPath XPath 2.0 processor update

We recently implemented quite a few built in XSD numeric data types in PsychoPath XPath 2.0 processor (ref, http://www.w3.org/TR/xmlschema-2/#built-in-datatypes). Now all (I mean, really all of xs:decimal ones :)) the data types in the xs:decimal hierarchy are available in PsychoPath, and these should be available in Eclipse WTP 3.2 M1 (which should be released sometime soon after the Eclipse Galileo release, at around June 09' end).

Now almost all the major built in Schema types are available in PsychoPath, except for few subtypes of xs:string (like xs:normalizedString, xs:token etc.). These shouldn't be much difficult to add.

Dave Carver reported, that the improvements we have done recently in PsychoPath have significantly improved it's compliance to the W3C XPath 2.0 test suite.

PsychoPath processor version is now enhanced from 1.0 to 1.1.

Sunday, May 17, 2009

Xerces-J XSD 1.1 assertions and PsychoPath XPath 2.0 processor update

I recently contributed few patches to the Eclipse PsychoPath XPath 2.0 engine, to support Schema aware XPath (2.0) expressions. These patches would enhance Schema aware support in PsychoPath XPath2 engine, for element and attribute nodes, for the XML Schema primitive types.

These enhancements in PsychoPath engine would make XPath expressions like following possible, to be evaluated by PsychoPath engine:

person/@dob eq xs:date('2006-12-10') // if dob is an attribute, and of schema type xs:date

person/dob eq xs:date('2006-12-10') // if dob is an element, and of schema type xs:date

@max ge @min // this would work if 'max' and 'min' have say schema types, xs:int

The patch in my local environment already exhibits these improvements. As promised by Dave Carver (the PsychoPath engine project lead), users would likely get these improvements in Eclipse WTP (Web Tools Project) 3.2.
2009-05-24: These changes are now committed to the Eclipse CVS server, and the improvements are flagged to be delivered in Eclipse WTP 3.2 M1, which should be quite sooner. Thanks to Dave Carver for testing all my patches, and committing them to the server.

PsychoPath engine already has a framework (thanks to Andrea Bittau and his team) for supporting Schema awareness (based on the Xerces-J XSD schema model). I just added in small pieces of code in attribute and element node implementations (particularly, improving the "typed value" of attribute and element nodes for built in XSD schema types), to enhance schema aware support.

I think, we are gradually moving to a more mature schema aware support in PsychoPath.

I'm also using these new PsychoPath processor capabilities, to implement schema aware XPath 2.0 evaluations in Xerces-J XSD assertions support.

I'm currently working on to construct a typed XPath data model instance, for XSD 1.1 assertions evaluations. Having this capability, would allow users to write XPath expressions like, following:

@max ge @min

or

person/@dob eq xs:date('2006-12-10')

In the absence of this (i.e, typed XDM nodes), currently users have to make explicit cast operations, like following:

xs:int(@max) ge xs:int(@min)

or

xs:date(person/@dob) eq xs:date('2006-12-10')

The XML Schema 1.1 assertions spec recommends a typed XDM instance.

We hope to provide this capability within Xerces-J, inline with the XML Schema 1.1 assertions specification.

2009-05-23: These improvements are now implemented, and I've submitted the code improvements to the Apache Xerces-J JIRA server. I'm hoping, we'll have these improvements committed on the Xerces-J SVN server some time soon.