Showing posts with label java. Show all posts
Showing posts with label java. Show all posts

Friday, March 29, 2019

XSLT 1.0 transformations for large xml input documents

I thought that, this topic could be of interest to XML community.

I've discovered an interesting aspect of JAXP API (Java API for XML Processing) that seems to have relations to streaming that we talk with XSLT 3.0. Please see following document, and an example given in its section 4.12 (that explains JAXP's StAX API and using it with JAXP's transformation APIs)


Using the cited JAXP code in above document, one can transform very large XML input documents (I've tried an XML input document with size of about 700 MB, that worked) using XSLT 1.0 (the JDK's built in JAXP implementation can do this. I've tried with JDK 1.8 which works fine for this). It can do certain kinds of XSLT 1.0 transformations with very large XML input documents, very well. When doing the same transformations with XSLT 2.0, or with XSLT 3.0 in non streaming way, we would usually get following errors 'java.lang.OutOfMemoryError: Java heap space'.

I've written few complete examples for this topic here, https://github.com/mukulga/largexml_xslt10.

Notes: My intention for writing this blog post is not to endorse in any way that XSLT 1.0 is better than XSLT 2.0/3.0 for every aspect. XSLT 2.0/3.0 have various new language features as compared to XSLT 1.0, that raise the productivity of XSLT developers and allow development of XSLT stylesheets with ease that could be much more complex in terms of functionality, than with XSLT 1.0.

Saturday, February 25, 2012

modular XML instances and modular XSD schemas

I was playing with some new ideas lately related to exploring design options, to construct modular XML instance documents vs/and modular XSD schema documents and thought to write my findings as a blog post here.

I believe, there are primarily following concepts related to constructing modular XML documents (and XSD schemas) when XSD validation is involved:
1. Modularize XML documents using the XInclude construct.
2. Modularize an XSD document via <xs:include> and <xs:import>. The <xs:include> construct maps significantly to modularlity concepts in XSD schemas, and <xs:import> is necessary (necessary in XSD 1.0, and optional in XSD 1.1) to compose (and also to modularize) XSD schemas coming from two or more distinct XML namespaces.

I don't intend to delve much in this post into concepts related to XSD constructs <xs:include> and <xs:import> since these are well known within the XSD and XML communities. In this post, I would tend to primarily focus on XML document modularization via the XInclude construct and presenting few thoughts about various design options (I don't claim to have covered every design option for these use cases, but I feel that I would cover few of the important ones) to validate such XML instance documents via XSD validation.

What is XInclude?
This is an XML standards specification, that defines about how to modularize any XML document information. The primary construct of XInclude is an <xi:include> XML element. Following is a small example of an XInclude aware XML document,

z.xml

<z xmlns:xi="http://www.w3.org/2001/XInclude">
    <xi:include href="x.xml"/>
    <xi:include href="y.xml"/>
</z>

x.xml

<x>
    <a>1</a>
    <b>2</b>
</x>

y.xml

<y>
    <p>5</p>
    <q>6</q>
</y>

We'll be using the XML document, z.xml provided above that is composed from other XML documents via an XInclude meta-data, to provide to an XSD validator for validation.

I essentially discuss here, the XSD schema design options to validate an XML instance document like z.xml above. Following are the XSD design options (that cause successful XML instance validations) that currently come to my mind for this need, along with some explanation of the corresponding design rationale:

XS1:

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="z">
          <xs:complexType>
               <xs:sequence>
                    <xs:any processContents="skip" minOccurs="2" maxOccurs="2"/>
               </xs:sequence>
          </xs:complexType>
    </xs:element>
   
</xs:schema>

This schema is written with a view that, the XML document (i.e z.xml) would be validated with XInclude meta-data unexpanded. An xs:any wild-card in this schema would weakly validate (since this wild-card declaration only requires *any particular* XML element to be present in an instance document, which is validated by this wild-card. the wild-card here doesn't specify any other constraint for it's corresponding XML instance elements) each of the included XML document element roots (i.e XML elements "x" and "y").

XS2:

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

        <xs:element name="z">
                <xs:complexType>
                     <xs:complexContent>
                         <xs:restriction base="T1">
                              <xs:sequence>
                                   <xs:element name="include"  minOccurs="2" maxOccurs="2" targetNamespace="http://www.w3.org/2001/XInclude"/>
                             </xs:sequence>
                         </xs:restriction>
                    </xs:complexContent>
                </xs:complexType>
        </xs:element>
   
    <xs:complexType name="T1" abstract="true">
          <xs:sequence>
               <xs:any processContents="skip" maxOccurs="unbounded"/>
          </xs:sequence>
    </xs:complexType>
   
</xs:schema>

This schema is also written with a view that, the XML document (i.e z.xml) would be validated with XInclude meta-data unexpanded. But this schema specifies slightly stronger XSD validation constraints as compared to the previous example (stronger in a sense that, this schema declares an XML element and specifies it's name and an namespace). This schema would need an XSD 1.1 processor, since the element declaration specifies a "targetNamespace" attribute. An XSD 1.0 version of this design approach is possible, which would involve using an XSD <xs:import> element to import XSD components from the XInclude namespace.

XS3:

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

       <xs:element name="z">
              <xs:complexType>
                  <xs:sequence>
                       <xs:any processContents="skip" minOccurs="2" maxOccurs="2" namespace="http://www.w3.org/2001/XInclude"/>
                 </xs:sequence>
                 <xs:assert test="count(*[local-name() = 'include']) = 2"/>
                 <xs:assert test="deep-equal((*[1] | *[2])/@*/name(), ('href','href'))"/>
             </xs:complexType>
      </xs:element>
   
</xs:schema>

This schema is also written with a view that, the XML document (i.e z.xml) would be validated with XInclude meta-data unexpanded. But this schema enforces XSD validation even more strongly than the example "XS2" above (since this schema also requires the XInclude attribute "href" to be present on the XInclude meta-data, which the previous XSD schema doesn't enforce). This schema validates the names of XML instance elements, that are intended to be XInclude meta-data via XSD 1.1 <assert> elements (this may not be the best XSD validation approach, but such an XSD design idiom is now possible with XSD 1.1 language).

XS4:

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="z">
         <xs:complexType>
               <xs:sequence>
                    <xs:element name="x">
                         <xs:complexType>
                             <xs:sequence>
                                  <xs:element name="a" type="xs:integer"/>
                                 <xs:element name="b" type="xs:integer"/>
                            </xs:sequence>
                        </xs:complexType>
                    </xs:element>
                    <xs:element name="y">
                         <xs:complexType>
                             <xs:sequence>
                                  <xs:element name="p" type="xs:integer"/>
                                  <xs:element name="q" type="xs:integer"/>
                             </xs:sequence>
                        </xs:complexType>
                   </xs:element>
              </xs:sequence>
         </xs:complexType>
     </xs:element>
   
</xs:schema>

This schema is written with a view that, the XML document (i.e z.xml) would be validated with XInclude meta-data expanded. This schema specifies the strongest of XSD validation constraints as compared to the previous three approaches (strongest in a sense that, the internal structure of XML element instances "x" and 'y" are now completely specified by the XSD document).

But to make this XSD validation approach to work, the XInclude meta-data needs to be expanded and the expanded XML infoset needs to be supplied to the XSD validator for validation. This would require an XInclude processor (like Apache Xerces), that plugs within the XML parsing stage to expand the <xi:include> tags.

For the interest of readers, following are few java code snippets (the skeletal class structure and imports are omitted to keep the text shorter) that enable XInclude processing and supplying the resulting XML infoset (i.e post the XInclude meta-data expansion) to the Xerces XSD validator,

try {           
     Schema schema = schemaFactory.newSchema(getSaxSource(xsdUri, false));
     Validator validator = schema.newValidator();
     validator.setErrorHandler(new ValidationErrHandler());
     validator.validate(getSaxSource(xmlUri, true));
}
catch(SAXException se) {
     se.printStackTrace();
}
catch (IOException ioe) {
     ioe.printStackTrace();
}

private SAXSource getSaxSource(String docUri, boolean isInstanceDoc) {

     XMLReader reader = null;

     try {
          reader = XMLReaderFactory.createXMLReader();
          if (isInstanceDoc) {
              reader.setFeature("http://apache.org/xml/features/xinclude", true);
              reader.setFeature("http://apache.org/xml/features/xinclude/fixup-base-uris", false);
          }
     }
     catch (SAXException se) {
          se.printStackTrace();
     }

     return new SAXSource(reader, new InputSource(docUri));

}
     
class ValidationErrHandler implements ErrorHandler {

      public void error(SAXParseException spe) throws SAXException {
           String formattedMesg = getFormattedMesg(spe.getSystemId(), spe.getLineNumber(), spe.getColumnNumber(), spe.getMessage());
           System.err.println(formattedMesg);
      }

      public void fatalError(SAXParseException spe) throws SAXException {
             String formattedMesg = getFormattedMesg(spe.getSystemId(), spe.getLineNumber(), spe.getColumnNumber(), spe.getMessage());
             System.err.println(formattedMesg);
      }

      public void warning(SAXParseException spe) throws SAXException {
           // NO-OP           
      }
       
}

private String getFormattedMesg(String systemId, int lineNo, int colNo, String mesg) {
      return systemId + ", line "+lineNo + ", col " + colNo + " : " + mesg;   
}

Summary: I would ponder that, is devising the above various XSD design approaches beneficial for an XSD schema design that involves validating XML instance documents that contain <xi:include> meta-data directives? My thought process with regards to the above presented XSD validation options had following concerns:
1) Providing various degrees of XSD validation strenghts for <xi:include> directives (essentially the un-expanded and expanded modes).
2) Exploring some of the new XML validation idioms offered by XSD 1.1 language for the use cases presented above (essentially using "targetNamespace" attribute on xs:element elements, and using <assert> elements).
3) Exploring the java SAX and JAXP APIs to enable XInclude meta-data expansion, and providing a SAXSource object containing an XInclude expanded XML infoset which is subsequently supplied further to the XSD validation pipeline.

I hope that this post was useful.

Saturday, June 4, 2011

Dealing with multiple roots within an XML Schema

I've been thinking on this problem for a while, and have collected some opinions, which I'm presenting here.

We'll be working with the following XML Schema documents:

a.xsd [1]
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="x" type="xs:string"/>

    <xs:element name="y" type="xs:string"/>

    <xs:element name="z">
       <xs:complexType>
          <xs:sequence>
             <xs:element ref="x"/>
             <xs:element ref="y"/>
          </xs:sequence>
       </xs:complexType>
    </xs:element>

</xs:schema>

b.xsd [2]
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:include schemaLocation="c.xsd"/>

    <xs:element name="z">
       <xs:complexType>
          <xs:sequence>
             <xs:element ref="x"/>
             <xs:element ref="y"/>
          </xs:sequence>
       </xs:complexType>
    </xs:element>

</xs:schema>

c.xsd [3]
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="x" type="xs:string"/>

    <xs:element name="y" type="xs:string"/>

</xs:schema>
The schema documents [1] and [2] are equivalent for the purpose of validating an XML instance document (it's just that the schema document b.xsd includes c.xsd).

Our application requires the following XML document to be successfully validated, by the schemas [1] or [2] above:

z.xml [4]
<z>
   <x>hello</x>
   <y>world</y>
</z>
All of this is just fine, and XML document [4] get's successfully validated by the schemas [1] or [2] above.

But the above schema design (either [1] or [2]), may present following problems sometimes:

The side effect of schema documents [1] or [2] is to also successfully validate the following XML documents,

<x>...</x>

OR

<y>...</y>
Since elements "x" and "y" are also valid roots defined in the schema (due to the global declarations of elements "x" and "y" in the schema). But the purpose of defining elements "x" and "y" in the schema, is to include them by reference else where in the schema document (as in element declaration "z" in schemas [1] or [2]).

This kind of schema design is sometimes necessary, for the reasons of modularity (for e.g using one declaration at multiple places) and re-usability (for e.g. by including a foreign schema in our own schema) -- this design can be more beneficial, if the complexity of the schema (for e.g with more schema components, and more & deep nesting of schema components) is more.

So how do we live with following design trade-off,
i.e having schema like [1] or [2] above (which gives us benefits of modularity and re-usability) and also a side effect of these schema documents validating multiple root elements (which risks an application to accept invalid XML documents -- in this example, the roots "x" and "y" are invalid for the application, while the root "z" is valid).

In this use case, if we desire that the application must reject XML documents with roots "x" or "y" but should accept documents with root "z", then to my opinion this problem cannot be solved completely with XML Schema language (there's no way currently in the XML Schema language, to forbid validating the top level XML element in instance document, with a global schema element declaration).

Solving this problem would require a little bit of non schema solution (for e.g a SAX java add-on along with schema validation).

Here's a sketch of a java SAX application which can be and-ed with the XML Schema validation (using schemas above), to achieve the desired overall XML validation effect (i.e successful validation for the root element "z" and prohibiting the XML roots "x" and "y"),

(java imports are omitted to keep the text short)
class SAXUtil extends DefaultHandler {

     String[] excludedElems = new String[] {"x", "y"};

     private boolean isRootElemOK(String docUri) {  
        boolean rootElemOk = true;
  
        try {
           SAXParserFactory saxParserFactory = SAXParserFactory.newInstance();
           saxParserFactory.setNamespaceAware(true);
           SAXParser saxParser = saxParserFactory.newSAXParser();
           saxParser.parse(docUri, this);
        }
        catch(SAXException ex) {
           if (ex instanceof RootElementSAXException) {
              RootElementSAXException expObj = (RootElementSAXException) ex;
              if ("100".equals(expObj.getErrCode())) {
                 rootElemOk = false;  
              }
           }
        }
  
        return rootElemOk;  
     }

     public void startElement(String uri, String localName, String qName, Attributes attributes) throws SAXException {
        if (!isElementAllowed(localName)) {
           throw new RootElementSAXException("100"); 
        }
        throw new RootElementSAXException("101");
     }

     private boolean isElementAllowed(String localName) {    
        boolean elemAllowed = true;       
        for (int elemIdx = 0; elemIdx < excludedElems.length; elemIdx++) {
           if (localName.equals(excludedElems[elemIdx])) {
              elemAllowed = false;
              break;
           }
        }       
        return elemAllowed;       
     }

     class RootElementSAXException extends SAXException {
        String errorCode;
  
        public RootElementSAXException(String errorCode) {
           this.errorCode = errorCode;
        }
  
        public String getErrCode() {
           return errorCode;
        }
     }

} // class SAXUtil

Following is an algorithmic summary of the above java validation add-on,

1) A SAX parser is instantiated and parsing is invoked/triggered with the parse() method.

2) The SAX parser cannot go beyond parsing the root element -- the algorithm is intentionally designed in this way (since the SAX "startElement" callback method would always throws an exception [user defined exception, RootElementSAXException], upon encountering the top most element). The constructor parameter to the exception ("100" or "101" in this case) RootElementSAXException determines, whether the top most element was allowed or not (which is determined by an element name forbidden-list "excludedElems", defined in the above java class).

Notes:

1) To terminate the SAX parsing prior to completing parsing the whole of XML document, a SAXException can be thrown from the SAX call back methods. A custom exception class (like RootElementSAXException in the above example) is desirable, to distinguish our application designed exception from the built in SAXException events.

2) It's recommended to use SAX API for this use case, since it'll be much more efficient than for e.g using DOM APIs, which would load the whole of XML document in memory (which doesn't look a sensible approach to me, for just knowing the name of top most element of XML document).

3) The exclude element name list can be externalized from the java application, to make the above program reusable for any kind of XML documents.

4) We may use something like the java JAXP validation APIs, to help achieve the "and" of the two validation steps (i.e, schema validation and the SAX application step) described here, if we want to integrate this approach in a java application.

5) The java code snippet presented above can be made XML namespace aware (i.e if the XML elements are in namespace), by considering the namespace name parameter in SAX callback methods (for e.g the method parameter "String uri", in the startElement callback method).

I hope that this post is useful.

2011-06-26:
The explanation given by me in this blog post originally, seems to convey that multiple global element declarations in XML Schema documents are allowable by the XML Schema language, and this is inherently a bad/wrong design present within the XML Schema language. One of the solution to prohibit certain XML elements to be global in an XML instance document, was presented earlier in this blog post (using an additional restricted SAX parsing step in an application).

All this is fine. But I wanted to follow up on my thoughts written earlier in this post, arguing now, that multiple global element declarations allowable in XML Schema language is not a bad/wrong design present in XML Schema language. One of the features of XML Schema language, which requires multiple global element declarations is XML Schema "substitution groups" (i.e one element substituting for another) -- and "substitution groups" is a core and important concept within XML Schema language.

Of-course, if not working with XML Schema "substitution groups" or otherwise, one could use the SAX add-on technique I presented earlier to prohibit certain global element declarations to validate the XML instance root element, if that suits someones application design.

Tuesday, December 28, 2010

Schema based XML compare

David A. Lee (producer of XMLSH -- A command line shell for XML) raised an interesting discussion a while ago on XML-DEV mailing list, about how to do XML Schema aware XML document comparison. The whole of this discussion thread can be read here. Michael Kay suggested to use the XPath 2.0 function deep-equal (where the input document trees need to be validated by a schema -- to enable type-aware comparison, before doing a comparison by this function) for this kind of use case. Following Michael's idea I was playing with this concept using IBM's XPath 2.0 engine (which is XML Schema aware and is a component of WebSphere Application Server feature pack for XML). For the interest of readers, here's a minimal Java program illustrating this program design.
package com.ibm.xpath2;

import javax.xml.namespace.QName;
import javax.xml.transform.stream.StreamSource;

import com.ibm.xml.xapi.XDynamicContext;
import com.ibm.xml.xapi.XFactory;
import com.ibm.xml.xapi.XPathExecutable;
import com.ibm.xml.xapi.XSequenceCursor;
import com.ibm.xml.xapi.XSequenceType;
import com.ibm.xml.xapi.XStaticContext;

public class XMLCompare {

    public static void main(String[] args) throws Exception {
        String dataDir = System.getProperty("dataDir.path");
  
        XFactory factory = XFactory.newInstance();
        factory.setValidating(XFactory.FULL_VALIDATION);
        factory.registerSchema(new StreamSource(dataDir + "/test.xsd"));
        
        XStaticContext staticContext = factory.newStaticContext();
        staticContext.declareVariable(new QName("doc1"), factory.getSequenceTypeFactory().                      documentNode(XSequenceType.OccurrenceIndicator.ONE));
        staticContext.declareVariable(new QName("doc2"), factory.getSequenceTypeFactory().                                      documentNode(XSequenceType.OccurrenceIndicator.ONE));
        XDynamicContext dynamicContext = factory.newDynamicContext();
        dynamicContext.bind(new QName("doc1"), new StreamSource(dataDir + "/test1.xml"));
        dynamicContext.bind(new QName("doc2"), new StreamSource(dataDir + "/test2.xml"));
                
        XPathExecutable executable = factory.prepareXPath("deep-equal($doc1, $doc2)", staticContext);
        XSequenceCursor result = executable.execute(dynamicContext);
        if (result.exportAsList().get(0).getBooleanValue()) {
           System.out.println("deep-equal == true");
        }
        else {
           System.out.println("deep-equal == false");
        }
    }
} 

Following are the XML and XML Schema documents used for the above example.

test1.xml
<test>10.00</test>
test2.xml
<test>10</test>

test.xsd
<?xml version="1.0" encoding="UTF-8"?>
<schema xmlns="http://www.w3.org/2001/XMLSchema">
  <element name="test" type="double" />
</schema>

For the above examples, if the schema type of element node "test" is xs:double then both the XML documents above are reported deep-equal (since the values 10 and 10.00 are same double values, and the element node was annotated with schema type xs:double and deep-equal function did a type aware comparison of XML documents). But if say the schema type of element node "test" is xs:string, then the XML documents shown above would be reported not deep-equal.

I hope that this post is useful.

Sunday, September 27, 2009

OO multiple inheritance, and Java

I have been thinking again, about multiple inheritance and why Java doesn't support it. I wrote a bit about this topic, some time ago.

There are so, so many resources on web about this, and it's actually very easy to find the answer to this, via a simple web search. Here is an article from where I started to know an answer to this, http://www.javaworld.com/javaqa/2002-07/02-qa-0719-multinheritance.html, which pointed me to this white paper by James Gosling and Henry McGilton. Really, I did not read this white paper by Java creators earlier (it never came across my eyes :)), in spite being familiar and working with Java since long time. Sometimes, we find gems on web in an unexpected ways (I mean, this paper is a gem for me :)). I'll try to read this paper (hopefully fully, and being able to understand it) over the next few days.

And here is a link in this white paper, which explains why Java doesn't support multiple inheritance. The following white paper link is also interesting, which gives a complete overview of C and C++ features, that were omitted in Java language (Java has been influenced from C and C++).

My personal opinion is, that if we must need to use multiple inheritance, we should just try to write programs in C++. On the contrary, my experience in using Java for about a decade, convinces me, that Java is suitable to solve almost any business application problem, and absence of multiple inheritance in Java, is not an hindrance to design good programming abstractions for problem domain. The advantages like Java's byte code portability and web friendliness far outweigh, any disadvantages caused by absence of multiple inheritance. On numerous occasions, I have created Java byte code on Windows, and used it without modification on Unix based systems (and vice versa). This is something which is built into the Java language, and it is cool!

Friday, June 26, 2009

Multiple inheritance in Java

I have always missed true multiple inheritance in Java (like, in C++). For e.g., we are not able to define a class as follows in Java:

class X extends A, B, C {

}

Though, I do not see any inheritance use case which cannot be solved by the current Java facilities, but I would love to have this facility in Java. I think, the most latest Java version (1.7) doesn't have this feature.

One workaround I can see, for multiple inheritance, is to define a class like following:

class X {
A a;
B b;
C c;
}

i.e., we could create private class members inside X (whose functionality we want to use in class X).

Though this might serve purpose for some of the cases, but it's not true multiple inheritance! This I think, is actually aggregation pattern.

Of course, Java has multiple inheritance of interfaces. But that is inheritance of method signatures, and not of implementation.

I guess, keeping the number of base classes to one, Java is much simpler syntactically, and has a simpler compiler implementation. Though I agree, that having a simple syntax (as the current Java inheritance facilities) which is powerful enough, and can solve many use cases is better, than having a complex syntactical facility, which might serve even more uses cases, but could also lead to semantically difficult programs, which may be difficult to maintain and debug, as complexity of the problem domain increases.

Sunday, June 21, 2009

Primitive long a subtype of float

The Java language specification defines, that primitive "long" is a subtype of primitive "float" (ref, http://java.sun.com/docs/books/jls/third_edition/html/typesValues.html#4.10.1).

But XML Schema Datatypes spec, shows no relationship between xs:float and xs:long (
ref: XML Schema 1.0 data types, XML Schema 1.1 data types).

I'm a little confused, that which concept is correct (Java's definition of this data-type inheritance, or XML Schema). I seem to be in favor of XML Schema definition. But perhaps, XML Schema type system is for XML oriented data, and Java type system is for a wider class of applications. But I'm not sure, if this is the reason for the differences of definitions in Java language spec and XML Schema.