XQuery

• 1. Overview (Current Status of XQuery Support)
• 2. Function Library
• 3. The Module System
  • 3.1. Using Relative URIs
  • 3.2. Builtin Modules
• 4. XQuery Caching
• 5. eXist Extension Functions
  • 5.1. Specifying the Input Document Set
  • 5.2. Querying Text (Fulltext Searching)
  • 5.3. Manipulating Database Contents
  • 5.4. Utility Functions
  • 5.5. XSL Transformations
  • 5.6. HTTP-Related Functions
• 6. Java Binding
• 7. Creating XQuery Modules
• 8. Using Collations
• 9. Serialization Options
• 10. Other Options
• 11. Pragmas
  • 11.1. exist:timer
  • 11.2. exist:batch-transaction

1. Overview (Current Status of XQuery Support)

eXist implements core XQuery syntax (with the exception of XML Schema-related features) as specified in W3C's recommendations outlined in the June 2006 candidate recommendation. Functions in the standard function library should also follow recommendations in the June 2006 version of the "XQuery 1.0 and XPath 2.0 Functions and Operators" recommendation. However, the function library is not yet completely aligned with the latest version of the spec. Some function signatures may differ, other functions could be missing. Please check against the local function library documentation.

The XQuery implementation is tested against the official XQuery Test Suite (XQTS). This suite contains more than 10,000 tests. As of June 2006, eXist passes > 80% of the test suite, but we are continuously trying to improve these results. Current test results are reported on the web site. A good part of the remaining failures is caused by smaller problems with value comparisons, formatting of floating point numbers and the missing schema support in eXist.

In addition to the standard features, eXist provides extended support for modules and implements the full axis feature, which means you can use the optional axes: ancestor, ancestor-or-self, following, following-sibling, preceding, and preceding-sibling.

2. Function Library

A complete list of eXist-supported XQuery functions can be viewed in the Built-in Functions section. Note that this list is dynamically generated using an XQuery script, so the database needs to be running to view this page. Each function description is taken directly from the signature provided by the class implementing the Function interface.

3. The Module System

Using eXist, you can write entire web applications in XQuery. This may result in rather complex XQuery scripts, consisting of several thousand lines of code. Being able to package related functions into modules is thus an important feature. eXist allows modules to be imported from a variety of sources:

• an URI

• a collection in the database

• a jar file, i.e. a Java archive

• a Java class, if the module is itself implemented in Java

For example, a typical import statement in an XQuery will look like this:

Provided that the module namespace does not point to one of the preloaded standard modules (see below), the query engine will try to locate the module source by looking at the URI given after the at keyword. In the example above, the module was specified using a full URI and the query engine will attempt to load the module source from the given URI. However, the module could also be stored in a database collection:

The query engine recognizes that the module should be stored in the local database instance and tries to directly compile it from there.

If the XQuery module is part of a Java application, it might also be an option, to pack the module into a Java archive (.jar file) along with the Java classes and use the following import to load the module from a Java package:

Finally, XQuery modules can also be implemented in Java (see below), in which case you can import them by specifying the class path of the Module class:

3.1. Using Relative URIs

If the location specified in an import statement is a relative URI, the query engine will try to load the module relativ to the current module load path. The module load path is determined as follows:

1. if the main XQuery was retrieved from the file system, the module load path points to that directory. This applies to queries executed through the XQueryServlet, XQueryGenerator or the Java admin client.

2. if the main XQuery was loaded from a database collection, the module load path is the URI of that collection.

   For example, if you access an XQuery via the REST server:

   http://localhost:8080/exist/servlet/db/modules/test.xq

   All relative module paths will be resolved relative to the /db/modules collection.

3.2. Builtin Modules

eXist comes with a set of utility modules, which are all implemented in Java. You can also write your own modules as described below. Some of these utility modules are frequently used in queries, so they are automatically imported into every query by default.

The query engine allows you to configure which modules will be auto-loaded. The <builtin-modules> element in conf.xml lists the namespaces and implementing classes of all modules to be preloaded into queries:

<xquery enable-java-binding="no">
    <builtin-modules>
        <module uri="http://exist-db.org/xquery/util"
            class="org.exist.xquery.functions.util.UtilModule"/>
        <module uri="http://exist-db.org/xquery/transform"
            class="org.exist.xquery.functions.transform.TransformModule"/>
                    

You never need to specify a location when importing a preloaded module. The namespace of the module is already known and eXist knows how to load it. Also, auto-loaded modules don't need to be explicitely imported into the main XQuery, though you still need to import them if you want to use them from within another XQuery module.

4. XQuery Caching

XQuery modules executed via the REST interface, the XQueryServlet or XQueryGenerator are automatically cached: the compiled expression will be added to an internal pool of prepared queries. The next time a query or module is loaded from the same location, it will not be compiled again. Instead, the already compiled code is reused. The code will only be recompiled if eXist decides that the source was modified or it wasn't used for a longer period of time.

Modules are cached along with the main query that imported them.

5. eXist Extension Functions

eXist offers a number of additional functions and operators, which are discussed in some detail in the following subsections.

5.1. Specifying the Input Document Set

A database can contain a virtually unlimited set of collections and documents. By default, database queries that use the XML:DB API will only process the documents in the current XML:DB collection. However, four additional functions are available to change this behavior: doc(), document(), collection() and xcollection(). The collection() and doc() functions are standard XQuery/XPath functions; whereas, xcollection() and document() are eXist-specific extensions.

Without an URI scheme, eXist interprets the arguments to collection and doc as absolute or relative paths, leading to some collection or document within the database. For example:

refers to a resource stored in /db/collection1/collection2.

references a resource relative to the base URI property defined in the static XQuery context. The base URI contains an XML:DB URI pointing to the base collection for the current query context, e.g. xmldb:exist:///db.

The base collection depends on how the query context was initialized. If you call a query via the XML:DB API, the base collection is the collection from which the query service was obtained. All relative URLs will be resolved relative to that collection. If a stored query is executed via REST, the base collection is the collection in which the XQuery source resides. In most other cases, the base collection will point to the database root /db.

You can also pass a full URI to the doc function:

in this case, the URI will be retrieved and the data stored into a temporary document in the database.

5.2. Querying Text (Fulltext Searching)

The standard XPath/XQuery function library contains most of the common string manipulation functions provided by most programming languages. However, these functions are insufficient for conducting keyword or phrase searches inside a larger portion of text or mixed content. This is a weak point if you have to work with document-centric (i.e. mainly free-form text), as opposed to data-centric documents. For many types of documents, the standard string functions do not yield satisfactory search results.

For example, suppose upon reading a chapter in an electronic text, you encountered something about "XML" and "databases", but later you could not recall the exact section where you read it. Using standard XPath, you could try a query like:

This query execution will likely be quite slow, since the XPath engine will, in this case, scan the entire character content of all chapter nodes and their descendants. And yet, there is no certainty that all possible text matches will be found - for example, "databases" might have been written with a capital letter at the start of the sentence, and so would not be included in the results.

To resolve this issue, eXist offers additional operators and extension functions for efficient, index-based access to the full-text content of nodes. With eXist, you could alternatively formulate the above query as follows:

This will return all chapters containing both keywords in the correct order, and as well, will find matches that have under 50 words between them. Additionally, the wildcard character ? in database? will match the singular as well as the plural instances of "database", and the search would NOT be case-sensitive. Furthermore, since the query is index-based, it will usually be an order of magnitude faster than the standard XPath query above.

Operators

In this section, we discuss each of eXist's text-search extensions. In cases where the order and distance of search terms is not important, eXist offers two additional operators for simple keyword queries: &= and |=.

Both of the above operators accept simple wildcards in the keyword string. A ? matches zero or one character, * matches zero or more characters. A character range [abc] (as a regular expression) matches any of the characters in that range. You may use a backslash to escape wildcard characters.

To match more complex patterns, full regular expression syntax is supported through additional functions, which are discussed below.

Functions

5.3. Manipulating Database Contents

The XML:DB extension functions can be used to create new database collections, or to store query output into the database. To illustrate, suppose we have a large file containing several RDF metadata records, but we do not want to store the metadata records in a single file, since our application expects each record to have its own document. In this case, we must divide the document into smaller units. Using an XSLT stylesheet would be one way to accomplish this - however, it is also a memory-intensive approach. A preferable option is to use XQuery to do the job.

The XQuery script below shows how to split a large RDF file into a series of smaller documents:

xquery version "1.0";

declare namespace  xdb="http://exist-db.org/xquery/xmldb";
declare namespace util="http://exist-db.org/xquery/util";

xdb:register-database("org.exist.xmldb.DatabaseImpl", true()),
let $root := xdb:collection("xmldb:exist:///db", "admin", ""),
    $out := xdb:create-collection($root, "output")
for $rec in /rdf:RDF/* return
    xdb:store($out, concat(util:md5($rec/@rdf:about), ".xml"),
        <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
            {$rec}
        </rdf:RDF>
    )

                    

Let's look at this script in some detail. First, since the extension functions are based on the XML:DB API, we have to register a database driver with xdb:register-database. Then we can retrieve the root collection using xdb:collection. The variable $root returned by xdb:collection holds the collection as a Java object, and we can pass this variable to xdb:create-collection to create a new sub-collection, called "output".

Next, the for-loop iterates through all child elements of the top RDF element. In each iteration, we use xdb:store to write out the current child node to a new document. Since a unique document name is required for each new document, we need a way to generate unique names. In this case, the URI contained in the rdf:about attribute is unique, so we simply compute an MD5 key from it, append the ".xml" extension, and use it as the document's name.

5.4. Utility Functions

The function namespace http://exist-db.org/xquery/util contains a number of common utility functions. The util:md5 function, used in the above example, is one such function. In this subsection, we discuss a few of the important available utility functions. For a complete list, please go to the Built-in Functions section.

However, one important function, util:eval, requires further explanation.

                                    xquery version "1.0";

declare namespace request="http://exist-db.org/xquery/request";
declare namespace util="http://exist-db.org/xquery/util";

declare function local:do-query() as element()
{
    let $n1 := request:get-parameter("n1", ""),
		$n2 := request:get-parameter("n2", ""),
		$op := request:get-parameter("op", "")
	return
		if($n1 = "" or $n2 = "") then
			<p>Please enter two operands.</p>
		else
			let $query := concat($n1, " ", $op, " ", $n2)
			return
				<p>{$query} = {util:eval($query)}</p>
};

<html>
	<body>
		<h1>Enter two numbers</h1>

		<form action="{request:get-uri()}" method="get">
			<table border="0" cellpadding="5">
			<tr>
				<td>First number:</td>
				<td><input name="n1" size="4"/></td>
			</tr>
			<tr>
				<td>Operator:</td>
				<td>
					<select name="op">
						<option name="+">+</option>
						<option name="-">-</option>
					</select>
				</td>
			</tr>
			<tr>
				<td>Second number:</td>
				<td><input name="n2" size="4"/></td>
			</tr>
			<tr>
				<td colspan="2"><input type="submit"/></td>
			</tr>
			</table>
		</form>

		{ local:do-query() }
	</body>
</html>
                                

5.5. XSL Transformations

eXist has a function for directly applying an XSL stylesheet to an XML fragment within the XQuery script. This can be very convenient, for example, if your application requirements are basic, and you do not want to install Cocoon to run the XSLT.

5.6. HTTP-Related Functions

eXist offers functions for handling HTTP request parameters and session variables that use the http://exist-db.org/xquery/request namespace. Functions in this namespace are only usable if the query is executed through the XQueryGenerator or the XQueryServlet (for more information consult eXist's Developer's Guide ).

6. Java Binding

eXist supports calls to arbitrary Java methods from within XQuery. The binding mechanism follows the short-cut technique introduced by Saxon (a collection of tools for processing XML documents). The class where the external function will be found is identified by the namespace URI of the function call. The namespace URI should start with the prefix java: followed by the fully qualified class name of the class. For example, the following code snippet calls the static method sqrt (square-root function) of class java.lang.Math:

declare namespace math="java:java.lang.Math";
math:sqrt(2)
				

Note that if the function name contains a hyphen, the letter following the hyphen is converted to upper-case and the hyphen is removed (i.e. it applies the CamelCase naming convention), and so, to-string() will call the Java method toString().

If more than one method in the class matches the given name and parameter count, eXist tries to select the method that best fits the passed parameter types at runtime. The result of the method call can be assigned to an XQuery variable. If possible, it will be mapped to the corresponding XML schema type. Otherwise, it's type is the built-in type object.

Java constructors are called using the function new. Again, a matching constructor is selected by looking at the parameter count and types. The returned value is a new Java object with the built-in type object.

Instance methods are called by supplying a valid Java object as first parameter. The Java object has to be an instance of the given class. For example, the following snippet lists all files and directories in the current directory:

declare namespace file="java:java.io.File";

<files>
    {
        for $f in file:list-files( file:new(".") )
        let $n := file:get-name($f)
        order by $n
        return
            if (file:is-directory($f)) then
                <directory name="{ $n }"/>
            else
                <file name="{ $n }" size="{ file:length($f) }"/>
    }
</files>       			
        		

7. Creating XQuery Modules

eXist supports XQuery-based library modules. These modules are simply collections of function definitions and global variable declarations, of which eXist knows two types: External Modules, which are themselves written in XQuery, and Internal Modules, which are implemented in Java. The standard XPath/XQuery functions and all extension functions described in the above sections are defined through internal modules.

You can declare an XQuery file as a module and import it using the import module directive. The XQuery engine imports each module only once during compilation. The compiled module is then made available through the static XQuery context.

Users can also provide additional Java modules. With the current release, it is relatively simple to add these modules, although the current API may change in the future. To register a module, eXist requires a namespace URI by which the module is identified, and the list of functions it supplies. For this, you need only to pass a driver class to the XQuery engine, and this class should implement the interface org.exist.xpath.InternalModule.

Moreover, the class org.exist.xpath.AbstractInternalModule already provides an implementation skeleton. The class constructor expects an array of function definitions for all functions that should be registered. A function definition (class FunctionDef) has two properties: the static signature of the function (as an instance of FunctionSignature), and the Java Class that implements the function.

A function is a class extending org.exist.xpath.Function or org.exist.xpath.BasicFunction. Functions without special requirements (e.g. overloading) should subclass BasicFunction. To illustrate, the following is a simple function definition:

public class EchoFunction extends BasicFunction {

public final static FunctionSignature signature =
new FunctionSignature(
	new QName("echo", ExampleModule.NAMESPACE_URI, ExampleModule.PREFIX),
	"A useless example function. It just echoes the input parameters.",
    new SequenceType[] {
        new SequenceType(Type.STRING, Cardinality.ZERO_OR_MORE)
    },
	new SequenceType(Type.STRING, Cardinality.ZERO_OR_MORE));

public EchoFunction(XQueryContext context) {
	super(context, signature);
}

public Sequence eval(Sequence[] args, Sequence contextSequence)
throws XPathException {
	// is argument the empty sequence?
	if (args[0].getLength() == 0)
		return Sequence.EMPTY_SEQUENCE;
	// iterate through the argument sequence and echo each item
	ValueSequence result = new ValueSequence();
	for (SequenceIterator i = args[0].iterate(); i.hasNext();) {
		String str = i.nextItem().getStringValue();
		result.add(new StringValue("echo: " + str));
	}
	return result;
}

}

In looking at this sample, first note that every function class has to provide a function signature. The function signature defines the QName by which the function is identified, a documentation string, the sequence types of all arguments, and the sequence type of the returned value. In the example above, we accept a single argument of type xs:string and a cardinality of ZERO_OR_MORE. In other words, we accept any sequence of strings containing zero or more items.

Next, the subclass overwrites the eval method, which has two arguments: the first contains the values of all arguments passed to the function, the second passes the current context sequence (which might be null). Note that the argument values in the array args have already been checked to match the sequence types defined in the function signature. We therefore do not have to recheck the length of the array: if more or less than one argument were passed to the function, an exception would have been thrown before eval gets called.

In XQuery, all values are passed as sequences. A sequence consists of one or more items, and every item is either an atomic value or a node. Furthermore, a single item is also a sequence. The function signature specifies that any sequence containing zero or more strings is acceptable for our method. We therefore have to check if the empty sequence has been passed. In this case, the function call returns immediately. Otherwise, we iterate through each item in the sequence, prepend echo:" to its string value, and add it to the result sequence.

In the next step, we want to add the function to a new module, and therefore provide a driver class. The driver class defines a namespace URI and a default prefix for the module. Functions are registered by passing an array of FunctionDef to the constructor. The following is an example driver class definition:

public class ExampleModule extends AbstractInternalModule {

public final static String NAMESPACE_URI = 
    "http://exist-db.org/xquery/examples";
	
public final static String PREFIX = "example";
	
private final static FunctionDef[] functions = {
	new FunctionDef(EchoFunction.signature, EchoFunction.class)
};
	
public ExampleModule() {
	super(functions);
}

public String getNamespaceURI() {
	return NAMESPACE_URI;
}

public String getDefaultPrefix() {
	return PREFIX;
}

}

Finally, we are able to use this newly created module in an XQuery script:

xquery version "1.0";

import module namespace "http://exist-db.org/xquery/examples"
at "java:org.exist.examples.xquery.ExampleModule";

example:echo(("Hello", "World!"))

The query engine recognizes the java: prefix in the location URI, and treats the remaining part (in this case, org.exist.examples.xquery.ExampleModule) as a fully qualified class name leading to the driver class of the module.

8. Using Collations

Collations are used to compare strings in a locale-sensitive fashion. XQuery allows one to specify collations at several places by means of a collation URI. For example, a collation can be specified in the order by clause of a XQuery FLWOR expression, as well as any string-related functions. However, the concrete form of the URI is defined by the eXist implementation. Specifically, eXist recognizes the following URIs:

1. http://www.w3.org/2004/07/xpath-functions/collation/codepoint

   This URI selects the unicode codepoint collation. This is the default if no collation is specified. Basically, it means that only the standard Java implementations of the comparison and string search functions are used.

2. http://exist-db.org/collation?lang=xxx&strength=xxx&decomposition=xxx

   or, in a simpler form:

   ?lang=xxx&strength=xxx&decomposition=xxx

   The lang parameter selects a locale, and should have the same form as in xml:lang. For example, we may specify "de" or "de-DE" to select a german locale.

   The strength parameter (optional) value should be one of "primary", "secondary", "tertiary" or "identical".

   The decomposition parameter (optional) has the value of "none", "full" or "standard".

The following example selects a german locale for sorting:

And returns the following:

You can also change the default collation:

Which returns true. Note that if you use the default codepoint collation instead, the comparison would evaluate to false.

9. Serialization Options

The serialization of query results into a binary stream is influenced by a number of parameters. These parameters can be set within the query itself, however the interpretation of the parameters depends on the context in which the query is called. Most output parameters are applicable only if the query is executed using the XQueryGenerator or XQueryServlet servlets, or the REST server.

Serialization parameters can either be set via pragmas (see the following section) or a declare option statement in the query prolog. In declare option, the serialization parameters can be specified as follows:

Here, single options are specified within the string literal, separated by a whitespace. Note also that the option QName must be exist:serialize, where the exist prefix is bound to the namespace http://exist.sourceforge.net/NS/exist, which is declared by default and need not be specified explicitly.

Note that these same options can be passed using the XPathQueryService.setProperty() and XQueryService.setProperty() methods in Java. These methods are defined in javax.xml.transform.OutputKeys and EXistOutputKeys. The latter eXist-specific options include the following:

• expand-xincludes=yes|no

• highlight-matches=elements|attributes|both|none

• process-xsl-pi=yes|no

The general options include the following:

• method=xml|xhtml|text

• media-type

• encoding

• doctype-public

• doctype-system

• indent=yes|no

• omit-xml-declaration=yes|no

For example, to disable XInclude expansion, and indent the output, you can use the following pragma:

For the output method parameter, eXist currently recognizes three methods: xml, xhtml and text. Note that unlike the xml method, the xhtml setting uses only the short form for elements that are declared empty in the xhtml DTD. For example, the br tag is always returned as <br/>. On the other hand, the text method only returns the contents of elements - for instance, <A>Content</A> is returned as Content. However, attribute values, processing instructions, comments, etc. are all ignored.

10. Other Options

To avoid that the server is blocked by a badly formulated query, eXist watches all query threads. A blocking query can be killed if it takes longer than a specified amount of time or consumes too many memory resources on the server. There are two options to control this behaviour:

Specifies the maximum amount of query processing time (in ms) before it is cancelled by the XQuery engine.

Defines a limit for the max. size of a document fragment created within an XQuery. The limit is just an estimation, specified in terms of the accumulated number of nodes contained in all generated fragments. This can be used to prevent users from consuming too much memory if they are allowed to pass in their own XQueries.

11. Pragmas

XQuery pragmas are a way to pass implementation-specific information to the query engine from within a XQuery. The syntax for pragmas has changed between the different drafts of the XQuery specification. In earlier eXist releases, pragmas were used similar to what is now the "declare option" prolog expression. The new syntax is quite different: pragmas can now be wrapped around an arbitrary XQuery expression (see the specificiation).

Currently, eXist recognizes only two pragmas: exist:timer and exist:batch-transaction.

11.1. exist:timer

Provides a simple way to measure the time for executing a given expression. For example:

creates a timer for the expression enclosed in curly braces and prints timing information to the trace logger. Please note that trace needs to be enabled in log4j.xml:

	                    
<root>
	    <priority value="trace"/>
	    <appender-ref ref="console"/>
	</root>
	                

11.2. exist:batch-transaction

Currently only for XQuery Update Extensions. Provides a method for batching updates on the database into a single Transaction, allowing a set of updates to be atomically guaranteed. Also for each affected document or collection, any configured Triggers will only be called once, the prepare() method will be fired before the first update to the configured resource and the finish() method fired after the last update to the configured resource.

Uses a single Transaction and Trigger events for the expressions enclosed in curly braces.

We will certainly add more pragma expressions in the near future. Among other things, pragmas are a good way to pass optimization hints to the query engine.