/****************************************************************************
**
** Copyright (C) 2007-2008 Trolltech ASA. All rights reserved.
**
** This file is part of the documentation of the Qt Toolkit.
**
** This file may be used under the terms of the GNU General Public
** License versions 2.0 or 3.0 as published by the Free Software
** Foundation and appearing in the files LICENSE.GPL2 and LICENSE.GPL3
** included in the packaging of this file.  Alternatively you may (at
** your option) use any later version of the GNU General Public
** License if such license has been publicly approved by Trolltech ASA
** (or its successors, if any) and the KDE Free Qt Foundation.
**
** Please review the following information to ensure GNU General
** Public Licensing requirements will be met:
** http://trolltech.com/products/qt/licenses/licensing/opensource/. If
** you are unsure which license is appropriate for your use, please
** review the following information:
** http://trolltech.com/products/qt/licenses/licensing/licensingoverview
** or contact the sales department at sales@trolltech.com.
**
** In addition, as a special exception, Trolltech, as the sole
** copyright holder for Qt Designer, grants users of the Qt/Eclipse
** Integration plug-in the right for the Qt/Eclipse Integration to
** link to functionality provided by Qt Designer and its related
** libraries.
**
** This file is provided "AS IS" with NO WARRANTY OF ANY KIND,
** INCLUDING THE WARRANTIES OF DESIGN, MERCHANTABILITY AND FITNESS FOR
** A PARTICULAR PURPOSE. Trolltech reserves all rights not expressly
** granted herein.
**
** This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
** WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
**
****************************************************************************/

/*!
    \module QtXmlPatterns
    \since 4.4
    \title QtXmlPatterns Module
    \contentspage Qt's Modules
    \previouspage QtXml
    \nextpage Phonon Module
    \ingroup modules
    \ingroup scripting

    \keyword Patternist

    \brief An overview of Qt's XQuery support.

    \tableofcontents

    XQuery is a pragmatic language that allows XML to be queried and
    created in fast, concise, and safe ways.

    The QtXmlPatterns module is part of the \l{Qt Desktop Edition}, and the
    \l{Qt Open Source Edition}.

    \section1 Introduction

    \quotefile snippets/patternist/introductionExample.xq

    The query opens the file \c library.xml, and for each \c book
    element that is a child of the top element \c bib, and whose
    attribute by name \c year is larger than 1991 and has
    Addison-Wesley as a publisher, it constructs a \c book element and
    attaches it to the parent element called \c bibliography.

    \section1 Why use XQuery?

    XQuery is made for selecting and aggregating information in safe
    and efficient ways. Hence, if an application selects and navigates
    data, XQuery could be used to perform the selection and navigation
    tasks quickly and bug-free. With QAbstractXmlNodeModel, these
    advantages are not constrained to operating only on XML files, but
    can be applied to other data as well.

    The strengths of XQuery can be summarized as follows:

    \list

    \o \bold{Pragmatic functionality}: XQuery has many expressions
       for sorting, selecting and filtering data. Typically, these
       can be written in one or two lines. Aggregation functions
       like \c avg(), \c abs() and \c max() help to make XQuery
       code even more concise.

    \o \bold{Development speed}: Pick any of the queries shown in
       this document and imagine writing the equivalent code in C++
       or Java. Compared to the one- or two-liner required in
       XQuery, the equivalent procedural code would require a
       significant investment of time and effort for development,
       testing, and maintenance.

    \o \bold{Type safety}: XQuery values are strongly typed. It
       is illegal, for example, to pass the \c{xs:time} value
           returned from \c{current-time()} to a function expecting an
           \c{xs:integer}. The value of an attribute or element is not
           restricted to being an \c{xs:string} but can be any atomic
           value, e.g., \c{xs:date} or \c{xs:double}.

    \o \bold{Interoperability}: XML specifications and
       vocabularies describe in minute detail what constitutes
       valid XML in different contexts. Adherence to these
       standards is important, because it ensures that XML
       generated by one application can be read and correctly
       interpreted by another. However, knowing and using all
       these standards is a daunting task. The QtXmlPatterns
       XQuery engine performs all the required standards checking
       transparently, leaving the user free to focus on writing
       his query, secure in the knowledge that the output will be
       usable by any other XML tool.

    \o \bold{High-level expression}: Because XQuery is a
       high-level programming language, XQueries can be written to
       be human readable, while also being easily optimizable by
       the XQuery engine at execution time.

    \endlist

    \section1 Using QtXmlPatterns

    Evaluation of an XQuery can be performed programatically, using
    the QtXmlPatterns C++ API, or by running the XQuery engine
    directly using the command line interface.

    \section2 Evaluating queries programatically using the C++ API

    A C++ application that uses classes from the QtXmlPatterns module
    includes the classes at compilation time with the following:

    \snippet doc/src/snippets/code/doc.src.qtxmlpatterns.qdoc 0

    The application is linked with the QtXmlPatterns module by adding
    the following line to the qmake \c{.pro} file:

    \snippet doc/src/snippets/code/doc.src.qtxmlpatterns.qdoc 1

    Note that if you build Qt yourself, QtXmlPatterns will \e not be
    built if exceptions are disabled, or if you compile Qt with a
    compiler that doesn't support member templates, e.g., MSVC 6.

    See the QXmlQuery documentation for the QtXmlPatterns C++ API.

    \section2 Command line utility

    \e xmlpatterns is a command line utility for running XQueries.  It
    takes as its single argument the name of a file containing the
    text of the XQuery to be evaluated.

    \snippet doc/src/snippets/code/doc.src.qtxmlpatterns.qdoc 2

    The XQuery in \c{myQuery.xq} will be evaluated and its output
    written to \c stdout.

    Passing the \c -help switch on the command line tells xmlpatterns
    to print brief descriptions of the other flags it accepts.

    xmlpatterns can be used for scripting, but the descriptions and
    messages it outputs are not designed to be parsed, and they may be
    changed in future releases of Qt.

    \section1 XQuery Language Tutorial

    See \l{A Short Path to XQuery} for a brief introduction to the
    XQuery language.

    \target QtXDM
    \section1 The Qt and XQuery Data Models

    XQuery and Qt don't represent data the same way. XQuery represents
    data as a sequence of items, where an item is either an atomic
    value or a node. Atomic values are the primitives specified in the
    W3C XML Schema. Nodes are normally XML elements or attributes, but
    nodes can also represent non-XML data items, when non-XML data is
    modeled with a custom subclass of QAbstractXmlNodeModel.

    \section2 Mapping XQuery Atomic Value Types to QVariant Types

    When XQuery Atomic Values are returned as XQuery result items via
    the QtXmlPatterns API, they are represented as instances of the
    QVariant class. The mapping from XQuery Atomic Value types to QVariant
    types (or to QXmlName) is as follows.

    \table
    \header
        \o An XQuery Atomic Value of type...
        \o is returned as an instance of...
    \row
        \o \c xs:QName
        \o QXmlName (see \l{Using QXmlNames}{Using QXmlNames} below)
    \row
        \o \c xs:integer
        \o QVariant::LongLong
    \row
        \o \c xs:string
        \o QVariant::String
    \row
        \o \c xs:string*
        \o QVariant::StringList
    \row
        \o \c xs:double
        \o QVariant::Double
    \row
        \o \c xs:float
        \o QVariant::Double
    \row
        \o \c xs:boolean
        \o QVariant::Bool
    \row
        \o \c xs:decimal
        \o QVariant::Double
    \row
        \o \c xs:hexBinary
        \o QVariant::ByteArray
    \row
        \o \c xs:base64Binary
        \o QVariant::ByteArray
    \row
        \o \c xs:gYear
        \o QVariant::DateTime
    \row
        \o \c xs:gYearMonth
        \o QVariant::DateTime
    \row
        \o \c xs:gMonthDay
        \o QVariant::DateTime
    \row
        \o \c xs:gDay
        \o QVariant::DateTime
    \row
        \o \c xs:gMonth
        \o QVariant::DateTime
    \row
        \o \c xs:anyURI
        \o QVariant::Url
    \row
        \o \c xs:untypedAtomic
        \o QVariant::String
    \row
        \o \c xs:ENTITY
        \o QVariant::String
    \row
        \o \c xs:date
        \o QVariant::DateTime
    \row
        \o \c xs:dateTime
        \o QVariant::DateTime
    \row
        \o \c xs:time
        \o (see \l{xstime-not-mapped}{No mapping for xs:time} below)
    \endtable

    \target Using QXmlNames
    \section3 Using QXmlNames

    To access the strings in a QXmlName returned by an
    \l{QXmlQuery::evaluateTo()} {XQuery evaluation}, the QXmlName must
    be accessed with the \l{QXmlNamePool} {name pool} from the
    instance of QXmlQuery that was used for the evaluation.

    \target xstime-not-mapped
    \section3 No mapping for xs:time

    An instance of \c xs:time can't be represented correctly as an
    instance of QVariant::Time, unless the \c xs:time is a UTC time.
    This is because xs:time has a zone offset (0 for UTC) in addition
    to the time value, which the QTime in QVariant::Time does not
    have. This means that if an XQuery tries to return an atomic value
    of type \c xs:time, an invalid QVariant will be returned. A query
    can return an atomic value of type xs:time by either converting it
    to an \c xs:dateTime with an arbitrary date, or to an \c xs:string.

    \section2 Mapping QVariant Types to XQuery Atomic Value Types

    The reverse mapping from QVariant to XQuery Atomic Value is
    important when you want to \l{QXmlQuery::bindVariable()} {bind a
    variable} in your program to a $name used in your XQuery.

    \table
    \header
        \o An instance of...
        \o can be bound to an XQuery variable of type...
    \row
        \o QVariant::LongLong
        \o \c xs:integer
    \row
        \o QVariant::Int
        \o \c xs:integer
    \row
        \o QVariant::UInt
        \o \c xs:nonNegativeInteger
    \row
        \o QVariant::ULongLong
        \o \c xs:unsignedLong
    \row
        \o QVariant::String
        \o \c xs:string
    \row
        \o QVariant::Double
        \o \c xs:double
    \row
        \o QVariant::Bool
        \o \c xs:boolean
    \row
        \o QVariant::Double
        \o \c xs:decimal
    \row
        \o QVariant::ByteArray
        \o \c xs:base64Binary
    \row
        \o QVariant::StringList
        \o \c xs:string*
    \row
        \o QVariant::Url
        \o \c xs:string
    \row
        \o QVariant::Date
        \o \c xs:date.
    \row
        \o QVariant::DateTime
        \o \c xs:dateTime
    \row
        \o QVariant::Time.
        \o \c xs:time. (see \l{Binding To Time}{Binding To QVariant::Time} below)
    \row
        \o QVariantList
    \o (see \l{Binding To QVariantList}{Binding To QVariantList} below)

    \endtable

    Types other than the ones listed in the table are not supported
    and will either cause undefined XQuery behavior or a nonexistent
    variable binding, depending on the context where the variable is
    used.

    \target Binding To Time
    \section3 Binding To QVariant::Time

    Because the instance of QTime used in QVariant::Time does not
    include a zone offset, an instance of QVariant::Time should not be
    bound to an XQuery variable of type \c xs:time, unless the QTime is
    UTC. When binding a non-UTC QTime to an XQuery variable, it should
    first be passed as a string, or converted to a QDateTime with an arbitrary
    date, and then bound to an XQuery variable of type \c xs:dateTime.

    \target Binding To QVariantList
    \section3 Binding To QVariantList

    A QVariantList can be bound to an XQuery variable name. All the
    QVariants in the list must be of the same type, and the variable
    the list is bound to must be of the same type. If the QVariants in
    the list are not all of the same type, the XQuery behavior is
    undefined.

    \section1 Using XQuery with Non-XML Data

    Although the XQuery language was designed for querying XML, with
    QtXmlPatterns one can use XQuery for querying any data that can
    be modeled to look like XML. Non-XML data is modeled to look like
    XML by loading it into a custom subclass of QAbstractXmlNodeModel,
    where it is then presented to the QtXmlPatterns XQuery engine via
    the same API the XQuery engine uses for querying XML.

    When QtXmlPatterns loads and queries XML files and produces XML
    output, it can always load the XML data into its default XML node
    model, where it can be traversed efficiently. The XQuery below
    traverses the product orders found in the XML file \e myOrders.xml
    to find all the skin care product orders and output them ordered
    by shipping date.

    \quotefile snippets/patternist/introAcneRemover.xq

    QtXmlPatterns can be used out of the box to perform this
    query, provided \e myOrders.xml actually contains well-formed XML. It
    can be loaded directly into the default XML node model and
    traversed. But suppose we want QtXmlPatterns to perform queries on
    the hierarchical structure of the local file system. The default
    XML node model in QtXmlPatterns is not suitable for navigating the
    file system, because there is no XML file to load that contains a
    description of it. Such an XML file, if it existed, might look
    something like this:

    \quotefile snippets/patternist/introFileHierarchy.xml

    The \l{File System Example}{File System Example} does exactly this.

    There is no such file to load into the default XML node model, but
    one can write a subclass of QAbstractXmlNodeModel to represent the
    file system. This custom XML node model, once populated with all
    the directory and file descriptors obtained directly from the
    system, presents the complete file system hierarchy to the query
    engine via the same API used by the default XML node model to
    present the contents of an XML file. In other words, once the
    custom XML node model is populated, it presents the file system to
    the query engine as if a description of it had been loaded into
    the default XML node model from an XML file like the one shown
    above.

    Now we can write an XQuery to find all the XML files and parse
    them to find the ones that don't contain well-formed XML.

    \quotefromfile snippets/patternist/introNavigateFS.xq
    \skipto <html>
    \printuntil

    Without QtXmlPatterns, there is no simple way to solve this kind
    of problem. You might do it by writing a C++ program to traverse
    the file system, sniff out all the XML files, and submit each one
    to an XML parser to test that it contains valid XML. The C++ code
    required to write that program will probably be more complex than
    the C++ code required to subclass QAbstractXmlNodeModel, but even
    if the two are comparable, your custom C++ program can be used
    only for that one task, while your custom XML node model can be
    used by any XQuery that must navigate the file system.

    The general approach to using XQuery to perform queries on non-XML
    data has been a three step process. In the first step, the data is
    loaded into a non-XML data model. In the second step, the non-XML
    data model is serialized as XML and output to XML (text) files. In
    the final step, an XML tool loads the XML files into a second, XML
    data model, where the XQueries can be performed.  The development
    cost of implementing this process is often high, and the three
    step system that results is inefficient because the two data
    models must be built and maintained separately.

    With QtXmlPatterns, subclassing QAbstractXmlNodeModel eliminates
    the transformation required to convert the non-XML data model to
    the XML data model, because there is only ever one data model
    required. The non-XML data model presents the non-XML data to the
    query engine via the XML data model API. Also, since the query
    engine uses the API to access the QAbstractXmlNodeModel, the data
    model subclass can construct the elements, attributes and other
    data on demand, responding to the query's specific requests.  This
    can greatly improve efficiency, because it means the entire model
    might not have to be built. For example, in the file system model
    above, it is not necessary to build an instance for a whole
    XML file representing the whole file system. Instead nodes are
    created on demand, which also likely is a small subset of the file
    system.

    Examples of other places where XQuery could be used in
    QtXmlPatterns to query non-XML data:

    \list
        \o The internal representation for word processor documents
        \o The set of dependencies for a software build system
        \o The hierarchy (or graph) that links a set of HTML documents
        from a web crawler
        \o The images and meta-data in an image collection
        \o The set of D-Bus interfaces available in a system
        \o A QObject hierarchy, as seen in the \l{QObject XML Model Example}{QObject XML Model example}.
    \endlist

    See the QAbstractXmlNodeModel documentation for information about
    how to implement custom XML node models.

    \section1 More on using QtXmlPatterns with non-XML Data

    Subclassing QAbstractXmlNodeModel to let the query engine access
    non-XML data by the same API it uses for XML is the feature that
    enables QtXmlPatterns to query non-XML data with XQuery. It allows
    XQuery to be used as a mapping layer between different non-XML
    node models or between a non-XML node model and the built-in XML
    node model. Once the subclass(es) of QAbstractXmlNodeModel have
    been written, XQuery can be used to select a set of elements from
    one node model, transform the selected elements, and then write
    them out, either as XML using QXmlQuery::evaluateTo() and QXmlSerializer,
    or as some other format using a subclass of QAbstractXmlReceiver.

    Consider a word processor application that must import and export
    data in several different formats. Rather than writing a lot of
    C++ code to convert each input format to an intermediate form, and
    more C++ code to convert the intermediate form back to each
    output format, one can implement a solution based on QtXmlPatterns
    that uses simple XQueries to transform each XML or non-XML format
    (e.g. MathFormula.xml below) to the intermediate form (e.g. the
    DocumentRepresentation node model class below), and more simple
    XQueries to transform the intermediate form back to each XML or
    non-XML format.

    \image patternist-wordProcessor.png

    Because CSV files are not XML, a subclass of QAbstractXmlNodeModel
    is used to present the CSV data to the XQuery engine as if it were
    XML. What are not shown are the subclasses of QAbstractXmlReceiver
    that would then send the selected elements into the
    DocumentRepresentation node model, and the subclasses of
    QAbstractXmlNodeModel that would ultimately write the output files
    in each format.

    \section1 Security Considerations

    \section2 Code Injection

    Like SQL, XQuery is vulnerable to
    \l{http://en.wikipedia.org/wiki/Code_injection} {code injection
    attacks}. If an XQuery is constructed by concatenation, where
    concatenated strings can come from user input, the final XQuery
    can become malevolent. The best way to prevent code injection
    attacks is to not construct XQueries from user-written strings, but
    only accept user data input using QVariant and variable bindings. See
    QXmlQuery::bindVariable()

    The articles
    \l{http://www.ibm.com/developerworks/xml/library/x-xpathinjection.html}
    {Avoid the dangers of XPath injection, Robi Sen} and
    \l{http://www.packetstormsecurity.org/papers/bypass/Blind_XPath_Injection_20040518.pdf}
    {Blind XPath Injection, Amit Klein} discuss the XQuery code
    injection problem in more detail.

    \section2 Denial of Service Attacks

    Applications using QtXmlPatterns are subject to the same
    software limits as any other system. Generally, these can not be
    checked. This means QtXmlPatterns does not prevent rogue queries
    from consuming too many resources. For example, a query could take
    too much time to execute, or could attempt to transfer too much
    data. Or a query could cause an unreasonable amount of recursion,
    which could crash the system.  XQueries can do these things
    accidentally, but they can also be meant as deliberate, denial of
    service attacks.

    \section1 Features and Conformance

    \section2 Conformance

    QtXmlPatterns aims at being a
    \l{http://www.w3.org/TR/xquery/#id-xquery-conformance} {conformant
    XQuery processor}. Apart from adhering to
    {http://www.w3.org/TR/xquery/#id-minimal-conformance} {Minimal
    Conformance}, QtXmlPatterns supports the
    \l{http://www.w3.org/TR/xquery/#id-serialization-feature}
    {Serialization Feature} and the
    \l{http://www.w3.org/TR/xquery/#id-full-axis-feature} {Full Axis
    Feature}. QtXmlPatterns passes 97% of the tests in the
    \l{http://www.w3.org/XML/Query/test-suite} {XML Query Test Suite},
    and it is expected this will improve over time. Areas where
    conformance may be questionable and where behavior may be changed
    in future releases are:

    \list

    \o Regular expression support is currently not conformant
           but follows Qt's QRegExp standard syntax.

    \o Operators for \c xs:time, \c xs:date, and \c xs:dateTime
    are incomplete.

    \o Formatting of very large or very small \c xs:double, \c
    xs:float, and \c xs:decimal values may be incorrect.

    \o Some corner cases involving namespaces and element
        constructors are incorrect.

    \endlist

    XML 1.0 and XML Namespaces 1.0 are supported, as opposed to the
    1.1 versions. When a strings is passed to a query as a QString,
    the characters must be XML 1.0 characters. Otherwise, the behavior
    is undefined. This is not checked.

    Since XPath 2.0 is a subset of XQuery 1.0, it is supported.

    The specifications discusses conformance further:
    \l{http://www.w3.org/TR/xquery/}{XQuery 1.0: An XML Query
    Language}.  W3C's XQuery testing effort can be of interest as
    well, \l{http://www.w3.org/XML/Query/test-suite/}{XML Query Test
    Suite}.

    Currently \c fn:collection() does not access any data set, and
    there is no API for providing data through the collection. As a
    result, evaluating \c fn:collection() returns the empty
    sequence. We intend to provide functionality for this in a future
    release of Qt.

    Processing of XML files supports \c xml:id. In practice, this
    allows elements that have an attribute named \c xml:id to be
    looked up efficiently with the \c fn:id() function. See
    \l{http://www.w3.org/TR/xml-id/}{xml:id Version 1.0} for details.

    Only queries encoded in UTF-8 are supported.

    \section2 Resource Loading

    When QtXmlPatterns loads an XML resource, e.g., using \c fn:doc()
    function, the following schemes are supported:

    \table
    \header
        \o Scheme Name
        \o Description
    \row
        \o \c file
        \o Local files.
    \row
        \o \c data
        \o The bytes are encoded in the URI itself. For instance, \c
        data:application/xml,%3Ce%2F%3E is \c <e/>.

    \row
        \o \c ftp
        \o Resources retrieved via FTP.
    \row
        \o \c http
        \o Resources retrieved via HTTP.
    \row
        \o \c https
    \o Resources retrieved via HTTPS. This will succeed if no SSL
    errors are encountered.

    \row
        \o \c qrc
        \o Qt Resource files. Expressing it as an empty scheme, :/...,
        is not supported.

    \endtable

    URIs are first passed to QAbstractUriResolver. Check
    QXmlQuery::setUriResolver() for possible rewrites.

*/

/*!
    \namespace QPatternist
    \brief The QPatternist namespace contains classes and functions required by the QtXmlPatterns module.
    \internal
*/
