XML Document Indexing in Apache Solr

Once your Apache Solr core is up and its schema.xml knows about your fields, the next job is getting documents into the index. XML is one of the most common formats to feed Solr — either in Solr’s own update message format, or as your own arbitrary XML that you transform on the way in. This post covers both routes, with working examples.

There are really two situations you’ll run into, and they use different tools:

flowchart TD
    A["Your data as XML"] --> Q{Is it already in<br/>Solr's add/doc format?}
    Q -->|yes| NATIVE["POST directly to /update"]
    Q -->|"no, it's arbitrary XML"| T{How do you want<br/>to transform it?}
    T -->|"stylesheet on ingest"| XSLT["POST with tr=stylesheet.xsl<br/>(XSLT → add format)"]
    T -->|"pull from files/URLs"| DIH["DataImportHandler<br/>+ XPathEntityProcessor"]
    NATIVE --> IDX["Indexed + committed"]
    XSLT --> IDX
    DIH --> IDX

    classDef data fill:#2563eb,color:#fff,stroke:#1e40af;
    classDef native fill:#059669,color:#fff,stroke:#065f46;
    classDef transform fill:#d97706,color:#fff,stroke:#92400e;
    class A data;
    class NATIVE native;
    class XSLT,DIH transform;
    class IDX native;

# Solr’s native XML update format

Solr understands a simple XML message for adding, deleting, and committing documents. An <add> wraps one or more <doc> elements, and each <field> maps to a field declared in your schema:

<add>
  <doc>
    <field name="id">book-001</field>
    <field name="title">The Left Hand of Darkness</field>
    <field name="author">Ursula K. Le Guin</field>
    <field name="genre">science-fiction</field>
    <field name="price">12.99</field>
  </doc>
  <doc>
    <field name="id">book-002</field>
    <field name="title">A Wizard of Earthsea</field>
    <field name="author">Ursula K. Le Guin</field>
    <field name="genre">fantasy</field>
    <field name="price">9.50</field>
  </doc>
</add>

A multi-valued field is just the same name repeated:

<field name="tag">classic</field>
<field name="tag">award-winner</field>

Deletes and commits use the same channel — by id or by query:

<delete><id>book-001</id></delete>
<delete><query>genre:fantasy</query></delete>
<commit/>

# Posting the documents

The most portable way to send that XML is a plain HTTP POST to the core’s /update handler:

$> curl "http://localhost:8983/solr/first_core/update?commit=true" \
        -H "Content-Type: text/xml" \
        --data-binary @books.xml

The commit=true on the URL tells Solr to make the documents searchable right away — handy for a demo, though in production you’ll usually let auto-commit handle that (more on why in the performance notes below).

Solr also ships a posting tool for convenience. In the 4.x line it’s the jar under example/exampledocs:

$> java -Durl="http://localhost:8983/solr/first_core/update" \
        -jar post.jar books.xml

# Indexing arbitrary XML with an XSLT transform

Real data is rarely already in Solr’s <add><doc> shape. Say your catalog exports look like this instead:

<catalog>
  <book sku="book-001">
    <name>The Left Hand of Darkness</name>
    <by>Ursula K. Le Guin</by>
    <category>science-fiction</category>
    <cost currency="USD">12.99</cost>
  </book>
</catalog>

Rather than pre-processing the file, you can let Solr transform it on ingest. Drop an XSLT stylesheet into conf/xslt/ that rewrites your XML into Solr’s update format:

<!-- conf/xslt/books.xsl -->
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:template match="/">
    <add>
      <xsl:for-each select="catalog/book">
        <doc>
          <field name="id"><xsl:value-of select="@sku"/></field>
          <field name="title"><xsl:value-of select="name"/></field>
          <field name="author"><xsl:value-of select="by"/></field>
          <field name="genre"><xsl:value-of select="category"/></field>
          <field name="price"><xsl:value-of select="cost"/></field>
        </doc>
      </xsl:for-each>
    </add>
  </xsl:template>
</xsl:stylesheet>

Then post your original XML and point Solr at the stylesheet with the tr parameter:

$> curl "http://localhost:8983/solr/first_core/update?tr=books.xsl&commit=true" \
        -H "Content-Type: text/xml" \
        --data-binary @catalog.xml

Solr runs the transform first, and indexes the resulting <add> document. This keeps your source format untouched and puts the mapping logic in one readable place.

# Indexing XML files with the DataImportHandler

When you have many XML files (or need to pull XML from a URL on a schedule), the DataImportHandler (DIH) with XPathEntityProcessor is the better fit. First, enable the handler in solrconfig.xml:

<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
  <lst name="defaults">
    <str name="config">data-config.xml</str>
  </lst>
</requestHandler>

Then describe how to walk the XML and map XPaths to fields in data-config.xml:

<dataConfig>
  <dataSource type="FileDataSource" encoding="UTF-8"/>
  <document>
    <entity name="book"
            processor="XPathEntityProcessor"
            stream="true"
            forEach="/catalog/book"
            url="/data/catalog.xml">
      <field column="id"     xpath="/catalog/book/@sku"/>
      <field column="title"  xpath="/catalog/book/name"/>
      <field column="author" xpath="/catalog/book/by"/>
      <field column="genre"  xpath="/catalog/book/category"/>
      <field column="price"  xpath="/catalog/book/cost"/>
    </entity>
  </document>
</dataConfig>

Kick off the import from the admin Dataimport screen, or with a request:

$> curl "http://localhost:8983/solr/first_core/dataimport?command=full-import"

stream="true" lets DIH process large files without loading them entirely into memory, which matters once your XML runs to hundreds of megabytes.

# A note on commits

Every example above passes commit=true for immediate feedback, which is perfect while you’re developing. For bulk loads, committing after every post is the classic performance mistake — batch your documents and let auto-commit manage visibility instead. That, along with schema, cache, and JVM tuning, is covered in the companion post on Apache Solr performance improvements.

# References

• Solr Reference Guide — Uploading Data with Index Handlers
• Solr Reference Guide — Uploading Structured Data Store Data with the Data Import Handler
• XPathEntityProcessor