Efficient way to convert a large xml file to csv

I think most XSLT implementations read the whole XML input file into memory before beginning to process it.  You need to test if an XSLT approach works with files of the size with which you deal.  Of course, you also need to test how files of a particular size work in the particular machine environment which is running the application.  There is not a one size fits all solution for parsing XML files using XSLT when memory issues are a concern.  I believe ISC supports a SAX parser. SAX parsers put less burdens on memory than XLST.

SAX is described here: https://docs.oracle.com/javase/tutorial/jaxp/intro/simple.html

The intention behind my post was, to give you one idea (of many other possibilities), how to convert XML to CSV. A empty (chars) element is just one of some other unhandled cases (missing tags, other tags inside of COLx tag, etc.).

If you need some speed and your XML-File obeys following constraints:

- the file is a correct XML-File
- contains only the 'Data', 'Details' and 'ColX' tags
- no selfclosing tags, like <sometag/>

then you could try the QAD-way (quick-and-dirty) of conversion.

Again, below an example routine (without excessive testing).
All ISC people and ordinary programer, please look the other way ;-))

Convert() Public
{
   set len=32000 // chunk size, a safe value is:
                 // 32767 - longestColumnValue - tagSizes
   set fi="c:\temp\example-t.xml" // inp file (xml)
   set fo="c:\temp\example-t.csv" // output file (csv)

   open fi:"ru":1 open:$t fo:"nw":1
   if '$t close fi quit

   set xml=$$inp(fi,len) // first xml-chunk

   set i=0, p=0
   while 1 {
      set i=$locate(xml,"\<\/?\w+\>",i,j,v) // next (opening or closing) tag

      if i {                                            // let see, what we got 
         if v="<Details>" { set row="", p=-1 // start a new row
         } elseif v="</Details>" { d out(fo,row) s p=0 // complete, write out
         } elseif p,v["<Col" { s p=j, o=$zstrip(v,"*AP") // new column, keep start
         } elseif p,v["</Col" {s $li(row,o)=$$val($e(xml,p,i-1)) // get value
         }                                   // everything else is don't care
         set i=j

      } else {
         set tmp=$$inp(fi,len) // next xml-chunk
         if tmp="" quit                                   // done

         // remove processed data, add new one
         if p>0 { set xml=$e(xml,p,*)_tmp,p=1,i=0 } else { s xml=$e(xml,i,*)_tmp,i=0 }
      }
   }
   close fi
   close fo
}

val(val)
{
   quit $zstrip(val,"<>w") // add handling of charcter entities like &lt; etc.
}

out(fo,row)
{
   use fo
   write $listtostring(row,";",1),! // delimiter!
}

inp(fn,len)
{
   use fn
   try { read xml#len } catch { s xml="" } // in case, $zeof-mode is off
   quit xml
}

The above converter reads a simple test XML-file with two million 'ColX' items in 5 seconds and creates a CSV file with 100000 rows and 20 columns (in each row).

Instead of file you can also use an stream.

You can use the same approaches that were suggested above, though you'll need to spend time figuring out the structure of the .xslx file so you can write your extraction logic.

To save the time/effort, the Apache POI project provides a Java library that can read/write MS Office formats, including XLSX:
https://poi.apache.org/
https://kalliphant.com/poi-convert-xlsx-to-csv-example/

Check

• Apache POI integration
• LibreOffice integration

Apache POI would be faster and more specialized, LibreOffice is the best bet on converting arbitrary documents between arbitrary formats.