Lucene is great, but some of the default settings are heavily biased towards interactive indexing and searching. If you’re building an index in a batch process style, set the IndexWriter.mergeFactor value to something big. I use 10,000, which makes it burn about 500 meg of RAM while indexing, but speeds it up a lot over the default value of 10. YMMV as ever.