Using a pool size of > 1 for inbound file service

Is there any issue with using a pool size setting greater than 1 for an  inbound file service? The reason I ask is we can easily dump 100k+ files to a folder in minutes, but using a pool size of 1 can take hours for that backlog to clear.

  • 0
  • 0
  • 254
  • 2
  • 3

Answers

If FIFO is not required, a pool size >1 allows for processes to run in parallel.

Is that for all inbound adapters, especially a recordmap based one?

 

The reason I ask, is while trying this out for a recordmap service I am seeing the following messages logged:

  • Skipping locked file key.
  • ERROR #5001: No records mapped from Submission

Is it possible to append the files outside of Ensemble and then use batch record map? The current version of Ensemble handles batch record maps much more efficiently than it can handle individual record maps. If you are on an older version, let me know and  I'll find out if the efficiency improvements are in it.

I think that increasing pool  size will give you some improvement, but batch record map can avoid a lot of overhead handling each individual record.

Part of the efficiency improvement depends on what you're doing with the record. If you're just loading data, it's very efficient. If you're performing a transformation on the incoming data then that could still take a significant amount of time.

The big efficiency improvement is in 2016.1. Before that you'll only get a minor improvement going to record map  batch. This is an incentive to upgrade to a new version.