Question
· Sep 9, 2016

Using a pool size of > 1 for inbound file service

Is there any issue with using a pool size setting greater than 1 for an  inbound file service? The reason I ask is we can easily dump 100k+ files to a folder in minutes, but using a pool size of 1 can take hours for that backlog to clear.

Discussion (5)1
Log in or sign up to continue

Is it possible to append the files outside of Ensemble and then use batch record map? The current version of Ensemble handles batch record maps much more efficiently than it can handle individual record maps. If you are on an older version, let me know and  I'll find out if the efficiency improvements are in it.

I think that increasing pool  size will give you some improvement, but batch record map can avoid a lot of overhead handling each individual record.

Part of the efficiency improvement depends on what you're doing with the record. If you're just loading data, it's very efficient. If you're performing a transformation on the incoming data then that could still take a significant amount of time.