There is Russian proverb for such cases "Rumors about my death is slightly exaggerated"^[1] .  Be it BigData, which is declared by Gartner as dead, but is here to stay, in slightly wider form and in more scenarios, or be it MapReduce. Yes, Google marketers claim to not use it anymore, after they have moved to better/more suitable for search interfaces, and yes, in Java world Apache Hadoop is not the best MapReduce implementation nowadays, where Apache Spark is the better/more modern implementation of the same/similar concepts.

But life is more complicated than that shown to us by marketing, there are still some big players, which are still using their own C++ implementation of MapReduce in their search infrastructure - like Russian Yandex search giant. And this is big enough for me to still count it as relevant.

^[1] As Eduard has pointed out that was Mark Twain who originally said "The report of my death was an exaggeration." Thanks for correction, @Eduard!

Good point - the less traffic is there, the better final result.
Although this would be not very much canonical from MapReduce point of view, but the more aggregation could be done on a single node/worker, the better for reducer.

And, as usual, pull requests welcome! https://github.com/tsafin/cache-map-reduce

Very good question. The push operation of our FIFO is safe, even in their "lock-free" way, because of $increment/$sequence usage and their guarantees. But pop operation is troublesome if there will be multiple workers retrieving the same head just at the same moment. 

So, yes, there is no "exactly one" guarantee, and if reduction phase will be running concurrently (it 's not yet planned such) then we have to lock each read-delete operation.

This gonna be problem for multiple node scenarios, so we will talk about this problem when we will approach remote execution and multiple nodes. Thanks for note!

You only partially correct:

• yes, as lazy developers we prefer to write only 1 method call, instead of 2 chained together;
• but, no, this is not %Connect (which may be expensive operation) which should move to the %OnNew, but rather other way around.

I.e. for the cases when we need both (not actually in 100% of a cases, rather 90%) we could create combined classmethod, which will create instance of a class via call to %New() and then will proceed the necessary side-effect. i.e. 

ClassMethod %ConnectNew(Config As %Object) As Sample.RemoteProxy { ... }

In general, you should rather avoid creating huge DOM tree of an objects, or proceeding network operations inside of %New constructor. Constructor needs to allocate just bare minimum of memory, necessary for beginning of operations, and initialize fields to their default values [that will be done automatically]. That's it.

Also about comment to move %Connect code inside of %New.

This is not, generally a good idea to insert potentially long and slow code inside of object constructor. I prefer to have slim and fast %New, which might be nested elsewher to some wrapping onjects. While keeping slow, and expensive functions like %Connect in this case, outside of constructor, independently callable.

For example, try to use incorrect login details here and then see how long it will take to fail such connection (i.e. timeout period).

Because:

1. I hate long list of arguments passed to function, especially when most of them are optional;
2. In similar cases I prefer to use named-arguments approach, whcih I saw otiginally in Perl (here is the quick link I've found which shows this idiom). Named arguments allow to pass arguments in any order, which allows to avoid many related errors if (optional) argument passed in the worng order.
3. Named-arguments were actually creating hash object in Perl, with which we worked later, accessing it's key-value pairs. 
4. But at the end of a day the new, JSON dynamic objects we have in Cache' are semantically equivalent to hash-objects we were operating in Perl in the past;

Thus similar idiom could be used in the ObjectScript.

P.S.

Though I agree, that was some stretching to use this idiom in this particular case, with not that much large number of arguments. But at least it didn't make code less readable. :)

Thanks, Dima, [I did expect you will publish it] and this advice is very interesting and easier to apply by "lazy devops engineer". Though some explanations and comments won't harm. Hope you'll find some time eventually to write article. 

P.S.

Could not resist and not say my few notes about your docker file:

- from pure micro-services point of view for the generic case of multiple ECP clients it makes no much sense IMHO to install csp gateway to each of instantiated docker instances;

- I'd invoke it at the master (ECP database server) instance, or probably as separate docker image;

- [though I suspect, that for HAproxy scenario you might needed to have this CSP-gateway services spread over each instance just for high-availability scenario. I'll be curious that Luca would recommend here from micro-services prospective?]

Let put aside software architecture (I'll write later some number of articles abut what I mean here), let talk about dirty details. 

If you have any oncrete details about the way you use Swarm, Ansible, Chef, or similar, then I (and community) will highly appreciate.

P.S.

It will simplify things a lot if we could configure ECP mapping at the runtime via some set of API calls, and not statucally via editing cache.cpf. Something like it's done in MongoDB for adding new shard:

sh.addShard("repl0/mongodb3.example.net:27327")

https://docs.mongodb.com/manual/reference/method/sh.addShard/

But not for the scenario of adding shard to shard-manager in particular, but for something more generic for ECP or mapping. I suspect there is something related already implemented for EM, but I have no clue how to use it for my case.

P.P.S.

And I know there is already implemented AssignShards call in the forthcoming product, but it's too much specific, creating particular set of mappings. I'd need to have it more generic. 

Could you please share those Terraform configs, or at leats key part of them (concerning Cache.cpf modifications and alike).