Architecture question: Where do you put your complexity?

Question

Question

Scott Beeson · Feb 2, 2016

I'm designing a workflow for PHR via FTP. What I've done is create a single FTP adapter to pick up files from multiple customers. A router then accesses a lookup table and determines if a particular customer is allowed to send a particular message type to the state. If not, the message doesn't go out. It seems like everyone is against this approach (other than my CEO, thankfully), including Intersystems:

Our recommendation though is to create separate interfaces to retrieve PHR data from individual
participants. This configuration ... gives us the ability to disable an individual data feed without affecting other other participants.

This seems like a terrible practice to me, but before explaining my reasons I wanted to see what everyone else thinks.

Discussion (5)1

Log in or sign up to continue

Luca Ravazzolo · Feb 2, 2016

Hi Scott,

Why do you want to use 1 single interface for multiple providers? Don't answer "why not" :-)

SPOF can be applied to anything. If that interface is critical and it goes down, it's like if your whole production, your all cluster and solution were down. Not good. It's just like in the microservices world. No difference and dangerous. Just like in the security world: it's not IF they'll attack you, it's just a matter of WHEN. Same here. Therefore, I need to ask you a second question: What is your service strategy for availability for the Production and this BS (assuming it's critical here)?

Personally, I would de-couple and leave the responsibility of files accuracy (name, timestamp, upload time-frame vs polling etc. or whatever else) to the customers. Do you have FIFO to respect? Other considerations?

Ensemble is ideal for having a centralized orchestrating BP handling all the incoming requests from the business services. Your BS implementation would be simpler as you implicitly would know what customer you're reading from. Be careful also of the number of files in the reading directory. OSs have limits and issues have been witnessed before (we don't know the numbers here but splitting in various DIRs alleviate the issue mentioned).

Anyway, it might just get down to personal preferences at the end or to one particular context variable we're not aware of here.

just some thoughts... there is no right or wrong in some of these architectures... just point of views, considerations, previous experiences and best practices...

HTH

0 0

Luca Ravazzolo · Feb 5, 2016

Thanks for sharing more info Scott. Sorry I'm late on this. Travelling...

Deployment & management: This should be totally automated. There shouldn't be the need for a GUI (it slow things down). I've given my views on another thread/post you started on this exact subject. There is much to chew on these things and you might be under time pressure, however, it's an unavoidable point (automation) if we all want to be more competitive.

Your last paragraph (human error) highlights why we need to embrace more automation. So to answer the original post question: I'd put my complexity in automation :)

I understand what you're saying BTW; I wish you well with this workflow and the whole project.

All the best

0 0

score 0 · Answer 1 · 2016-02-02T12:15:46-05:00

Luca Ravazzolo · Feb 2, 2016

SPOF? single point of failure? and probably other reasons? de-coupling? Monitoring of single interfaces etc.

0 0

score 0 · Answer 2 · 2016-02-02T13:29:27-05:00

SPOF:
Generally when you talk about a single point of failure you are talking about redundant systems. A network or a power supply. Systems where if one thing fails, the system itself still works. In this scenario we are talking about multiple workflows, or systems, which all have single points of failure. So I feel like it's apples to oranges. Besides that, how often does a single Ensemble adapter/service fail? That seems highly unlikely. More likely that the production would fail which would make this irrelevent anyway.

de-coupling:
This is not inherantly a good or bad thing. Maybe in programming where coupling is easily defined, but in this context I need more details as to why this would be good.

Monitoring:
If you're talking about actual up/down monitoring, refer back to my points on SPOF. Also, I could just as easily monitor the individual FTP servers if I wanted to. On top of that, montioring is as abstract as I want it. I could monitor (and do) all inbound messages from each customer, regardless of what adapter/ip/port/system they come in on, into an aggregate picture that tells me when something is wrong with any feed from anybody.

score 0 · Answer 3 · 2016-02-02T16:47:43-05:00

To address the filename collisions, FIFO and directory limits, all customers will upload to a separate subfolder. THe adapter will then pick them up using the "Subdirectory Levels" parameter set to 1 or 2.

As for the question of why, since you took some time to reply I'll gladly share my reasoning now :)

1. Management

If I have 18 customers and have to create 18 adapters, what happens if the DNS name for my FTP server changes? I have to update all 18 adapters.

2. Deployment

Creating and configuring an adapter takes time. If everything is managed through two lookup tables, I can easily automate the deployment with a CSP page by filling in two fields and clicking a button.

3. Resources

Granted, I don't know how Ensemble works at this level, but my assumption is that 18 adapters monitoring 1 folder would use more resources than 1 adapter monitoring multiple (sub)folders.

4. Clutter

Productions are scary enough. This may seem petty, but it truly bothers me and leads into the next point...

5. Human error

The opposite of SPOF is MPOE (Multiple Points of Error)! What if I double click the wrong service to disable it? What if I have the wrong one selected? This might apply to the lookup table interface as well, but what about individual adapter settings? I've duplicated them 18 times, the chance of making a typo is significant.