We currently have an application nominally written in Java / Spring boot, but all the control flow is in NiFi.
For example, there are the following layers in the java application:
- The controllers take an object and return a service call (literally 2 lines)
- The services pass data to a transformer.
- The transformers pass the data to a converter.
- The converters usually call some API (external) and then convert the return object into a response object that extends to the stack for the caller
On the NiFi side, there are a number of process groups that poll several databases and then feed a large network of other processors (many are custom .nars). The whole flow of control is here (that is, if an object like this has already been processed, do one thing, or do something else) and while it is not very complicated, there is much if (the whole flow of control lives here or in the database).
We want to get away from this development model because it makes it extremely difficult to tell a user when something is really done (unlike what we do now, which is the return of 200 when it comes to NiFi) and the exceptions are very difficult to do . spread.
How can we address this? For scale, we probably have hundreds, if not thousands, of process groups in NiFi.