workflows – Microservices: pipeline flow in which each task is in a different domain

I am creating a workflow composed of several microservices. The domains are divided into different topics. In the example, I can define it as:

  • Event processing domain
  • Domain of configurations
  • Stock inventory domain

Each domain contains its own private database, public APIs, etc. and eventually it runs in Kubernetes.

Responsibilities:

  • Event processing domain: to define the complete workflow of the event life cycle
  • Configuration domain: all events must be compared with a complex configuration mechanism, and decide if this event should be discarded or continue the channeling
  • Domain inventory inventory: for each event, we need to add information from this inventory.

The flow is composed like this:
enter the description of the image here

I am trying to follow several patterns to achieve that:

  • Architecture directed by events, but with commands, through the ESB.
  • The entire workflow is defined in the event processing domain, but the tasks themselves are defined in different domains (since they have access to your database, etc.)

https://www.infoq.com/articles/events-workflow-automation

My main concerns are:

-I know there are some channeling tools to handle those flows (Data Flow / Flink / etc.), but the problem is that all the workflow and the jars must be handled and written in a great tool, which causes 2 problems Main:

  • Responsibilities: each team that is responsible for the domain now has to implement it outside its domain, and without a database and more access to the data, since it is executed in a different context
  • A small monolith is created, all teams must implement and write code in this central

– Those services really manipulate the data that is received, it is not a simple command (do this, to that), so it fits more to a pipe service elsewhere.

I'm trying to choose the best strategy.

  1. A pipe that must be defined / changed easily (add more conditions, change the order, etc.)
  2. Avoid direct communication from service to service and use event-driven architecture
  3. The most challenging problem: the fact that each microservice develops independently

2 tools that I am reviewing –

Any discussion would be appreciated 🙂

Does Hstore and jsonb have the same pipeline internally?

Is hstore implemented identically as jsonb, only with a restriction that the data must be flat? If not, when / why should hstore be used instead of jsonb?

continuous integration – Azure Pipeline Builds does not compile everything

An idea of ​​why Azure Pipeline Builds correctly builds my project but does not create all my executable files, dll in my "drop" artifact?

I have a solution composed of 10 projects.

enter the description of the image here

My project & # 39; Client & # 39; is an old classic WinForm project from VB.NET. This is not visible in my fall artifact:

enter the description of the image here

My project of automatic update is also missing.

It seems that the compilation system does not build my executable projects. What I can do? It is normal? Is it limitation of something or incorrect settings?

How to implement the pipeline with execve in c

As it does the communication between processes using pipe and execve, here is the code of the father.

int main (int argc, char * argv[], char * envp[]) {}
int fd[2]; / * Father will write and Son read through that channel * /
pid_t pid; / * Stores the pid, for the treatment of father and son * /

int status;
int num1 = 24, num2;

yes (tube (fd) < 0) { /* Cria o pipe 1 */
    perror("pipe") ;
    return -1 ;
}

pid = fork(); /* Cria processo filho. */

if(pid == -1) {
    perror("fork") ;
    return -1 ;
}

if(pid > 0) {/ * Parent process * /

// close (fd[0]);
close (fd[1]);

printf ("Parent process sending the number: [%d] for the child  n ", num1);

write (fd)[0], and num1, sizeof (int));
dup2 (0, fd)[0]);
dup2 (1, fd)[1]);

close (fd[0]);

}} else {
printf ("Secondary process executing ...  n");
execve ("child", argv, envp);
}

if (wait (& status)! = pid) / * Wait for the child * /
err_sys ("Wait error");
pr_exit (status); / * and print the status * /

read (fd)[1], And Num2, sizeof (int));

printf ("Parent process received the number: [%d] of the child  n ", num2);
int soma = num1 + num2;

printf ("Parent process .. sum: [%d] N ", sum);

return 0;

}

and here of the son:

int main (int argc, char * argv[], char * envp[]) {}
int num1, num2 = 69;
read (fd)[0], And Num1, sizeof (int));

printf ("Process child read the number: [%d] from father n, num1);

write (fd)[1], and num2, sizeof (int));
printf ("Son process is sending the number: [%d] for the father  n ", num2);

close (fd[0]);
close (fd[1]);

output (0);

}

Xna content pipeline: load / generate new resources when started in MonoGame

I have a question about how to load resources at the beginning of my game written in MonoGame. I have several spritesheets that I am loading at the beginning of the game. This takes about 5 to 10 seconds on my computer, but I have some reports that take up to a minute on other computers … that's too long.
The reason is not to load Texture2Ds but to write new RenderTargets at startup. I have some colorful note sheets where I generate a black and white version when I start. That costs time.

Now to my question: I tried to perform this operation later in the game so that the game starts more quickly and I tried to start the color operation -> white / black when the player creates a new game, that would be a perfect situation to do it. But I always have two different types of errors:

1. Operation not called in the UI thread

or

2. Unable to call Present when a rendering goal is active

As you may have guessed, I do the loading operation on a separate subprocess so that the UI does not freeze. But the real problem seems to be that I can not perform RenderTarget operations when I'm already doing SpriteBatch operations (Begin and End).

How can I fix the problem?

I would be happy for any kind of suggestion.

Yheeky

javascript – Angular pipeline to suppress sensitive information

I have created an angular channel to suppress confidential information such as credit cards, bank accounts, ABA numbers, etc.

This works well, but I would like to know if this is the best possible way to implement the logic.

Here is the code Typescript for the logic of the pipeline.

The SuppressInfoPipe export class implements PipeTransform {

transform (valueToSupress: string, unSuppressedCount?: number): string {


we leave deletedOutput = & # 39; & # 39 ;;

const valueToRemainUnsuppressed =
valueToSupress.substring (valueToSupress.length - unSuppressedCount, valueToSupress.length);

leave astariskLength = valueToSupress.length - aSuppressedCount;

for (be i = 0; i <astariskLength; i ++) {
suppressedOutput = suppressedOutput.concat (& # 39; * & # 39;);
}
suppressedOutput = suppressedOutput.concat (valueToRemainUnsuppressed);

return deletedOutput;
}

}

it takes the input of the string and the number of characters that will not be hidden and then returns the suppressed output.

Comments and suggestions are welcome.

Should I do a FPS game on fixed function Pipeline or OpenGL of programmable pipe?

I have a "FPS" game that I have programmed in Pipleline of fixed function and one made in OpenGL of programmable pipe. While the programmable tubing has many strange things that you can edit, it does not have the load identity that I need for the gun to be connected to the camera. There is little or no information on this subject and most of the information I can find is in the fixed function line instead of the programmable one. Keep in mind that with the fixed channel function, I can only use the glloadidentity function and attach it to then move on to something else. In the programmable, I do not know how to do this, so I spent a whole week looking for how to do it.

Should I use the fixed function pipe one and leave the programmable pipeline?

What should I do?

Thank you!

openGL resource utilization pipeline

How does the openGL representation pipeline work from the point of view of RAM / VRAM, CPU / GPU communication?

P.S.
I tried to search the web, but nothing concrete until now …

computer architecture: in the pipeline (at least, in MIPS), why is the address of the incremented program counter stored in the IF / ID pipe register?

In D. Paterson's book, Organization and computer design, fifth edition, there is a paragraph that says

Obtaining instructions: the upper part of Figure 4.36 shows the
instructions that are read from memory using the address on the PC and
then it is placed in the IF / ID pipeline record. The address of the PC is
increased by 4 and then rewrite on the PC to be ready for the
next clock cycle. This increased address is also saved in the IF / ID
pipe record in case it is needed later for an instruction, such
as beq
. The computer can not know what kind of instruction is being
recovered, so it must prepare for any instruction, potentially happening
necessary information by the pipeline.

I am trying to understand why an increased address should be saved in the IF / ID pipeline record, at least in MIPS. I understand that it might be necessary for some instruction later.

However, how does an instruction like beq use the value of the program counter?

architecture – design of data processing pipeline for data processing

I have a use case for which I need to build a data processing pipeline

  • The contact with the client takes the data that comes from different data sources such as CSV, database, api that should be the first mapped to the fields of a universal scheme. There could be ~ 100k rows every day that should be processed.
  • Then, some of the fields must be cleaned, validated and enriched. For example, the email field must be validated by calling a External API To verify if it is valid and does not bounce, the address field must be standardized to a particular format. There are other operations such as the estimation of the city, the state from the zip code, the validation of the telephone number. At least 20 operations planned, more to come in the future.
  • The above rules are not fixed and can change according to what the user wants to do with their data (saved from the user interface). For example, for a particular data, a user can only choose to standardize their phone number, but not verify if it is valid: therefore, the operations performed on the data are dynamic.

This is what I am currently doing:

  1. Load the data as a panda data frame (they have been considered sparkle.) But the data set is not that big[max 200 mb-]use spark). Have a list of the operations defined by the user that must be performed in each field as

    shares = {"phone number": [‘cleanse’, ‘standardise’], "zipper": [“enrich”, “validate”]}

As I mentioned earlier, the the actions are dynamic and they vary from the data source to the data source according to what the user chooses to do in each field. There are many custom businesses like this that can be applied specifically to a specific field.

  1. I have a custom function for each operation that the user can define for the fields.
    I call them according to the "actions" dictionary and I pass the data frame to the function; the function applies the logic written in the data frame and returns the modified data frame.
def cleanse_phone_no (df, configurations):
# Logic
back modified_df

I'm not sure if this is the right approach to do it. Things will get complicated when you have to call external APIs to enrich certain fields in the future. So I'm considering a producer-consumer model

to. Have a producer module that believes it is divided. each row in the file(1 contact record) as a single message in a queue like AMQ or Kafka

second. Have the logic to process the data in the consumers: they will take one message at a time and process them

do. The advantage I see with this approach is that it simplifies the data processing part, the data is processed one record at a time. There is more control and granularity. The disadvantage is that it will generate an overload in terms of computation as a record processed one by one, which I can overcome to some extent by using multiple consumers

Here are my questions:

  • What is your opinion about the approach? Do you have any suggestions for a better approach?
  • Is there a more elegant pattern that I can use to apply the custom rules to the data set I am currently using?
  • Is it advisable to use a producer-consumer model to process the data one row at a time as a complete data set (taking into account all the logical complexity that would come in the future)? If so, should I use AMQ or Kafka?