multithreading – What is the best way to decouple and define the orchestration (coordination) of concurrent tasks?

I have a data synchronisation concurrent algorithm. It does the following: get data and files from server, send data and files to server, save them to database / filesystem. Imagine the system like this:

  1. You have 1000 functions. Each one does some atomic operation. For instance, fetch latest objects of type X and insert them into DB; upload this file of type Y and so on. Each function is independent and can act on its own, it does not communicate with or affect other functions. On the other hand, none of them is a pure function, because they all use theese common resources (fetching data from the server, puting data on DB, saving files on filesystem)
  2. You have a single entry point for the sychronization mechanism. The outside of the sync system can start the sync, say, by doing a Sync.start() call. Also, the sync has a single exit point. The sync can finish with either success, either failure (if any of those functions from (1) fail, the whole sync will fail). The ouside of the sync system can subscribe to onSyncSuccess / onSyncError events.
  3. You have this black box in the middle of the system. This could be, for instance, a single threaded algorithm calling those 1000 functions from (1). But I made it concurrent.

Now consider this. This concurrent algorithm right now is rigid because the way in which the functions are called is hardcoded. If I want to take a bunch of functions from (1) that right now are executing sequentially, and if I want to make them execute parallel, it would be impossible without refactoring the whole class hierarchy.

I was thinking about the concept of direct acyclic graphs, and I made my own domain-specific language in Kotlin to define such task graphs. Now I could write the whole orchestration declaratively like this:

notifySyncWasStarted()
runSequentialy {
    task { doTask1() }
    runInParallel {
        task { doTask2() }
        task { doTask3() }
    }
    task { doTask4() }
}
notifySyncWasStopped()

So first task1 gets executed, then task2 and 3 in the same time, then task4. By keeping this graph in a single file, I could easily modify the way tasks are executed. For instance, I could easily swap tasks:

notifySyncWasStarted()
runSequentialy {
    runInParallel {
        task { doTask4() }
        task { doTask2() }
    }
    task { doTask3() }
    task { doTask1() }
}
notifySyncWasStopped()

Here, (task 4 and 2) gets executed, then 3, then 1. This works by using the fork-join paradigm, I create threads then join them into the parent thread.

In contrast, right now, the algorithm is spread around multiple classes, each of them was designed to run the tasks in a specific manner. Changing how tasks are ran would mean to refactor the classes and how they communicate to each other.

The question is: What is the best way to decouple and define the orchestration (coordination) of concurrent tasks? So that this orchestration could be easily changed in the future? Is my solution optimal or the way to go (direct acyclic graphs, fork-join, plus a domain specific language)? Or maybe there are some other design patterns that do the same thing?

What tool/methodologie is used in security orchestration of OWASP appsec pipeline?

I’m reading materials about appsec pipeline. It seems interesting. OWASP has a project about it here https://owasp.org/www-project-appsec-pipeline/
I was reading a slide share about it but the author is not explicit on the tools/system/methodology that he uses for what he calls “Security Orchestration” (https://www.slideshare.net/weaveraaaron/building-an-appsec-pipeline-keeping-your-program-and-your-life-sane)

What does he even mean by that ? Is it Kubernetes type of orchestrations ? Or what is it ?

The only thing I found about this talks and tools are here https://www.appsecpipeline.org/index.html but the site seems to have stopped at the “Intake” stage of appsec testing

domain name system – backend orchestration for nginx

I am trying to find a way to orchestrate (dynamically add and remove) backends through DNS service discovery for an Nginx reverse proxy.

Nginx configuration (based on an example):

resolver 10.0.0.2 valid=10s;

server {
    location / {
        set $backend_servers backends.example.com;
        proxy_pass http://$backend_servers:8080;
    }
}

The backend will consist of multiple virtual machines (multiple regions) from cloud providers like DigitalOcean, Linode, Vultr, etc.

How can I easily add and remove VM IP to backends.example.com, so that they can be queried by a public DNS solver (like Google or Cloudflare) or a private solver (for example, 10.0.0.2)?

Do I use DNS Providers API to add / remove IPs, or maybe I host some code triggered by DNS lookups to respond with IPs from VMs obtained through Cloud Providers API?

How was the industry dealing with containers and container orchestration before Docker and Kubernetes?

The first analogous approach was probably BSD Jails, or in Linux and some other environments, a chroot environment (also sometimes called chroot jail).

Chroot basically changes which directories are visible to an application, essentially virtualizing only the file system. When used to isolate applications for some kind of security purpose, it is generally important that the running process does not run as root, since root can easily undermine chroot isolation. This is (possibly) somewhat less important in the docker, although I think there have been security issues related to that in the docker. In chroot, it is sometimes a function, since it usually configures the chroot file system as root and executes the process as something else when it is ready.

The BSD jails are closer to the containers of the docker, since they have their own user accounts in addition to an isolated file system, as well as a virtualized network environment. Unlike chroot, a caged process does not have the ability to see processes that run outside the jail. The main thing that is missing is probably the ease of use to extract an image from somewhere (instead of just using tar) and probably the approach of the command line interface to expose ports, map shared volumes, etc.