testing – Testable architecture of micro-services

Into:

There are mutliple ways to test code: unit tests /e2e / manual testing /..

I’m developing a project that it’s implementation details changes very quickly (and sometimes the core functions as well).

Some of our microservers talk to each other directly while some others communicate using events like kafka.

Problem:

When I create an e2e test (for the backend image only), before each test(s), I build the docker image of my micro service and run it again on (each) test.

I find it really hard to set up this kind of e2e test for a micro service that directly talks to other microservices (sending get/post/.. requests).

As a result, I also build/pull the other images and run them before each tests as well. But it’s not that easy because you can end up implementing a version of docker-compose in you tests infrastructure.

I would like to minimize the amount of errors that can come from other services and test a specific microservice.

Possible solution:

Changing the microservices architecture.

When ever it is possible, a micro service will communicate with others using events. So in the tests, we only need to setup a kafka and the microservice that we try to test.

I only though of this solution from testing perspective and not from “what is best”, for example, it’s faster to communicate without kafka.

Question:

What are the pros and cons of my proposal? From your experience, is it maintainable?

php – What is the common API endpoint architecture in backends?

Hey I always used to create some pretty messy endpoints and I decided that it’s about time to start writing cleaner code, I’ll provide an example of how would I design an endpoint and I’ll be thankful for any insights why is it good or bad practice to do it like this also I’d be more than happy if you provide your own tweaks with explanation why would you do it like that.
Scenario: user post endpoint
Pathname: /api/user
Method: POST
Controller:

class UserPostEndpoint extends AbstractController
{
    private UserPostHandlerInterface $handler;
    private SerializerInterface $serializer;

    function __construct(UserPostHandlerInterface $handler, SerializerInterface $serializer)
    {
        $this->handler = $handler;
        $this->serializer = $serializer;
    }

    public function handleRequest(Request $request): Response
    {
        try {
            $userPostRequestDTO = $this->serializer->unserialize($request->getContent(), UserPostRequestDTO::class);
            $response = $this->handler->handle($userPostRequestDTO); // <- ApiSuccessResponse(statusCode = 201)
        } catch(SerializationException $e) {
            $response = new ApiErrorResponse('Invalid request content provided', 400);
        } catch(ApiRequestContentValidationException $e) {
            $response = new ApiErrorReponse($e->getPlainMessage(), 400);
        } catch(Exception $e) {
            $response = new ApiErrorReponse('Internal server error', 500);
        } finally {
            return $response;
        }
    }
}

Handler would contain some basic things like: invoke createUser in UserFactory, persist the entity with repository.
ApiErrorReponse: message, statusCode
ApiSuccessReponse: UserPostResponseDTO, statusCode(201)
Would you add / change / remove something from the given example to make it more “correct”?
Thanks for the insights

clean code – Preferred architecture for tree stuctured data (Angular/Typescript)

After reading some of “Clean Architecture” I got the idea that perhaps I should reconsider the way I have architected a chart of accounts/accounts object in an accounting program.

I have accounts which live in a tree structure like this:

Budget account (root) 
   Sections (0..n)
     Accounts (0..n)
       Subaccounts (0.n)
          LineItems(0..n)`

Only line items have an actual concrete ‘total’ value, which is entered by a user and stored as a number field. Each level of the hierarchy derives the total from the sum of its children’s total, so the Budget Account total is ultimately equal to the sum of all the line items.

My question is this — Should the ‘total’ property above ‘lineitem’ of each account be implemented as a function equal to something like

get total() :number {
  return (sumOf(children.total())
} 

, or should it be designed as a number field on each account which is updated in response to user lineitem changes by responding to an observable which announces changes in the values of any of accounts children?

In the former case, accuracy and maintainability, simplicity, concurrence are all probably increased if there is only one set of actual data to be maintained at the line item level, but it just seems like a heck of a lot of processing that has to happen each time a page with accounts has to show its total. For example if a page header {{budgetAccount.total}} that in turn would provke a process where every single account is recursed through a loop to calculate the total — on every page referesh! And if you print the entire chart of accounts, the same numbers are added over and over and over each time an account.total is requested. However, with the speed of and capacity of todays machines and browser and I don’t have akeen sense if the burden of all this processing is actually meaningful or not and worth persisting some of these imemdiate totals within the accoutn object, so they don’t need recalculation everytime some uses the getter.

For context, might be 20K-30K lineitems in my actual budget, split over maybe 1,000 accounts at various levels. But of course, an exceptionally large budget may come along with as many as 100,000 or 500,000 line items. In my application, the entire budget structure is kept in memory once retrieved from the back end..

What the factors that should be considered in this decision?

architecture – How to store a branching linked list in a database

I am trying to model an HVAC system for simulation purposes, which will consist of a series of components linked together by airflow and electrical flow paths. I’m trying to figure out how to best represent this in the database. Basically each component has to have one or more inputs and one or more outputs (with both indicating whether they are airflow or electrical), and we need some way of tracking what component they are linked to “next” in the chain.

Now, the simple way I can think of is having a table that is more or less:

Component ID | Connection Type  | Connection Number | Flow Type | Connected Component
      1      |       output     |        1          | Electrical|          2
      1      |       output     |        2          |     Air   |          2 
      1      |       output     |        3          |     Air   |          3
      2      |       input      |        1          | Electrical|          1
      2      |       input      |        2          |     Air   |          1
      3      |       input      |        1          |     Air   |          1

But this feels like its “wrong” in some way. It seems right in the sense that if I wanted to know all of a component’s connections, I could easily get it, but it contains redundant data (I could assume component 2’s inputs by looking for rows where the connected component is 2). It also feels hard to get the whole system from this data, as I’d have to step component by component until I built the whole thing, however, I’m not sure that’s really an issue.

Anyone deal with something like this?

e.g. the above data represents a simple system like:

 1 = 2
   
     3

Where 1 is connected to both 2 and 3, with an air and electrical connection to 2 but only an air connection to 3 (this is largely demonstrative, not an actual use case).

computer architecture – In a single cycle datapath, do ‘decode’ and ‘operand fetch’ occur simultaneously?

After instruction has been fetched, does it go to control unit and register file at the same time or one after the other?
For example if the control unit and register read both have 80ps delay, and we’re calculating the total delay for a cycle, would we take their delay as 80ns(if they occur simultaneously) or if they happen one after the other so in that case delay of decode and operand fetch would be 160ns. Which one is correct?

aws – Serverless game server architecture

First post on the software engineering stack exchange, I hope the question fits the purpose of this sub.

I’m building a serverless game server with the following architecture and was wondering if anyone here has already attempted this and if it’s a valid strategy. Also if you’re aware of any caveats I should know of, I will be grateful for any input here.

I’ve run into a problem here which I’m not completely sure how to address.

The infrastructure

I am using is AWS Lambda and SQS

How it works

The core of the game is intended to work as follows:

A game consists of a number of rounds.
Each round has a duration of 10 seconds, has one question and one correct answer.
Each player must have their answer in before the end of the round.

I.e. a game would look like this:
Game started at 10:00:00
round 1 starts at 10:00:00
round 1 ends at 10:00:10
round 2 starts at 10:00:10
round 2 ends at 10:00:20
etc…

My approach:
When all players have confirmed their participation, the game is created.
This will queue up a delayed SQS message that will trigger a lambda function for each round. Delayed by round number * 10 seconds each.

The code of this lambda function will simply round up all the answers that have been entered for its assigned round, assign a score to each and persist that to a DB and die.

The problem

Delayed SQS messages are cool and all, they do exactly what you’d expect, they only appear in the queue after your specified timeout, however, that is currently not a valid trigger for a lambda function. I’m aware SNS does serve this purpose but AFAIK there is no way to delay an SNS.publish() invocation.

What I’m bending over backwards here to avoid is to have a lambda function waiting for X amount of seconds until it publishes an SNS notification.

Are there any solutions that could help me approach my goal?

Thanks in advance

architecture – Should I deploy redundant SQL database servers in a cloud environment?

As a System Architect I need to deploy a resillient database server in an isolated private cloud.
The cloud is OpenStack with Ceph backend and the Database is SQL.

Is a single database server VM with a local backup sufficient or do I need a pair of database server VMs in active / standby hosted apart so that they don’t share the hardware fault?
The answer must consider the likelihood that a single hardware fault (eg CPU) destroys a running SQL server VM and corrupts both the primary and backup databases preventing automated recovery.

Assumptions:

  1. This is NOT about load balancing nor hardening – the load is light and the network is isolated. It IS about automatic recovery from hardware faults.
  2. 10 minute outage is OK, 1 hour is not.
  3. OpenStack Instance HA will automatically create a replacement SQL Server VM away from the hardware fault and mount the database and continue providing service unless both primary and backup databases are corrupt.
  4. The primary and backup databases are each on dedicated virtual volumes, with resillient Ceph backend, so if one filesystem is corrupted the other may not be.

Why do I care?
a) The active / standby is additional complexity
b) Having an extra VM consumes some of the finite CPU and RAM from the private cloud

Thanks to all who participate for your time and energy.

computer architecture – Cache Blocks Direct Mapping

If the main memory address has 18 bits (7 for tag,7 for line and 4 for word)
and each word is 8 bits.
I found that the main memory capacity is 256-KBytes, total cache lines is 128 line,
total cache words is 128*16(16 word per block/line) = 2048 words.
Then what will be the size of cache words?
I am very confusing on it. I can’t get the definition of cache words.
Can anyone tell me what is the cache words?
Thank you!

Mutli tenant app with onion architecture and multiple presentations

Currently I started designing a new multi tenant SaaS application based on the onion architecture in C#

Using a rest web API to expose my services and consume them on 3 platforms. Two web apps (one for the tenants) and one for the tenant’s clients and a mobile application for the tenants.

I was thinking the 3 apps could use the same API but I’m running into some issues as I also have some deamons that should run on the server and add data (incoming mails, SMS, phonecalls, ..) via the API and the deamons needs a different way of authenticating etc.

Is it possible to use the same application core services in different presentation layers or is it “not done”?

Currently out of guesses on which route to go and can’t seem to find the right answer.