microservices – Service integration with large amounts of data

I am trying to assess the viability of microservices/DDD for an application I am writing, for which a particular context/service needs to respond to an action completing in another context. Whilst previously I would handle this via integration events published to a message queue, I haven’t had to deal with events which could contain large amounts of data

As a generic example. Let’s say we have an Orders and Invoicing context. When an order is placed, an invoice needs to be generated and sent out.

With those bits of information I would raise an OrderPlaced event with the order information in, for example:

public class OrderPlacedEvent
{
    public Guid Id { get; }
    public List<OrderItem> Items { get; }
    public DateTime PlacedOn { get; }
}

from the Orders context, and the Invoicing context would consume this event to generate the required invoice. This seems fairly standard but all examples found are fairly small and don’t seem to address what would happen if the order has 1000+ items in the order, and it leads me to believe that maybe integration events are only intended for small pieces of information

The ‘easiest’ way would be to just use an order ID and query the orders service to get the rest of the information, but this would add coupling between the two services which the approach is trying to remove.

Is my assumption that event data should be minimal correct? if it is, how would I (or even, is it possible to?) handle such a scenario where there are large pieces of data which another context/service needs to respond to, correctly?

How to write algorithm of distribution amounts

I have an array of N integer values, and I need to sum elements and distribute these sums by equal parts in the range from 150 to 250, but without to mix indexes. For example {50,100,50,250,200,49, 100} will be (50+100+50), (250), (200+49), (100). This is the best solution here. But I can’t write the optimal algorithm for this case and I have a result with a small amount in the middle or a big number in the end. But I need the most optimal result. Does anyone have any ideas is it possible to write this distribute algorithm?

web crawlers – Remove large amounts of pages from Bing Index Explorer

You can’t make search engines forget about old URLs on your site. Once search engines find content at a URL, they will continue to periodically crawl that URL indefinitely, even after you remove the content change the URL to show a 404 error.

Instead of returning a 404 error you could change to a more relevant status:

  • 301 Permanent Redirect – For the case in which there is a new page that replaces the old page.
  • 410 Gone – For the case that the removed page has no replacement.

Using those statuses can make search engine crawlers back off some, especially over time. However I have old URLs on my site that have been redirecting for 20 years, and Googlebot still comes and checks on them occasionally.

You say your server is getting “slammed” by these requests because they have to go through WordPress. You can add rules to .htaccess to short circuit these URLs and serve them much faster. You can usually serve tens or hundreds of 404 requests for the same cost as one real page this way. These rules should go at the top of your .htaccess file, before any rules for WordPress.

RewriteRule ^/?html/ - [G]
RewriteRule ^/?v2/ - [G]
RewriteRule ^/?g/ - [G]
RewriteRule .asp$ - [G]
RewriteRule .html$ - [G]

The first three rules would show a 410 Gone for everything in the folders you mention. The last two rules would showe 410 Gone by extension on the URL. You should only use them if there are no more .asp or .html URLs left on your site.

If you can’t get the rare case that it might be harming your server, you could use block those URLs in robots.txt.

Disallow: /html
Disallow: /v2
Disallow: /g
Disallow: *.asp
Disallow: *.html

Both Googlebot and Bingbot understand and honor wildcard rules, but most bots don’t. The last two rules will only help for specific bots that support advanced rules.

I would only block the URLs with robots.txt if the requests cause performance problems even after trying the rewrite rules. If you disallow the URLs, search engines may still index the URLs and send some visitors to them. See Why do Google search results include pages disallowed in robots.txt? If you allow the URLs to be crawled, the bots will continue to hit the URLs, but search engines will not include them in their search indexes.

sql server – Why does my table keep getting corrupted after removing large amounts of data?

I have a database that is about 400 gigs. The majority of the space is consumed by images of utility bills stored in binary form as varchar(max) in a single table. It’s a poor database design that I inherited. I have already exported the utility bills as pdfs to a network folder. Now I am trying to replace the images in the database with an empty string. I have been updating in small batches of 40k to 80k records at a time. The problem is that after a few batches, the table gets corrupted. I’ll get a message that the update failed. When I run CHECKDB I get

Msg 8909, Level 16, State 1, Line 1 Table error

It’s happened twice so far and I have to restore the database from a backup.

I need to know why this is happening and if there is a way to avoid it. I’m doing this in a test environment and I need to get this problem figured out before I can work on the production database.

Thank you for your help.

post processing – Methods (mechanics) to select a set of pictures from large amounts

Divide and conquer.

You’re obviously shooting a lot of shots of a given subject.

Let’s break things down a bit.

Trip
    5 zillion pix.

Instead

Trip
    Day 1
    Day 2
    ...

Now here’s a new concept: For each day, we will have a number of subjects

Trip
    Day 1
        Cathedral at Cologne

A step further, a given subject with have one or more shoots. A shoot is a bunch of related shots taken in hopes of getting a zinger. A shoot will usually mean that you don’t change lenses, you don’t pick up your jacket. Likely a shoot from first to last is under 2 minutes and will have 5 to 30 shots.

Trip
    Day 1
        Cathedral at Cologne
            St. Stephan's Tomb

Different software will have different techniques.

It was a 10 day trip. You want a 100 pic show, so you need to average 10 pix a day.

Now: For each shoot: Pick ONE image of what you have and give it 1 star. The rest are unrated.

Set your viewer to 1 or more stars. This means that for the 13 pictures of St. stephan’s tomb that didn’t get a star they vanish.

How many subjects today? 4? Ok. 10/4 = 2.5. Double that. 5.

For each subject pick 3-5 of the shoots bests and give them 1 more star each. This gives you a few too many. Suck it up.

Repeat for each day.

So right now you have about 20 shots per day

Do you need all the subjects? Maybe not. Eliminate.

Did some event stick out? Go back and pick out a few more from that set.

But at this point you are picking 100 pix out of 200. Some of this depends on how you organize your narrative. If you are giving a “Religious Archtecture of Europe” you concentrate on churches and monasteries. If it’s Fortifications of Europe, it’s more castles and Siegfried line.

In passing: A good slide show has 7-9 shots a minute. You don’t say anything for 3 seconds. They won’t hear you anyway.

Multibit 0.5.1 wallet unconfirmed amounts no seed words

please help. I am not technical and I gave bitcoin as a gift in 2017 in a Multibit wallet which is now defunct. I have opened the wallet but it says the amounts are unconfirmed. I have repaired the wallet twice. I don’t have the seed words and don’t know how to get them.

unity – Sending different message on a game level after different amounts of time

So, I am working on an interactive game to aid with anxiety management. In one of the gameplay options, the player has to press buttons for a certain amount of time. Only after the required time has lapsed (with the player holding the buttons down), will the avatar move. So, for example the instruction will be “Press the buttons for 20 seconds to move the player”. This part is working.
As this is an anxiety game, however, I want to incorporate some breathing activity/instruction into the gameplay. I have chosen to use the 4-7-8 second rule, whereby the player breathes in for 4 seconds, holds their breath for 7 seconds and exhales for 8 seconds. I have no idea how to incorporate something like this into my existent game. Any tips or pointers would be appreciated. I understand that a second timer would be needed, but how would that be able to discern between the 4 seconds, 7 seconds and 8 seconds?

Attached is my existent timer script:

using UnityEngine;
using System.Collections;
using UnityEngine.UI;
using UnityEngine.SceneManagement;

public class Timer : MonoBehaviour
{
    public int timeLeft;
    public Text countdownText;
    private bool timerStarted;

    int start;

    public void BeginTimer()
    {
        StartCoroutine(LoseTime());
    }

    void Awake()
    {
        start = timeLeft;
    }

    public void StopTimer()
    {
        StopAllCoroutines();
        Respawn.Instance.Player2.transform.position = Respawn.Instance.RespawnPoint2.transform.position;
        PlayerController.Instance.timerStarted = false;
        timeLeft = start;
        countdownText.text = "Time Left= " + start;
    }

    private IEnumerator LoseTime()
    {
        while (timeLeft > 0)
        {
            yield return new WaitForSeconds(1);
            timeLeft--;
            countdownText.text = "Time Left= " + timeLeft;
        }
       
        PlayerController.Instance.Character.GetComponent<Rigidbody2D>().AddForce(PlayerController.Instance.height, ForceMode2D.Impulse);
         //SphereController.Instance.Player.GetComponent<Renderer>().material;
        print("hi");
    }

    public void ChangeLevel()
    {
        SceneManager.LoadScene(SceneManager.GetActiveScene().buildIndex + 1);
    }
}

And my playercontroller script:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using System.IO.Ports;
using UnityEngine.UI;

public class PlayerController : MonoBehaviour
{
    public static PlayerController Instance { set; get; }
    public float Speed;
    public Vector2 height;
    public float xMax, xMin, yMin, yMax;
    public bool buttonPressed = false;

    public Rigidbody2D Character;
    public Timer timer;
    public Rigidbody2D _rigidbody;

    public bool timerStarted = false;

    private SerialPort sp = new SerialPort("\\.\COM4", 9600);

    Animator anim;
    string value;
    int button;

    private void Awake()
    {
        FetchReferences();
        Instance = this;

        //print("hi");

        if (AvatarChooser.ava == 1)
        {
            anim = GameObject.FindWithTag("Player").GetComponent<Animator>();
            anim.runtimeAnimatorController = Resources.Load<RuntimeAnimatorController>("3");
            //print("hi");
        }

        if (AvatarChooser.ava == 2)
        {
            anim = GameObject.FindWithTag("Player").GetComponent<Animator>();
            anim.runtimeAnimatorController = Resources.Load<RuntimeAnimatorController>("1");
        }

        if (AvatarChooser.ava == 3)
        {
            anim = GameObject.FindWithTag("Player").GetComponent<Animator>();
            anim.runtimeAnimatorController = Resources.Load<RuntimeAnimatorController>("2");
        }
    }

    (ContextMenu("FetchReferences"))
    private void FetchReferences()
    {
        if (!Character) Character = GameObject.FindWithTag("Player").GetComponent<Rigidbody2D>();
        if (!timer) timer = GameObject.Find("Timer").GetComponent<Timer>();
    }

    private void Start()
    {
        if (!sp.IsOpen)
        {
            sp.Open(); // Open 
        }
        sp.ReadTimeout = 1;

        //print(AvatarChooser.ava);
    }

    private void Update()
    {
        if (sp.IsOpen)
        {
            try
            {

                value = sp.ReadLine(); //Read the information
                button = int.Parse(value);

                if (button == 9)/*(Input.GetKeyDown(KeyCode.Space))*/  //jump
                {
                    buttonPressed = true;

                   if (!timerStarted)
                    {

                        timerStarted = true;
                        timer.BeginTimer();
                    }
                }
                else
                {

                    timer.StopTimer();
                }
            }
            catch (System.Exception)
            {

            }
        }
    }


    private void FixedUpdate()
    {
        Character.position = new Vector3
                    (
                        Mathf.Clamp(GetComponent<Rigidbody2D>().position.x, xMin, xMax),
                        Mathf.Clamp(GetComponent<Rigidbody2D>().position.y, yMin, yMax)
                    );
    }
   
    void OnDisable()
    {
        sp.Close();
    }

}

receiving – Using BTCPay To Accept Undefined Payment Amounts

Following advice from users on this site, I have implemented a BTCPay Server and got it set up to receive payments. I am comfortable using the API.

My need is to set up receive addresses for many clients (1000+) but the clients pay indeterminate amounts. That is, I cannot generate fixed-amount invoices for the clients because they pay whichever amount they wish to.

I need to generate a receive address for each client, and then credit their account balance (in USD) on my site with however much they sent to the address in BTC once it has been received and settled.

Does anyone know how I can achieve this in BTCPay Server? I have listed my ideas below but am looking for the ‘correct’ solution as opposed to what feels like a hack.

Current Ideas

  1. Generate an invoice for a tiny amount, and then handle the over-payment that the client makes when they send the BTC. The over-payment must not be refunded.

  2. Ask the client how much they are expecting to pay on the website and generate an invoice for about this amount. I would love to avoid this method as the use case is for them to send however much they want to.

databases – Storage of Large Amounts of Game Data (Player stats, matches)

I’m curious as to what the best way would be to store my data. Currently, I’ve setup a matchmaking system, and player stats. When a match ends, my server submits the player data to my ruby on rails api, saving the data to my PostgreSQL database, making at least 10~ queries at the end of a match. However, some people mentioned using MongoDB instead. Why would I want to use Mongo over postgres? Postgres works fine, and it’s what I’ve worked with at my current job to store millions of records. Does anyone have any opinions on this, or what would be statistically the best option?

network – Sudden large amounts of ports and services on my router

I’ve conducted a scan of my network and I’ve noticed a worrying amount of ports and services now running on my router.

I have a Virgin Media Superhub2. Below are the ports and services that are open. I cannot find information on the internet.

enter image description here

Can anyone shed some light as to what has possessed my router? Some of the services are things I’ve never seen before!