Storing timeseries data with dynamic number of columns and rows to a suitable database

I have a timeseries pandas dataframe which dynamically increases the columns every minute as well as adds a new row:

Initial:

timestamp                100     200     300
2020-11-01 12:00:00       4       3       5

Next minute:

timestamp                100     200     300   500
2020-11-01 12:00:00       4       3       5     0
2020-11-01 12:01:00      14       3       5     4

The dataframe has these updated values and so on every minute.

so ideally, I want to design a database solution that supports such a dynamic column structure. The number of columns could grow to over 20-30k+ and since it’s one minute timeseries, it will have 500k+ rows per year.

I’ve read that relational db’s have a limit on the number of columns so that might not work here, but also, since I am setting the data for new columns and assigning a default value(0) to previous timestamps, I lose out on the DEFAULT param that’s there on MySQL.

Eventually, I will be querying data for 1 day, 1 month to get the data for the columns and their values.

Please suggest a suitable database solution for this type of dynamic row and column data.

cluster – Best options for storing and sharing data over mutiple servers, to be accessed by multiple users

In the environment I currently work in there are several servers dedicated to various departments. However we recently got our hands on several identical servers potentially, around 10+. I’m hoping someone can point me in the right direction on a good option to utilize all of these servers effectively.

Essentially we store large files on the server to be downloaded and less frequently uploaded to the servers.

My goal is to create a good option for the data to be accessible in one (or a few) central location(s) that I can point scripts to and the users can access easily.

What I would ideally like to do, is link up all of these servers as one, or a couple Virtual Machines, or one Networked File Share, that has a good amount of bandwidth and redundancy to keep the files secure and allow the data to be copied down as quickly as possible.

Some things I have been looking into to accomplish these tasks:

-DFS

-Failover Clustering

-Starwind Virtual SAN (The free version)

As a note these servers are quite old, from around 9 years ago.

Any advice on configuring this would be greatly appreciated, I’m a bit out of my depth

What is the purpose of storing difficulty in block?

The difficulty is calculated by the node itself, after every block addition. So why does every block contain unnecessary information about difficulty if it is anyway calculated by the node?

dnd 5e – Does reading the Book of Exalted Deeds prevent casting non-upcast 5th-level spells into a Ring of Spell Storing?

The Ring of Spell Storing states (emphasis mine):

This ring stores spells cast into it, holding them until the attuned wearer uses them. The ring can store up to 5 levels worth of spells at a time. When found, it contains 1d6 − 1 levels of stored spells chosen by the GM.

Any creature can cast a spell of 1st through 5th level into the ring by touching the ring as the spell is cast. The spell has no effect, other than to be stored in the ring. If the ring can’t hold the spell, the spell is expended without effect. The level of the slot used to cast the spell determines how much space it uses.

While wearing this ring, you can cast any spell stored in it. The spell uses the slot level, spell save DC, spell attack bonus, and spellcasting ability of the original caster, but is otherwise treated as if you cast the spell. The spell cast from the ring is no longer stored in it, freeing up space.

The Book of Exalted Deeds states:

(…) Once you’ve read and studied the book, any spell slot you expend to cast a cleric or paladin spell counts as a spell slot of one level higher. (…)

Thus, any 5th-level spell cast using a 5th-level spell slot would count as using a 6th-level spell slot, which cannot be stored into a Ring of Spell Storing. Is this correct? Is there some rule I’ve missed or way around this besides ending the benefits from the Book?

Storing external databases data into one

I need to store different datasets coming from a provider where each dataset has it’s own release path.
These datasets can be combined together to get the full picture of the data available from the provider.

I know from the doc of the publisher the combination of versions that are allowed.
My pain point is that I need to keep track of the version for each dataset.

Example of data:

“Main” dataset from publisher “ABC” has version “1.0”.
“Ext” dataset from publisher “ABC” has version “release_3”

My schema is as follow:

Provider
ProviderId

Dataset
DatasetId
ProviderId

Version
VersionId
DatasetId

Main
MainId
VersionId

Ext
ExtId
MainId
VersionId

Based on that, is it a problem that FK “VersionId” in tables “Main” and “Ext” references a different record in table “Version” ?

I’m afraid that any user querying the DB will not expect to have diverging Version (as the FK name is identical in both tables).
Unfortunately that’s the reality of the data provided by the publisher.

Is there a better design to accomplish the same result ?

NB: It is possible that in the future, datasets from different providers need to be combined.

Thanks

mongodb – Which NoSQL for storing images as 2D arrays (BLOBs)

I have large number of images that I need to access on per-pixel-basis in my app. The images are downloaded from web and are quite large (4k). Update is done several times a day (tousands of images), reading is non-stop.

Currently I use PostGis and store images tiled in raster column type. However, performance is not the best, mainly updates are quite slow (transaction, index update etc).

So I would like to use NoSQL database, since I really dont need SQL.

My workflow is:

Insertion – download image -> split it to tiles of size 64×64 (compress them if necessary with arithetic or huffman coder) -> store tiles in database

Selection – read pixel (x, y) -> locate tile -> retrieve tile -> “unpack” tile in memory -> locate pixel (x, y) -> return value to user

What database should I use for this purpose? I have looked to MongoDB and Cassandra. The later one seems more appropriate for this task.

What are some methods of storing personal notes alongside code?

When exploring a new codebase, what would be some methods of storing personal notes or annotations alongside the code? If I were the only maintainer, I might consider added an actual code comment into the code, but doing so in a team would just pollute the code and the git logs.

Ideally there might be something that is not editor-specific, but I’m open to editor-specific suggestions. I personally code in VIM 50% and JetBrains IDEs (With IdeaVIM) 50% of the time. Jetbrains has a Code Notes plugin, but unfortunately that plugin does not work with the current version of the IDE.

Example notes and annotations:

foo = get_foo(bar, boom); // Actually returns a proto-foo, not a foo
foo += 1 // Alice says this is to avoid the lochness bug
if ( !flashing ) brake_light = 1;  // Bob insists on checking flashing in new code even though that feature has been deprecated for 17 years.
foo = processThing(foo); // TODO Figure out what side effects this has

unity – How do I storing a Score in each level?

I have a problem when I store the DistanceScore to text, the problem is why the DistanceScore always storing at text number 1, this happens when I play at level 2 which is the DistanceScore could be store in text number 2and so on, FYI I have 3 levels here.

here the preview
enter image description here

enter image description here

I played level 2 and decide to defeat for knowing the distance

as you can see on the top of level 2 gameplay is distance so when i dead the distance saved and storing to text each level.

enter image description here

and when I go to the main menu the Level 1text updated instead of Level 2.

this is my script:

public Item() _item;

public int mapIndex;

public enum ITEM_TYPE
{
    LEVEL
}

public ITEM_TYPE _type;

private void Start()
{
    loadItemInformation();
}

public void loadItemInformation()
{
    for (int i = 0; i < _item.Length; i++)
    {
        if (_type == ITEM_TYPE.LEVEL)
        {
            _item(i).bestScore.enabled = true;
            _item(i).distance.enabled = true;

            float distanceUi = PlayerPrefs.GetFloat("distance" + i.ToString());
            float value = PlayerPrefs.GetFloat("Time" + i.ToString());
            float minutes = Mathf.FloorToInt(value / 60);
            float seconds = Mathf.FloorToInt(value % 60);

            if (value == 0)
            {
                _item(i).bestScore.text = "No Record Yet";
            }

            else
            {
                _item(i).distance.text = "Unstopable ";
                _item(i).bestScore.text = "Best Time = " + string.Format("{0:00}:{1:00}", minutes, seconds);
            }

            _item(i).distance.text = "Long distance = " + distanceUi.ToString();
            

        }
    }
}

}

    (System.Serializable)
    public class Item
    {
        public TextMeshProUGUI bestScore;
        public TextMeshProUGUI distance;
    }

here the distance script:

#region Distance
(Header("Distance"))
public Transform checkPoint;
public TextMeshProUGUI distanceText;
M_GameManager _GameManager;
private float distance;
private float highDistance;
float index;
#endregion


private void Start()
{
    _GameManager = FindObjectOfType<M_GameManager>();
    checkPoint = GameObject.FindGameObjectWithTag("Finish").transform;
    distanceText = GameObject.FindGameObjectWithTag("DistanceTxt").GetComponent<TextMeshProUGUI>();
    index = (float)(PlayerPrefs.GetInt("Level"));// typo here i call it set and now I've fix it
}
index = PlayerPrefs.GetFloat("Level");
distance = (checkPoint.transform.position.x - transform.position.x);
    if (transform.position.x > checkPoint.transform.position.x)
    {
        distance = 0;
        //PlayerPrefs.SetFloat("distance" + index, distance);
    }
    if (_GameManager.isDead || _GameManager.isTimeout || _GameManager.fuelFinished)
    {
        PlayerPrefs.SetFloat("distance" + index, distance);
        PlayerPrefs.Save();
    }
    distanceText.text = distance.ToString("F1") + "m";

and for _item(i).bestScore.text this work to storing each level but why for the distance not working.

cache – Storing statistics for cached content

I have a database-driven website where the content only changes about once per day. The page consists of a list of businesses. Every time a page is viewed each business receives a “view.” Since the content does not change very often I would like to serve it from a cache to avoid lookups/speed up the request (e.g. Nginx or Varnish). What is the best practice for serving cached content while still capturing statistics every time the business listings are viewed?

I would like to avoid Javascript. The other option that comes to mind is the use of a static image with query parameters: clear.png?listings=1,2,3,4,5,6,7,8,9,10, but that feels sloppy. Is there a better way?

encryption – is this a good practice for storing private keys?

Couple observations:

he also have to provide his password and I’m gonna use his password to decrypt his private key and sign the transaction using it

not soure what exactly this means, but you shouldn’t use user’s private key. Transaction signing should be done in client context (user memory). User shall not delegate signing to shared process or send private key anywhere outside.

Since private key is decrypted in memory (even for short time frame), it is in memory in clear text, so password can be extracted from there. This is a common issue for any kind of software-based systems. HSMs do key extraction and key operation in tamper-evident zone. This is main purpose of HSM.

In fact, certain Windows components implement this (such as EFS, for example). Private key is encrypted with user password, so even systems administrator cannot extract the key unless they know password.

And there is one problem: password management. If user lose their password, encryption key is lost as well. If password is reset, the key is lost. Windows correctly handles standard password changes using windows security dialog, where user types current password and provide a new one. In this case, Windows automatically re-encrypt required keys with new password and everything is transparent to the user.

Update

an update to your edits:

offline backups aren’t different than online backups in terms of security. Therefore offline backup shall include private keys and passwords only in encrypted form. Suggested pasword management solution is way too far from acceptable (considering this is a platform that does financial transactions).

Such cases are solved in different way: you designate a group of persons as key recovery agants (KRA) and store their public keys somewhere on a system. Once the private key is generated, it is automatically encrypted using public keys of recovery agents. In the case if user loses their access to the key, it can be recovered by providing a quorum of recovery agents. Key recovery agents shall not have access to encrypted private key database. These should be different persons. One person group manages the encrypted private key database and another group owns key recovery certificates.

As of HSMs, there are network-attached HSMs (nCipher nShield, SafeNet Luna and others. You can use them on VPS over network. Alternatively, there are cloud-based HSMs.