java – BLOB storage for High Performance Computing System

We are designing a distributed system capable of processing large scale amount of BLOB data. Each node should load its own portion of data in a high speed way and is responsible for processing its own data.

Due to network limitations it is better that each node does not burden network bandwidth too much. Practically it would be better for each processing node to be a part of a database cluster and loads data from its own node which is a member of the data cluster. Therefore, it is better for this database system to work with minimal memory.

It is mandatory for the database to be responsible for managing the shards and replicas, and adding or removing nodes to it should be done with minimum cost.

What is the suitable database system for this design?

QuantumSpider.com KVM East/West Coast ★ Storage and Performance ★ US & EU ★ Starting at $3/m!


* QuantumSpider.com KVM East/West Coast ★ Storage and Performance ★ US & EU ★ Starting at $3/m!


Pick the best plan that suits your needs. US & new server location in EU below.

All plans include:

  • root access
  • 1 IPv4 Address
  • IPv6 capable
  • 1Gbps Bandwidth
  • Easy to use control panel to self manage


View all plans here

  • GET MORE FOR LESS
  • IPv6 capability, High Storage and RAM
  • Fast 24/7 Support Response
  • Windows or Linux, custom ISO
  • Upgrade whenever you need

US Optimized KVM options

Tiny KVM

Small KVM

Medium KVM

Large KVM

Huge KVM

vCPUS

1

1

2

4

8

RAM

1GB

1GB

2GB

4GB

8GB

SWAP

1GB

1GB

2GB

4GB

8GB

Storage

30GB

60GB

100GB

200GB

400GB

1Gbps Bandwidth

1TB

1TB

2TB

5TB

8TB

Pricing

$5/month

$10/month

$20/month

$40/month

$60/month

ORDER HERE

ORDER HERE

ORDER HERE

ORDER HERE

ORDER HERE

All plans include:

  • root access
  • 1 IPv4 Address
  • IPv6 capable
  • Premium 300 Mbit/s
  • DDoS protection
  • Easy to use control panel to self manage

View all plans here

  • GET MORE FOR LESS
  • IPv6 capability, High Storage and RAM
  • Fast 24/7 Support Response
  • Windows or Linux, custom ISO
  • Upgrade whenever you need

EU PureSSD KVM options

Tiny PureSSD KVM

Small PureSSD KVM

Medium PureSSD KVM

Large PureSSD KVM

Super PureSSD KVM

vCPUS

1

1

2

4

8

RAM

1GB

1GB

2GB

4GB

8GB

SWAP

1GB

1GB

2GB

4GB

8GB

Storage

20GB

40GB

80GB

120GB

200GB

1Gbps Bandwidth

1TB

1TB

2TB

5TB

8TB

Pricing

$5/month

$10/month

$20/month

$40/month

$60/month

ORDER HERE

ORDER HERE

ORDER HERE

ORDER HERE

ORDER HERE

Premium cPanel Shared Hosting Provider – Professional Web Services for Businesses and Individuals since 2013

Caching or in-memory table in Azure for performance

I am building an Angular web application that retrieves part of its data from a Azure SQL Database table via APIs developed in Azure Functions (with Azure API Management at the API Gateway). The data in the table (30k records) do not change for at least 24 hours. The web app needs to display this data in a grid (table structure) with pagination and users can apply filter conditions to retrieve and show a subset of the data in the grid (again with pagination). They can also sort the data on a column in the grid. The web app will need to be accessed by few hundred users on their iPad/tablet with 3G internet speed. Keeping the latency in mind, I am considering one of these two options for optimum performance of the web app:

1) Cache all the records from the DB table in Azure Redis Cache with cache refresh every 24 hours, so that the application will fetch the data to populate the grid from the cache, thus avoiding expensive SQL DB disk I/O. However, I am not sure how the filtering based on a field value or range of values will happen from Redis Cache data. I have read about using Hash data type for storing multivalued objects in Redis and SortedSet for storing sorted data, but I am particularly not sure about filtering data in Redis based on the range of numeric values (similar to BETWEEN clause in SQL) in Redis Cache. Also, is it at all advisable to use Redis in this way for my use case?

2) Use in-memory OLTP (memory optimized table for this particular DB table) in Azure SQL DB for faster data retrieval. This will allow to handle the filtering and sorting requests from the web app with plain SQL queries. However, I am not sure if it’s appropriate to use memory optimized tables for improving just table read performance (from what I read, Microsoft suggests to use it for insert-heavy transactional operations).

Any comments or suggestions on the above two options or any other alternative way to achieve performance optimization?

f stop – How can I compute low light performance for different formats?

How can I compare low light performance for different formats?

Easy. By calculating the 35mm equivalent f-stop.

The 35mm equivalent f-stop divisor is the divisor in the f-stop multiplied by crop factor.

An example:

  • You have 17-55mm f/2.8 lens for a system with crop factor 1.6x and 24-105mm f/4 lens for full frame.
  • The 24-105mm f/4 lens for full frame is simply f/4 equivalent.
  • The 17-55mm f/2.8 lens is f/4.48 equivalent because 2.8*1.6 = 4.48

Thus, this example shows that a full frame 24-105mm f/4 lens has better low-light performance than a 17-55mm f/2.8 crop lens.

Someone could complain that the same true f-stop, same shutter speed and same ISO gives the same exposure, and therefore, because shutter speed is limited by motion blur and ISO is limited by noise, the true f-stop (and not the 35mm equivalent f-stop) would tell the low-light performance.

Such a complainer would ignore the fact that on full frame, you can use higher ISO than on crop. For example, if you have a 1.6x crop camera where ISO 800 is the highest acceptable ISO and higher ISOs have too much noise, a full frame camera with equivalent generation of technology would allow you to use 1.6^2 * 800 = 2048 as the ISO.

performance tuning – Determinant of 11×11 Matrix consumes immense amount of memory

I’m trying to take the determinant of an 11×11 matrix, constructed out of some functions I’ve defined

lagz(n_, m_, z_, zbar_) := 
 1/Sqrt(Pi*a^2*n!*m!)* E^(z*zbar/(2.*a^4)) * (a)^(m + n)*
  D( E^(-(z*zbar)/a^4), {z, n}, {zbar, m})
lagnlz(n_, l_, z_, zbar_) := 
 Sqrt(n!/(Pi*a^2*(n + l)!))* (z/a)^l LaguerreL(n, l, z*zbar/a^2)*
  E^(-z*zbar/(2.*a^2))

lag(n_, l_, r_, (Theta)_) := 
 lagnlz(n, l, z, zbar) /. {z -> r*E^(I*(Theta)), 
   zbar -> r*E^(-I*(Theta))}
lagcc(n_, l_, r_, (Theta)_) := 
 lagnlz(n, l, z, zbar) /. {z -> r*E^(-I*(Theta)), 
   zbar -> r*E^(I*(Theta))}

The main functions are lag and lagcc, functions of n, l, z, and zbar (n and l are integers that specificy the function and z and zbar are coordinates). I pick which of these enter the matrix with the following code:

MaxEn = 5;
MaxState = Binomial(MaxEn + 1, 2);
nlBasis = {{0, 0}};
part = 10;
For(En = 1, En < MaxEn, En++, 
 instances = 
  FindInstance(2 n + l == En && n + l >= 0 && n >= 0, {n, l}, 
   Integers, En + 1);
 For(iter = 1, iter <= En + 1, iter++,
  nlBasis = Join(nlBasis, {{n, l}} /. instances((iter)))
  )
 )
nlBasis // MatrixForm;
(CapitalOmega) = .0001;
stateEnergy = 
  Table(2*nlBasis((i, 1)) + 
    nlBasis((i, 2)) *(1 - (CapitalOmega)), {i, 1, MaxState});
stateEnergy // MatrixForm;
lowest = Ordering(stateEnergy, part); 

The variable in the above code named “part” determines the size of the matrix (for part = 10 corresponds to a 10×10 matrix). Also, each row is a function of a different set of coordinates z and zbar (meaning z1, zbar1, …). Actually to be more clear, I substitute z and zbar for x and theta. I then construct the matrix with the code:

Table( lag(nlBasis((lowest((j)), 1)), nlBasis((lowest((j)), 2)), 
  Subscript(x, i), Subscript((Theta), i)), {i, 1, part}, {j, 1, 
  part})

All of this runs very quickly. I use the Subscript(x,i) and Subscript(theta,i) as variables However, when I try to find the determinant of this matrix, it takes a relatively long time (on the order of a few minutes). For the 11×11 case, it takes much longer and often just Aborts mid computation. Is there any reason this determinant takes so long to compute, and is there any way to make it faster? Ultimately, I intend to find the maximum of this determinant in all 2*part variables. I was thinking perhaps it’s the symbolic aspect of this determinant that is taking so long and that if the numbers were already plugged in, it might find the determinant faster. However, I don’t know how to make the “FindMaximum” function plug in the coordinates into the matrix before it computes the determinant, unless it is doing this already and it’s still this slow. Regardless, I’m curious as to why this takes so long and if there is any way to speed this up.

python – How to write this 7-nested loop in a better way while keeping (or improving) performance?

I’m doing a challenge online, the problem is to find all happy numbers from 1 up to 1.000.000.0.

In a part of my code , I have to sum all of the digits to the power of two for numbers from 0 to 1.000.000.0 then I have to find if the sum is inside a given set of integers and if this is true put this number inside a list . At the end I wrote this messy solution to complete this task :

for a in range(10):
    l = a*a
    for b in range(10):
        l += b*b
        for c in range(10):
            l += c*c
            for d in range(10):
                l +=d*d
                for e in range(10):
                    l += e*e
                    for f in range(10):
                        l += f*f
                        for g in range(10):
                            l += g*g
                            if l in set : list.append(a*1000000+b*100000+c*10000+d*1000+e*100+f*10+g) 
                            l -= g*g
                            if g == 9 : 
                                l -= f*f
                                if f == 9 :
                                    l -= e*e
                                    if e == 9 :
                                        l -= d*d
                                        if d == 9 :
                                            l-= c*c
                                            if c == 9 :
                                                l-= b*b
                                                if b == 9 :
                                                    l-= a*a

But even if It’s good enough in terms of performance it is also very bad written.

I tried this but it’s slower :

from itertools import product
for a,b,c,d,e,f,g in product(range(10),repeat=7):
    k = a*a + b*b + c*c + d*d + e*e + f*f + g*g
    if k in set : list.append(a*1000000+b*100000+c*10000+d*1000+e*100+f*10+g)

I don’t find a way to mantain ( or improve ) efficiency and at the same time writing it in a better way. How could it be achieved?

MIUI 12 performance boost?

My question is in reference to Redmi Note 7, which I have seen slowing down in performance over time. So my question:
Will the new MIUI 12 update on all Xiaomi phones bring increase performance on say gaming and add more fluidity to using the device?

performance – Rate limit a method by generating particular load on demand in C#

I am working on a project where I want to generate random throughput on a particular method so that I can do some performance testing on it. This way I can test my method by generating random throughputs and see how my method works under all those scenarios.

For example: I need to call my doIOStuff method at an approximate rate of x requests per second from multiple threads where x will be less than 2000 mostly but it really doesn’t matter in this case. It doesn’t have to be accurate so there is some room for an error but the overall idea is I need to make sure that my method doIOStuff is executed no more than x times in a sliding window of y seconds.

Assuming we start n threads and want a maximum of m calls per second. We can achieve this by having each thread generate a random number between 0 and 1, k times per second and call doIOStuff method only if the generated number is less than m / n / k.

Below is the code I got which uses global variables and it does the job but I think it can be improved a lot where I can use some cancellation tokens as well and make it more efficient and clean.

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Threading;


namespace ConsoleApp
{
    class Program
    {
        const int m_threads = 100;
        const int n_throughput = 2000;
        const int k_toss_per_second = 2000; // Note that k_toss_per_second x  m_threads >= n_throughput

        static void Main(string() args)
        {
            var tasks = new List<Task>();

            for (int i = 0; i < m_threads; i++)
                tasks.Add(Task.Factory.StartNew(() => doIOStuff()));

            Task.WaitAll(tasks.ToArray());
            Console.WriteLine("All threads complete");
        }


        static void callDoIOStuff()
        {
           int sleep_time = (int) (1000 * 1.0d / k_toss_per_second);
           double threshold = (double) n_throughput / m_threads / k_toss_per_second; 
           Random random = new Random();
           while (true) {
                Thread.Sleep(sleep_time);
                if (random.NextDouble() < threshold)
                    doIOStuff();
            }
        }

        static void doIOStuff()
        {
            // do some IO work
        }
    }
}

I wanted to see what can we do here to make it more efficient and clean so that it can used in production testing for generating random throughput load.

postgresql – Performance of indexing varchar or text as an alternative to UUIDs?

I was reading up on nanoid and was possibly considering using the generated ids as the primary key for a table. The generated IDs are strings such as V1StGXR8_Z5jdHi6B-myT.

In my research I came across the following comment:

One of the benefits at least in postgres is that uuid v4 can be treated like a native type when stored which makes it fast to search.

Is it necessarily true that a primary key based on a UUID column would be more performant than a primary key based on a text or varchar column? If so, is there some other data type I can use to store nanoids which would match the performance of the native UUID type?

performance tuning – Efficiently populate sparse matrix with band structure

I’m trying to efficiently populate elements of a very large (2^20 x 2^20) symmetric matrix with 1s – luckily the matrix is very sparse, <0.1% filling. Further, the matrix has a very well defined periodic banded structure, as shown here:

enter image description here.

In reality, this matrix is the result of a series of KroneckerProducts of 2×2 matrices, which is what gives it that characteristic banded structure – I’m hoping to find a way to speed up the generation without using kronecker products, because even with sparse matrices, the computation can take several seconds or minutes depending on the final dimensionality.

My first question relates to creating this sparse matrix efficiently. I’ve toyed with lots of different ways of generating even simple bands for the sparse array. For simply populating on the diagonal, the quickest method clearly seems to be to use the {i_,i_} notation, as shown here:

dim = 15;

aa = SparseArray({i_, i_} -> 1, {2^dim, 2^dim}) // RepeatedTiming;
bb = SparseArray(Band({1, 1}) -> 1, {2^dim, 2^dim}) // RepeatedTiming;
cc = SparseArray(Table({ii, ii} -> 1, {ii, 2^dim}), {2^dim, 2^dim}) //RepeatedTiming;
dd = SparseArray(Normal(AssociationThread(Table({ii, ii}, {ii, 2^dim}) -> Table(1, {ii, 2^dim}))), {2^dim,2^dim}) // RepeatedTiming;

Column({aa((1)), bb((1)), cc((1)), dd((1))})

aa((2)) == bb((2)) == cc((2)) == dd((2))
0.000309
0.031
0.081
0.054

True

However, when we try to do off-diagonal entries, this gets much worse, presumably because the condition has to be continually checked:

dim = 15;

aa = SparseArray({i_, j_} /; j - i == 1 -> 1., {2^dim, 2^dim}) // RepeatedTiming;
bb = SparseArray(Band({1, 2}) -> 1, {2^dim, 2^dim}) // RepeatedTiming;
cc = SparseArray(Table({ii, ii + 1} -> 1, {ii, 2^dim - 1}), {2^dim, 2^dim}) // RepeatedTiming;
dd = SparseArray(Normal(AssociationThread(Table({ii, ii + 1}, {ii, 2^dim - 1}) -> Table(1, {ii, 2^dim - 1}))), {2^dim, 2^dim}) // RepeatedTiming;

Column({aa((1)), bb((1)), cc((1)), dd((1))})

aa((2)) == bb((2)) == cc((2)) == dd((2))
0.185
0.031
0.095
0.052

True

From those two examples then it seems like Band is our best choice, but Band is still painfully slow, especially when compared to the {i_,i_} for the diagonal. Further, this is more frustrating, because in MATLAB the same problem can be accomplished an order of magnitude faster (this took ~1.4 ms):

enter image description here

But the fact that the original {i_,i_} case for the diagonal was so fast suggests that there is a more efficient way to do this.

So then my first question is: given all of that, is there a more efficient way to populate the bands of our sparse matrix, so that the speed can at least rival the equivalent in MATLAB?

And my second question, a bit predicated on the first: with whatever method is the most efficient, what is the best way to generate the periodic banding structure present in the final matrix (see above). You can accomplish it with Band by manually inserting spaces with 0s, but doing so can’t be the most efficient way.

Finally, because of that period-2 banded structure of the final matrix, where each quadrant is a recursive block of ever smaller diagonal matrices with side length smaller by a factor of 2, maybe you could generate all the smaller diagonal blocks, and then just place them in the final matrix – I’m not sure how this would be accomplished however. Of course, remember that the matrix is symmetric, so I would think that would help with efficient generation because really just one triangle has to be generated and then flipped.

Addendum: MATLAB code for generating the plot, as requested in the comments. This takes on the order of milliseconds for my machine, even with N=15.

N = 4; 
a_id    = (0:2^N-1)';

dimH        = length(a_id);
AA          = sparse(dimH, dimH);

for i = 1:N
    (~,k1,k2) = intersect(a_id, bitxor(a_id,2^(i-1)));
    AA        = AA + sparse(k1,k2,ones(length(k1),1)/2,dimH,dimH);
end

DreamProxies - Cheapest USA Elite Private Proxies 100 Private Proxies 200 Private Proxies 400 Private Proxies 1000 Private Proxies 2000 Private Proxies ExtraProxies.com - Buy Cheap Private Proxies Buy 50 Private Proxies Buy 100 Private Proxies Buy 200 Private Proxies Buy 500 Private Proxies Buy 1000 Private Proxies Buy 2000 Private Proxies ProxiesLive Proxies-free.com New Proxy Lists Every Day Proxies123