parallelization – Prevent resurrection of dead subkernels

How do I prevent Mathematica from trying to resurrect subkernels that it has deemed to be dead? I would like to avoid this:

Kernels::rdead:
   Subkernel connected through KernelObject[3, *compute node*] appears dead. 

Parallel`Developer`QueueRun::req:
   Requeueing evaluations {6} assigned to
    KernelObject[3, *compute node*, <defunct>].

LaunchKernels::clone:
   Kernel KernelObject[3, *compute node*, <defunct>] resurrected as
    KernelObject[9, *compute node*].

I would prefer it if Mathematica aborts the entire computation if a subkernel dies, or continues on with the remaining subkernels. If a subkernel dies it’s most likely because it ran out of memory, so there is little sense in relaunching it.

parallelization – Evaluating values from a indexed ParametricNDSolve solution in parallel

I have a ParametricNDsolve solution for a differential equation for a huge system given by s1. I have no problem in evaluating the value using the solution. It’s, in fact, speeding up the process using Parallel cores.

I have represented the variables by a, b, c and d, some of which are indexed by j, and the parameters are g, n1and n2.
My attempt is to create a table of all the required variables and replace s1 to them as in Tab1and finally evaluate t shown by Tab2

Tab1 = Table( Table(Table({a(j, t)(g, n1, n2), b(j, t)(g, n1, n2), 
   c(g, n1, n2)(t), d(g, n1, n2)(t)}, {j, 1, 2}), {g, 1, 3, 1}), {n1, 1, 4}, {n2, 1, 4}) /. s1; (*first step*)

Tab2=Tab1/.t->12 (say)

So,Tab1 is only a replacement of the solution s1 to the given variables a, b, c and d, which takes me more time to evaluate than Tab2. Is there any way I can parallel map or parallelEvaluate Tab1 using all cores available such that s1 is mapped to the table of variables simultaneously in parallel

parallelization – Tracking progress of ParallelTable Computations

I have a notebook where I work with large datasets, where the data is stored as an A x B array of signals, where each signal is often 1000 samples long. In the past I would track the progress of various filters by having a ‘counter’ variable incrementing and using ProgressIndicator to monitor the progress.

With larger datasets I’m able to make huge improvements with ParallelTable, but as everyone knows using SetSharedVariable kills the computation speed, so I’m losing the ability to monitor the progress (some routines take 30 minutes or more to complete). Is there some other way to track the progress of ParallelTable aside from my method?

Below is a simple Table based example of my code:

Make a 100 x 100 array of 1000 sample length time signals.

counter = 0;
data = Table( Sin(10 2 Pi t^2), {100}, {100}, {t, 0, 1, 0.001});
Dynamic@ProgressIndicator( Dynamic@counter, {1, 100*100})
t0=AbsoluteTime();
dataF = Table( counter++; 
BandpassFilter( data((i, j)), {10*2*Pi, 30*2*Pi}, SampleRate -> 1000), {i, 100}, {j, 100});
AbsoluteTime() - t0

The above code completes on my machine in about 8 seconds. Simply changing to ParallelTable reduces down to under 2 seconds. Utilizing the following code works, but it slows down considerably.

counter = 0;
SetSharedVariable@counter;
data = Table( Sin(10 2 Pi t^2), {100}, {100}, {t, 0, 1, 0.001});
Dynamic@ProgressIndicator( Dynamic@counter, {1, 100*100})
t0=AbsoluteTime();
dataF = ParallelTable( counter++; 
BandpassFilter( data((i, j)), {10*2*Pi, 30*2*Pi}, SampleRate -> 1000), {i, 100}, {j, 100});
AbsoluteTime() - t0

parallelization – Parallelize import of many files

I’m trying to speed up the import of a large number of files, using ParallelTable to store them in a indexed variable, eqDump. The files are in the folder “Frames” and are named “Conf_Run.1000”, “Conf_Run.2000”, … Here is what I’ve tried,

    Clear[eqDump];    
    SetSharedFunction[eqDump];
    ParallelTable[
      eqDump
                    ,"Table"
                    ,HeaderLines -> 9]
    ,{t, 5000, 1000000, 5000}];

But the execution doesn’t even seems to start, the kernels remain idle. I don’t know whats happening, since I think it should work in the same way as here for example. I’ve tried also to SetSharedVariable

Thank you very much!

parallelization – How to close a specific Kernel after completion of evaluation in ParallelDo

I have a particular code that I have to run over several different parameters. I am using ParallelDo to parallelize the process. Due to the nature of the evaluations some Kernels finish the evaluations much before the others. After that I print the data generated by that Kernel to a file and the Kernel becomes idle. Is there any way to free up the Kernel so that I can use it for a future evaluation?

If you are curious about why some Kernels finish the work before others, it’s because I have to solve a differential equation for a fixed number of time periods, say N, but the parameter is the frequency, hence some time periods are much larger than the other and hence take longer to evaluate.

parallelization – How to use Reap and Sow with ParallelDo

Consider the two cases:

ClearAll["Global`*"]
j = Reap[Do[{Sow[{i, i}, a]}; Sow[{i, i^2}, b], {i, 100}], _, Rule];
a /. Last@j
h = Reap[ParallelDo[{Sow[{i, i}, a]}; Sow[{i, i^2}, b], {i, 100}], _, 
   Rule];
a /. Last@h

The output of the first is an array, while the output of the second is a. Obviously i'm using ParallelDo, Sow Y Reap on the wrong path

What is the correct way to use Reap Y Sow with a ParallelDo?

MATLink Parallelization: call user-defined MATLAB functions

I am trying to run parallel simulations of finite elements in Mathematica using the AceGen and AceFEM software along with a MATLAB script called from Mathematica.

I can run some of the simulations that do not require the MATLAB script in parallel using the pseudo approach below.

(*Launch multiple kernels*)
Get("AceFEM`");
kernels = ParallelEvaluate(
Get("AceFEM`Remote`");
$KernelID)

(*Solve FEM problem*)
Output=ParallelTable(
   
   {output1,output2}
   ,{problem,ListOfFEMProblems});

For problems that require the MATLAB script, I tried to modify the code to load MATLink and the MATLAB script in all cores in two different ways.

Focus 1:

(*Launch multiple kernels*)
Get("AceFEM`");
kernels = ParallelEvaluate(
Get("AceFEM`Remote`");
$KernelID)

(*Load MATLink*)
Needs("MATLink`");
OpenMATLAB();
MatlabFunction = MFunction("MatlabFunctionName", "OutputArguments" -> 2);

(*Solve FEM problem*)
Output=ParallelTable(
   {MATLABout1,MATLABout2}=MatlabFunction(Param1,Param2,Param3,Param4);
   
   {output1,output2}
   ,{problem,ListOfFEMProblems});

With the following error:

(kernel 4) Set::shape :  Lists {MATLABout1,MATLABout2} and     MATLink`MFunction(MatlabFunctionName,OutputArguments->2)(Param1,Param2,Param3,Param4) are not the same shape.

Focus 2:

(*Launch multiple kernels*)
Get("AceFEM`");
kernels = ParallelEvaluate(
Get("AceFEM`Remote`");
$KernelID)

(*Load MATLink*)
ParallelNeeds("MATLink`");
ParallelEvaluate(
   OpenMATLAB();
   MatlabFunction = MFunction("MatlabFunctionName", "OutputArguments" -> 2);
   );

(*Solve FEM problem*)
Output=ParallelTable(
   {MATLABout1,MATLABout2}=MatlabFunction(Param1,Param2,Param3,Param4);
   
   {output1,output2}
   ,{problem,ListOfFEMProblems});

With the mistakes:

(kernel 1) Global`OpenMATLAB::shdw :  Symbol OpenMATLAB appears in multiple contexts {Global`,MATLink`}; definitions in context Global` may shadow or be shadowed by other definitions.
(kernel 1) Global`MFunction::shdw :  Symbol MFunction appears in multiple contexts {Global`,MATLink`}; definitions in context Global` may shadow or be shadowed by other definitions.

Appears for each of the 4 kernals. As well as what happened previously Set::shape error.

(For clarity, calling the MATLAB function works perfectly for non-parallel problems)

What is the correct approach to MATLink parallelization?

Parallelization: is it possible to generate iterators dynamically during parallel processing?

I have a parallel processing code written in Wolfram Mathematica, similar to:

ParallelDo(
 DoSomething(i),
 {i, Select(Range(Power(2, k) - 1), # >= IntegerReverse(#, 2) &)}
)

My problem is that exit from Select(Range()) It can be quite large, and Mathematica seems to first generate a complete list and then start iterating; I would like to avoid this to save memory and time. Basically, what I'm looking for is something similar to Python generators, that is, generating the iterator dynamically, only when necessary.

What is some (preferably simple) way to achieve this? In other languages, I would do it by doing the logic of parallelization by myself, but I would like to make the most of what Mathematica already provides.

parallelization – Parallelize returns without evaluating

I am not sure why this simple command cannot be executed in parallel.

Parallelize[ Integrate[ Sin[x], x] 
+  Integrate[ Tan[x], x] ]

Error message:

Parallel :: nopar1: [Integral] Sin [x] [Differential D] x + [Integral] Tan [x] [Differential D] x cannot be parallelized; proceed with the sequential evaluation.

c # – Should parallelization be reserved for more important system workflows?

I wonder, in general, if a part of a program can be safely paralleled with significant performance gains, should we? Or should we consider the importance of this work because it can affect scalability? What is your thinking process when you decide to parallelize something?

For example, in my application, users can send a graphical data structure to our server for processing. Each node can be independently processed in parallel and is IO Work without blocking. On average, users will send 3 to 4 nodes, but they could be up to 100. On average, a node takes about a second to process. This is a high traffic area, but performance is not critical since the user needs an immediate response.

Parallelization in this case can accelerate it dramatically. But my concern is because there is a lot of traffic and there may be many nodes, it could potentially monopolize many resources, slowing down other areas. But I also heard an argument that it should be parallel anyway because the operating system is smart enough to manage all threads and programming.

Would you paralyze the previous scenario? Would your answer change if the job became an IO block?