Python 3.x – How to make loops in scrapy?

I'm scraping a website from the Dmoz site. I want to make a loop in functions because the in loop that I am using in each function, I will have to put again and again in each function. Although its functionality is the same. The second thing I want to solve is to make a loop in performance response. follow because if I scraped more pages, I'll have to write this again and again. Is there any way of my two problems. I tried several times but I failed.

                                # save and call another page
performance response.follow (self.about_page, self.parse_about, meta = {& # 39; items & # 39 ;: items})
performance response.follow (self.editor, self.parse_editor, meta = {& # 39; items & # 39 ;: items})

def parse_about (auto, answer):
# do your stuff on the second page
items = response.meta['items']
        names = {& # 39; name1 & # 39 ;: & # 39; Headings & # 39 ;,
& # 39; name2 & # 39 ;: & # 39; paragraphs & # 39 ;,
& # 39; name3 & # 39 ;: & # 39; 3 projects & # 39 ;,
& # 39; name4 & # 39 ;: & # 39; About Dmoz & # 39 ;,
& # 39; name5 & # 39 ;: & # 39; languages ​​& # 39 ;,
& # 39; name6 & # 39 ;: & # 39; You can make a difference & # 39 ;,
& # 39; name7 & # 39 ;: & # 39; additional information & # 39;
}

finder = {& # 39; find1 & # 39 ;: & # 39; h2 :: text, #mainContent h1 :: text & # 39 ;,
& # 39; find2 & # 39 ;: & # 39; p :: text & # 39 ;,
& # 39; find3 & # 39 ;: & # 39; li ~ li + li b a :: text, li: nth-child (1) b a :: text & # 39 ;,
& # 39; find4 & # 39 ;: & # 39; .nav ul a :: text, li: nth-child (2) b a :: text & # 39 ;,
& # 39; find5 & # 39 ;: & # 39; .nav ~ .nav to :: text & # 39 ;,
& # 39; find6 & # 39 ;: & # 39; dd :: text, # about-submit :: text & # 39 ;,
& # 39; find7 & # 39 ;: & # 39; li :: text, # about-more-info to :: text & # 39;
}
for name, search in zip (names.values ​​(), finder.values ​​()):
articles[name] = response.css (find) .extract ()
performance items

PHP Tutorial Arrays & Loops | Promotion Forum

PHP, like many other languages, uses matrices to store information. Arrays can help keep data organized, easily retrievable and even translated into / from other languages ​​(such as JavaScript) to allow PHP to play with the user information that someone sent in a form. There are many different things we can do with the arrangements, but today we will simply discuss what they are, how to create one and how to "cycle" through them. I will show you an example that shows why the loop is an important skill that you should know in PHP or in any other language.

If you want to learn some basic PHP, you can check my PHP Tutorial for beginners that explains how to create / start a PHP file, declare variables, analyze variables (with if / else statements), display information on the page and combine strings of information together. Here we'll do a little bit of all that, so make sure you've read that tutorial first.

This tutorial will be repetitive to help you understand the matrices and loops. Enjoy!

tutorial.php

Code:

"Ghost", "age" => 25);
// we have started a matrix with a name and age value
?>

In this example we have two matrices. One of them is empty, and one of them is storing my current age and my username. As you can see, starting a matrix is ​​as easy as having its variable (like $ data) and then using the syntax training();
Within the matrix, you can do a variety of things. First we will take a look at our $ info matrix, which is a associative matrix.

tutorial.php

Code:

"Ghost", "age" => 25); // associative matrix
echo $ info["name"]; // echo the value of the name of our matrix
$ myage = $ info["age"]; // set our age value to a variable
?>

In this example we did two things. First, we echoed my name on the page. If we opened our PHP page, I would literally just say "Ghost" on the page. Age is not echoing, so it would not appear.

The reason why I made the variable $ myage was to show you that you can assign a value of matrix to a variable. This can be very useful for large arrays to prevent your script from having to read through a large array to get information. For example, if a matrix has 100 different data within it, it can be very beneficial to declare a variable for information like the one I made with $ myage if you plan to use the value much later.

Associative matrices essentially "associate" a name key with a value.
A key is what you see in the [] part of the recovery of the matrix. For example in $ info[“name”], the key is "name". In the following example, our key will be a number instead of a string.

tutorial.php

Code:

"Phantom", "age" => 25, "role" => 2); // associative matrix

$ groups = array (0 => "User", 1 => "Moderator", 2 => "Administrator);
// This is a numerical matrix.

// Let's use the string concatenation of the PHP tutorial for beginners to show some sentences:
echo $ groups[0] . "It's the paper 0 
"; echo $ groups[1] . "is the role 1
"; echo $ groups[0] . "It's paper 2
"; ?>

Therefore, a number matrix increases by 1 each time a new value is added to the matrix. We can configure the matrix as I did, or we can add a value to the matrix quite simply. Watch this:

tutorial.php

Code:

"Phantom", "age" => 25, "role" => 2); // associative matrix

$ groups = array ();
// This is an empty matrix.

$ groups[] = "User";
$ groups[] = "Moderator";
$ groups[] = "Administrator";

$ role = $ info["role"];
echo "{$ info["name"]} is a company {$ groups[$role]} ";
?>

In this last example, we are using the information of role = 2 in the $ info matrix to find the name of the function in our matrix $ groups. We are also adding the roles to our $ groups array in a different way than before. When you use $ arrayName[] , is telling PHP that everything that comes after the equal sign must be added to that matrix.

So when we use $ groups[] = "User" first, then the groups look like this:

Code:

$ groups = array (0 => "User");

So obviously we add Moderator and Administrator too.
You may also have noticed that we use { Y } Around our variables / matrix references in our sentence. This is a way to make sure that quotes and syntax do not break our string value within the original "double quotes". It is similar to just using a variable within a string, but it is an additional syntax that does not change the value of our real variable, it only allows PHP to read it without any problem.

In our previous example, our tutorials page now shows:
"Ghost is an administrator of the company."

So, now that you know how to set up an associative array ($ info), a numeric array ($ groups) and an empty array to which we can add values ​​to ($ array[] = $ newvalue), it's time to learn about one more thing … multidimensional arrays.

Multidimensional matrices are matrices with matrices inside them.
We'll see:

tutorial.php

Code:

"Ghost", "age" => 25, "role" => 2, "friends" => array ("Frank", "James", "Chris")); // associative matrix

$ groups = array ("User", "Moderator", "Administrator");
/ *
Yes, numeric arrays do not require that you include the syntax # => in front of each value
You can simply list the values ​​within the matrix, and PHP will automatically apply the keys 0, 1, 2 to our User, Moderator and Administrator values.
* /

$ role = $ info["role"];
echo "{$ info["name"]} is a company {$ groups[$role]} ";
?>

This prints "Ghost is a company administrator". in the page.

As you can see, I made some new things. I did not use the numeric keys in the $ groups array because PHP solves them by default, and I also used the name of $ role in a sentence. I did this because $ role = 2, from the $ info array and $ groups[2]= the name of the group (Administrator). Remember, although we did not supply the numeric keys, PHP automatically made $ groups[0] = "User", 1 = Moderator and 2 = Administrator.

However, the most important thing I did was include a matrix within the matrix. Our "friends" key in the $ info matrix makes our $ info array not only associative, but also a multidimensional matrix. Also, you may have noticed that the $ info[“friends”] that is your own matrix is ​​a numerical matrix.

If we wanted to meet the first friend of this user, we could do …

Code:

echo $ info["friends"][0];

This will print "Frank" on the page because that is key 0 of the friends array: the first result.

Now let's go through our matrix.
We love the loops because they are perfect for repetitive functions. Instead of executing duplicate code for each element of the array, a loop will search all the different values ​​within the array and do things with it. We will go through our array of friends to do things with each other.

tutorial.php

Code:

"Ghost", "age" => 25, "role" => 2, "friends" => array ("Frank", "James", "Chris")); // associative matrix

$ groups = array ("User", "Moderator", "Administrator");
/ *
Yes, numeric arrays do not require that you include the syntax # => in front of each value
You can simply list the values ​​within the matrix, and PHP will automatically apply the keys 0, 1, 2 to our User, Moderator and Administrator values.
* /

$ role = $ info["role"];
echo "{$ info["name"]} is a company {$ groups[$role]} 
"; // FOREACH LOOP foreach ($ info["friends"] like $ friend) { echo $ friend "is a friend of {$ info["name"]}
"; } ?>

In this example we see the following printed on the page.:
Ghost is an administrator of the company.
Frank is a friend of Ghost.
James is a friend of Ghost.
Chris is friends with Ghost.

This is because we use a foreach loop to do the same with each matrix value. We are saying "for each name in the array of friends … do something". Also, the syntax is quite simple, since foreach ($ array as $ variable) is easy to remember.

We could also change foreach to show some keys / matrix labels for values ​​like this … This adds an additional piece to our foreach statement by dividing the key KEY and VALUE. That way we can access the key # or name, and the value separately. It is useful if you need to do something like "if KEY == this, do something different". An example of this would be if you wanted to repeat everything except a matrix value, but in general, you will use most loops when you want to do the same with each value. In general, it will not pass through several matrix values ​​if all of them require a separate logic depending on the key, but there are times when it may be necessary.

Code:




foreach ($ info as $ key => $ value) {
echo "$ key";
}

This would be printed …
"name age role friends", because those are the key names in our matrix. Then, as we separate the key and the value in separate variables in our foreach, we can access both the name of the key and the value itself.

Consider the following …

Code:

"value1", "key2" => "value2");
foreach ($ arr as $ key => $ value) {
echo "The key $ is $ value 
"; } ?>

This shows:
The key1 is value1
The key2 is value2

We can also go through a matrix with a "for loop". It's my favorite type of loop, but it's a bit more complicated. I like the loops because they are very efficient and fast, but foreach can also be good. In fact, I first used foreach and I think it's a good loop to learn at the beginning because it's easy to remember.

We need to know the amount of values ​​within the matrix.

tutorial.php

Code:

"Ghost", "age" => 25, "role" => 2, "friends" => array ("Frank", "James", "Chris")); // associative matrix

$ groups = array ("User", "Moderator", "Administrator");

$ role = $ info["role"];
echo "{$ info["name"]} is a company {$ groups[$role]}, and it's {$ info["age"]year old. 
"; $ num = count ($ info["friends"]); // count the number of values ​​in the matrix info => friends for ($ x = 0; $ x <$ num; $ x ++) { echo $ info["friends"][$x] . "is a friend of {$ info["name"]}
"; } ?>

In this example we see the following printed on the page.:
Ghost is an administrator of the company, and he is 25 years old.
Frank is a friend of Ghost.
James is a friend of Ghost.
Chris is friends with Ghost.

We are going to break the for loop.
We begin by obtaining the value of $ num. It's the number of friends, which is 3.
Then we start our for loop with a new variable that we call $ x.
$ x = 0
Then we say $ x <$ num. This means that our for loop will begin with $ x equal to 0, and will be repeated as long as $ x remains less than (<) $ num or in other words, while $ x is less than 3.

Then we use $ x ++;
This tells our for loop that each time the loop is executed, we want to increase $ x by 1. When you have a number stored in a variable, using $ variable ++ is the same as saying $ variable = $ variable + 1. It simply adds 1 to itself.

Therefore, in our for loop, $ x starts at 0. Each time the loop is executed, it increases by 1, and will continue to loop while $ x is less than $ num (3). So, in this case, the first loop $ x = 0, the second $ x = 1 and the third $ x = 2. After our first 3 loops, $ x would be equal to 3, and the loop stops because our rules they say that the loop should only be executed if $ x is LESS THAN 3, it is not equal to it.

In each of our loops, we access the name of friends with:
$ info[“friends”][$x]

So, our 3 loops are accessing the friend's name in this way:
$ info[“friends”][0]
$ info[“friends”][1]
$ info[“friends”][2]

Because 0, 1, 2 are the values ​​of our variable $ x for our 3 loops
0 refers to the first friend in our $ info[“friends”] array, 1 refers to the second friend and 2 refers to the third friend (because remember, the arrays start at 0 by default for the keys)

You may wonder why we use loops. Well, the loops are perfect for when the amount of information can vary in quantity. If all people have only one age and one name, we do not need to complete the information. However, for things like friends where people can have 0 friends, 3 friends or even hundreds of friends, the loops allow us to execute code for each value in the matrix, regardless of the amount. They can even duplicate each other: a loop inside a loop. This could be taken further to have dozens of nested loops, but there are very few cases where it would be necessary. The loops are good, but they can definitely get out of control.

For large amounts of data, it is important that you do not make your script / page load forever due to a large loop. An example of this would be a forum with 1 million publications in a category. Instead of looping through all 1 million content, we would get the first 25-100 results instead and we would do it. We would pass to the other content if the user went to page 2, for example. Loops are only as good as the programmer, so make sure you're not running unnecessary loops in Big Data unless you really have to.

So let's look at a final example of our for loop, with some matrices.

tutorial.php

Code:

"Ghost", "age" => 25, "friends" => array (array ("name" => "Frank", "age" => 23), array ("name" => "Chris", "age" "=> 26), array (" name "=>" James "," age "=> 27));

$ num = count ($ info["friends"]); // count the number of values ​​in the matrix info => friends
for ($ x = 0; $ x < $num; $x++){ // loop through all the friends results
 // in this example, our friends array actually holds a bunch of separate arrays! 

 if($info["friends"][$x]["age"] > $ info["age"]) {// if our friend's age is greater than ours
$ data[] = $ info["friends"][$x]; // add this friend to the data matrix
}

} // end of for loop

print_r ($ data);
?>

In this last example we are really putting everything together.
We start with an empty matrix, $ data.
We also have our $ info matrix that stores our name, age and our friends. The friends array contains several arrangements, which have information about the friend. When we go through the friends matrix, we can access the information of each friend using the appropriate matrix key (name or age). In this example, we are trying to find all the friends who are older than us. We discovered correctly that Chris and James are older than us, so they are added to the $ data matrix.

The result would be the same as creating the $ data array in this way:

Code:

"Chris", "age" => 26), array ("name" => "James", "age" => 27));
?>

So, as you can see, you can access the data in the matrix with the loop, analyze them and then do things with that information, like building the $ data matrix separately so that only our older friends have it. Obviously, this is a very specific example, but you can do much more, including echo values ​​on the page, trigger another script to execute, or really anything you can imagine.

performance – Delete nested for loops in python

I have the following work code, but it is too slow because I have to execute the same code for several files inside a folder. Please, help me avoid the nested loop to make it more efficient.

def create_nodes (self):
for child in self.tree.getroot ():
columnNames = child.attrib
events = [grandchild.attrib for grandchild in child]
    return (columnNames, events)

def create_game_data (self):
columns = {}
for the column in self.create_nodes ()[0].llaves ():
columns[column] = []
        columnValues ​​= []
        for event in self.create_nodes ()[1]:
columnValues.append (self.create_nodes ()[0][column]) if column in self.create_nodes ()[0].keys () else columnValues.append ("NA")
columns[column] = columnValues
return columns

python: remove unnecessary loops when parsing the cable modem status table

I am writing a Python script that will be used to generate a line protocol to insert metrics into an InfluxDB. I am analyzing the status page of my cable modem. That part works. I receive the data and I can extract the table of information that I need.

The HTML for the table looks like this (yes, it has the commented lines)

The one underneath works, but feels inefficient. I feel that I have many loops and these feel unnecessary. Although I'm not sure how to be more efficient.

The only change between this and my real code is that I have moved the HTML to pastebin to reduce the length of this publication. That pastebin is here and the HTML is not written or editable by me. It is generated in the cable modem. As I can not see the results of my cable modem executing the code I use to extract it, I hope this is good enough.

from bs4 import BeautifulSoup
import requests

results_url = "https://pastebin.com/raw/bLZLFzy6"
content = requests.get (results_url) .text

measure = "modem"
hostname = "home-modem"


def strip_uom (data):
"" "Unit of measure of band

Some of our fields come with a unit of measurement. Tear that and keep it alone
The value of the field.
""
uom = ["dB", "Hz", "dBmV", "Ksym/sec"]
    data set = []
    for d in the data:
yes d.split ("")[-1] in uom:
dataset.append (d.split ("")[0])
plus:
dataset.append (d)
return data


soup = BeautifulSoup (content, & # 39; html.parser & # 39;)
dstable = soup.find (& # 39; table & # 39 ;, {& # 39; id & # 39 ;: & # 39; dsTable & # 39;})

# Pull the headings of the downstream table
# We want to make these labels friendly, so make them lowercase and eliminate spaces
dstable_tags = [td.get_text().lower().replace(" ", "_") for td in dstable.tr.find_all('td')]

# Extract the rest of the data for each row in the table; associate it with the correct label; Strip UoM
downstream_data = []
for the row in dstable.find_all (& # 39; tr & # 39;)[1:]:
column_values ​​= [col.get_text() for col in row.find_all('td')]
    downstream_data.append (dict (zip (dstable_tags, strip_uom (column_values))))

# Print line protocol lines for the inputs.exec plugin of telegraphy to manage
for the data in downstream_data:
line_protocol_line = f "{measurement}, hostname = {hostname}"
fields = []
    For key, value in data.items ():
# Check if our value is a number. If not, surround it with quotes.
# Do not use the float () value, since some numbers are returned as
# valid integers
try:
_ = float (value)
fields.append (f & # 39; {key} = {value} & # 39;)
except ValueError:
fields.append (f & # 39; {key} = "{value}" & # 39;)
fieldset = ",". join (fields)
line_protocol_line = line_protocol_line + f ", channel = {data['channel']} {fieldset} "
print (line_protocol_line)

Finally, the output generated by this script is as follows:

modem, hostname = netgear-cm400, channel = 1 channel = 1, lock_status = "Blocked", modulation = "QAM 256", channel_id = 121, frequency = 585000000, power = 5.6, snr = 37.0, correctable = 19443, not correctable = 11263
modem, hostname = netgear-cm400, channel = 2 channel = 2, lock_status = "Blocked", modulation = "QAM 256", channel_id = 6, frequency = 591000000, power = 5.7, snr = 37.0, correctable = 19531, uncorrectables = 9512
modem, hostname = netgear-cm400, channel = 3 channel = 3, lock_status = "Blocked", modulation = "QAM 256", channel_id = 7, frequency = 597000000, power = 5.7, snr = 36.8, correctable = 17457, not correctable = 9736
modem, hostname = netgear-cm400, channel = 4 channel = 4, lock_status = "Blocked", modulation = "QAM 256", channel_id = 8, frequency = 603000000, power = 5.9, snr = 37.0, correctable = 12750, not correctable = 11156
modem, hostname = netgear-cm400, channel = 5 channel = 5, lock_status = "Blocked", modulation = "QAM 256", channel_id = 122, frequency = 609000000, power = 2.6, snr = 36.3, correctable = 1855538, not correctable = 18388
modem, hostname = netgear-cm400, channel = 6 channel = 6, lock_status = "Blocked", modulation = "QAM 256", channel_id = 10, frequency = 615000000, power = 2.6, snr = 37.0, correctable = 846194, not correctable = 14615
modem, hostname = netgear-cm400, channel = 7 channel = 7, lock_status = "Blocked", modulation = "QAM 256", channel_id = 11, frequency = 621000000, power = 2.6, snr = 37.6, correctable = 281431, not correctable = 13998
modem, hostname = netgear-cm400, channel = 8 channel = 8, lock_status = "Blocked", modulation = "QAM 256", channel_id = 12, frequency = 627000000, power = 2.4, snr = 36.1, correctable = 78059, not correctable = 13695

I have loops and comprehension lists to get the labels, associate data with labels for each row and generate the line protocol lines. I have two additional understandings in those and a loop to see if I need to delete an included unit of measure. Can I eliminate some of these to be more efficient?

I am running all this code in Python 3.6.7.

Python: update of personalized steps of Scipy basinhopping and restricted loops

I'm looking for the global minimum of a given function and trying to use its gradient (here just like Jacobin) to guide the step counter. However, my X It is the solution and so is my gradient. I am also trying to recover as quickly as possible the first X for which f (x) <1, therefore I am using a restriction.

  • How can I update the X Entrance and the Jacobin?
  • My f (x) <1 It is not very effective, so is there any alternative to fulfill my requirement?

This is my code (more or less):

MyBounds class (object):
def __init __ (self, xmax =[2*np.pi, 2*np.pi, 2*np.pi, 2*np.pi, 1.2, 1.2, 1.2, 1.2], xmin =[0, 0, 0, 0, 0, 0, 0, 0] ):
self.xmax = np.array (xmax)
self.xmin = np.array (xmin)

def __call __ (self, ** kwargs):
x = kwargs["x_new"]
        tmax = bool (np.all (x <= self.xmax))
        tmin = bool(np.all(x >= self.xmin))
returns tmax and tmin

MyTakeStep class (object):
def __init __ (self, stepsize = 1):
self.stepsize = stepize

def compute_step (self, jacobi_matrix, x, i):
yes jacobi_matrix[i] < 0: r = np.random.uniform(0,      2*np.pi-x[i])
        elif jacobi_matrix[i] >    0: r = np.random.uniform (0-x[i], 0)
else: r = 0
go back

def __call __ (self, x):
print ("ENTERING FROM CALL")
print ("THIS IS X:", x)
jacobi_matrix = jacobian (x)
print ("x:", x)
print ("jacobi:", jacobi_matrix)
X[0] + = self.compute_step (jacobi_matrix, x, 0)
X[1] + = self.compute_step (jacobi_matrix, x, 1)
X[2] + = self.compute_step (jacobi_matrix, x, 2)
X[3] + = self.compute_step (jacobi_matrix, x, 3)
X[4] + = self.compute_step (jacobi_matrix, x, 4)
X[5] + = self.compute_step (jacobi_matrix, x, 5)
X[6] + = self.compute_step (jacobi_matrix, x, 6)
X[7] + = self.compute_step (jacobi_matrix, x, 7)
print ("newx:", x)
return x

def f (x):
# components of the objective function
result = g1
result + = g2
result + = g3
return result

def jacobian (x):
print ("input_list in Jacobi:", x)

# define full derivatives
dG_dphi = dg1_dphi + dg2_dphi + dg3_dphi
dG_dr = dg1_dr + dg2_dr + dg3_dr
gradient = np.hstack ((dG_dphi, dG_dr))

print ("G:", gradient.shape, gradient, " n")
return gradient

def callback (x, f, accept):
print ("x:% 65s | f:% 5s | accept:% 5s"% (str ([round(e,3) for e in x]), str (round (f, 3)), accept))

def hopping_solver (min_f, min_x, input_excitation):
# define limits
mybounds = MyBounds ()
mytakestep = MyTakeStep ()
comb = [deg2rad(phi) for phi in  input_excitation[:4]]+ input_excitation[4:]
    print ("comb:", comb)
min_f = 10
tol = 0
Cons = {& # 39; type & # 39 ;: & # 39; ineq & # 39 ;, & # 39; fun & # 39 ;: lambda x: 1-f (x)}
k = {"method": & # 39; Nelder-Mead & # 39 ;, & # 39; restrictions & # 39 ;: cons, & # 39; jac & # 39 ;: jacobian, & tollet & # 39 ;: tol }
optimal_c = optimize.basinhopping (f,
x0 = comb,
Niter = 1000000,
T = 8,
step size = 1,
minimizer_kwargs = k,
take_step = mytakestep,
accept_test = mybounds,
callback = callback,
interval = 100000,
disp = true
niter_success = None)
print (optimum_c)
min_x, min_f = optimal_c['x'], optimum_c['fun']
    comb = min_x
sol = np.array (list ([nprad2deg(phi)paraphienlalista(optimum_c[nprad2deg(phi)forphiinlist(optimal_c[nprad2deg(phi)paraphienlalista(óptimo_c[nprad2deg(phi)forphiinlist(optimal_c['x'][:4])]) + list (optimum_c['x'][4:]))
min_x = sun
returns min_x, min_f

Any help is very appreciated, thanks in advance.

Performance – VBA code optimization (loops)

I added a timer to my code and the bottleneck occurs when I traverse 47 rows and input data from a dictionary that I loaded with values.

Since I use these files for many different things, I have configured public variables to avoid configuring them for each new code.

Timer statistics:

3.4453125 Open the customer database, determine in which account we are
3.484375 Find the last 4 weeks and order range
7.31640625 Adding columns and forming
7.61328125 during the last 4 weeks, add the station space and the P & L data to the dictionary
7.6484375 Find the cumulative range, add space and P & L for 2019
100.90234375 by adding dictionary data to the order file

Anyway, below is the code …

Sub Orders_Historicals_autofilterdict ()

Dark start as double
start = timer

& # 39; & # 39; --------------------------------------
Static variables
& # 39; & # 39; --------------------------------------
Defined Calls Variables

Dim orderFile As Variant
Dim orderStart As Long
Dim orderEnd As Long

Dim clientdataFile As Variant

Dim internalize file as variant
Dim dateStart As Long
Dim station Begin as long
Dim stationEnd as long

Dim currentStation as a chain
Dim currentWeek As String

Dim dictData As New Scripting.Dictionary
Dim stationclearanceData As New Scripting.Dictionary
Dim stationplData As New Scripting.Dictionary

Dim key as variant

Dim fileOnly As String
Dim networkOnly As String

Dim i so long
Dim w so long
Dim t so long

Dim PlTotal as long
Dark clear total total while
Dim stationHash As String

& # 39; & # 39; --------------------------------------
& # 39; & # 39; Dictionary of the abbreviations of the order
& # 39; & # 39; --------------------------------------
Application.ScreenUpdating = False

Set file order = ActiveWorkbook.ActiveSheet

Notebooks of work. Open client location

Set clientdataFile = ActiveWorkbook.Sheets (dan_location) & # 39; / Change sheet when used on a different computer

clientdataFile.Activate

For i = 1 To Cells (Rows.count, 1) .End (xlUp) .row
If dictData.Exists (Cells (i, clientOrder) .Value) Then
Otherwise: dictData.Add Cells (i, clientOrder) .Value, i
It will end if
Following

& # 39; & # 39; --------------------------------------
& # 39; & # 39; Determine account / network and open internal associate with order
& # 39; & # 39; --------------------------------------
orderFile.Activate

fileOnly = ActiveWorkbook.Name

fileOnly = Left (fileOnly, InStr (fileOnly, ".") - 1)

If InStr (fileOnly, 2)> 0 Or InStr (fileOnly, 3)> 0 Then
fileOnly = Left (fileOnly, Len (fileOnly) - 1)
It will end if

networkOnly = ActiveWorkbook.Name

networkOnly = Mid (networkOnly, InStr (networkOnly, "IO.") + 3)

networkOnly = Left (networkOnly, InStr (networkOnly, ".") - 1)

Workbooks.Open Filename: = clientdataFile.Cells (dictData (fileOnly), clientInternal) .Value

Set internalFile = ActiveWorkbook

internalFile.Sheets (WT_newWeek) .Activate

Debug.Print Timer - start and "Open client database, determine what account we are in"
& # 39; & # 39; --------------------------------------
& # 39; & # 39; Search last 4 dates and column header for orders
& # 39; & # 39; --------------------------------------
For i = 1 to 700
If Cells (i, 1) = WT_newWeek Then
dateStart = i
ElseIf cells (i, 1) = "Station" Then
stationStart = i + 1
Output for
It will end if
Following

For i = stationStart To 700
If the cells (i, 1) .Value = Cells (stationStart - 2, 1) .Value and "Total" then
stationEnd = i - 1
Output for
It will end if
Following

orderFile.Activate

For i = 1 to 700
If the cells (i, 1) = "Station" then
orderStart = i + 1
Output for
It will end if
Following

For i = orderStart To 700
If Len (Cells (i, 1)) = 0 And Len (Cells (i - 1, 1)) = 0 And Len (Cells (i - 2, 1)) = 0 Then
orderEnd = i - 3
Output for
It will end if
Following


Debug.Print Timer - start & "Find last 4 weeks and order range"

& # 39; & # 39; --------------------------------------
& # 39; & # 39; Add dates to order header and format
& # 39; & # 39; --------------------------------------

Cells (orderStart - 1, OF_buyAlgoColumn) = "Algorithm recommendation"
Cells (orderStart - 1, OF_totalplColumn) = "Total P & L"
Cells (orderStart - 1, OF_totalclearanceColumn) = "% of total settlement"

Cells (orderStart - 1, OF_clearanceColumn1) = internalFile.Sheets (WT_newWeek). Cells (dateStart, 1)
Cells (orderStart - 1, OF_clearanceColumn2) = internalFile.Sheets (WT_newWeek). Cells (dateStart - 1, 1)
Cells (orderStart - 1, OF_clearanceColumn3) = internalFile.Sheets (WT_newWeek). Cells (dateStart - 2, 1)
Cells (orderStart - 1, OF_clearanceColumn4) = internalFile.Sheets (WT_newWeek). Cells (dateStart - 3, 1)

Cells (orderStart - 1, OF_plColumn1) = internalFile.Sheets (WT_newWeek). Cells (dateStart, 1)
Cells (orderStart - 1, OF_plColumn2) = internalFile.Sheets (WT_newWeek). Cells (dateStart - 1, 1)
Cells (orderStart - 1, OF_plColumn3) = internalFile.Sheets (WT_newWeek). Cells (dateStart - 2, 1)
Cells (orderStart - 1, OF_plColumn4) = internalFile.Sheets (WT_newWeek). Cells (dateStart - 3, 1)

Range (Cells (orderStart - 2, OF_clearanceColumn1), Cells (orderStart - 2, OF_clearanceColumn4)) = "Settlement"
Range (Cells (orderStart - 2, OF_plColumn1), Cells (orderStart - 2, OF_plColumn4)) = "P & L"

Cells (orderStart - 1, OF_stationColumn) .Copy
Range (Cells (orderStart - 1, OF_buyAlgoColumn), Cells (orderStart - 1, OF_plColumn4)). PasteSpecial xlPasteFormats

Cells (orderStart, OF_stationColumn) .Copy
Range (Cells (orderStart - 2, OF_clearanceColumn1), Cells (orderStart - 2, OF_plColumn4)). PasteSpecial xlPasteFormats

Range (Cells (orderStart - 2, OF_buyAlgoColumn), Cells (orderEnd, OF_plColumn4)). HorizontalAlignment = xlCenter

Cells (orderStart, OF_stationColumn) .Copy
Range (Cells (orderStart, OF_buyAlgoColumn), Cells (orderEnd, OF_plColumn4)). PasteSpecial xlPasteFormats

Cells (orderStart, OF_totalColumn) .Copy
Range (Cells (orderStart, OF_plColumn1), Cells (orderEnd, OF_plColumn4)). PasteSpecial xlPasteFormats
Range (Cells (orderStart, OF_totalplColumn), Cells (orderEnd, OF_totalplColumn)). PasteSpecial xlPasteFormats

Range (Cells (orderStart, OF_totalclearanceCearance Column), Cells (orderEnd, OF_clearanceColumn4)). NumberFormat = "0%"

Range (Cells (orderStart - 2, OF_buyAlgoColumn), Cells (orderEnd, OF_plColumn4)). FormatConditions.Delete

Range (Columns (OF_buyAlgoColumn), Columns (OF_plColumn4)). AutoFit

Debug.Print Timer - start and "Add columns and format"

& # 39; & # 39; --------------------------------------
& # 39; & # 39; Add space and P & L by date to the dictionary
& # 39; & # 39; --------------------------------------

For i = OF_clearanceColumn1 To OF_clearanceColumn4

currentWeek = Cells (orderStart - 1, i) .Value

internalFile.Sheets (currentWeek) .Activate

For t = 1 to 700

If the cells (t, 1) = "Station" then
StationStart = t + 1
Output for
It will end if
Following

For t = stationStart To 700

If the cells (t, 1) .Value = Cells (stationStart - 2, 1) .Value and "Total" then
stationEnd = i - 1
Output for
It will end if

If stationclearanceData.Exists (Cells (t, WT_stationColumn) .Value & currentWeek) Then
Plus:
In error, resume next
stationclearanceData.Add Cells (t, WT_stationColumn) .Value & currentWeek, Cells (t, WT_mediaactColumn) .Value / Cells (t, WT_mediaestColumn) .Value
stationplData.Add Cells (t, WT_stationColumn) .Value & currentWeek, Cells (t, WT_profitColumn) .Value
It will end if

Following

orderFile.Activate

Following

Debug.Print Timer - start and "go through the last 4 weeks, add free station space and P & L data to the dictionary"

& # 39; & # 39; --------------------------------------
& # 39; & # 39; Add cumulative settlement and P & L to the dictionary
& # 39; & # 39; --------------------------------------

internalFile.Sheets ("Cumulative"). Activate

For t = 5 to 70000

If Cells (t, 1) = "" and Cells (t + 1, 1) = "" and Cells (t + 2, 1) = "" Then
stationEnd = t + 1
Output for
It will end if
Following

For t = 5 to park

If the cells (t, CT_yearColumn) = 2019 then

If stationclearanceData.Exists (Cells (t, CT_hashColumn) .Value) Then
Plus:
In error, resume next
stationclearanceData.Add Cells (t, CT_hashColumn) .Value, Cells (t, CT_clearanceColumn) .Value
stationplData.Add Cells (t, CT_hashColumn) .Value, Cells (t, CT_invoiceColumn) .Value - Cells (t, CT_actcostColumn) .Value
It will end if
It will end if
Following

Debug.Print Timer - start and "find cumulative range, add space and P & L for 2019"


orderFile.Activate

& # 39; & # 39; --------------------------------------
& # 39; & # 39; Go through the stations in the order file and update according to the dictionary values
& # 39; & # 39; --------------------------------------

For w = orderStart To orderEnd

If the cells (w, OF_stationColumn) <> "" then

If Cells (w, OF_stationColumn) <> Cells (w - 1, OF_stationColumn) Then

stationHash = Cells (w, OF_stationColumn) .Value & "" & Cells (w, OF_trafficColumn) .Value & "Total"

In error, resume next
Cells (w, OF_clearanceColumn1) = stationclearanceData (Cells (w, OF_stationColumn) .Value & Cells (orderStart - 1, OF_clearanceColumn1) .Value)
Cells (w, OF_clearanceColumn2) = stationclearanceData (Cells (w, OF_stationColumn) .Value & Cells (orderStart - 1, OF_clearanceColumn2) .Value)
Cells (w, OF_clearanceColumn3) = stationclearanceData (Cells (w, OF_stationColumn) .Value & Cells (orderStart - 1, OF_clearanceColumn3) .Value)
Cells (w, OF_clearanceColumn4) = stationclearanceData (Cells (w, OF_stationColumn) .Value & Cells (orderStart - 1, OF_clearanceColumn4) .Value)

Cells (w, OF_plColumn1) = stationplData (Cells (w, OF_stationColumn) .Value & Cells (orderStart - 1, OF_plColumn1) .Value)
Cells (w, OF_plColumn2) = stationplData (Cells (w, OF_stationColumn) .Value & Cells (orderStart - 1, OF_plColumn2) .Value)
Cells (w, OF_plColumn3) = stationplData (Cells (w, OF_stationColumn) .Value & Cells (orderStart - 1, OF_plColumn3) .Value)
Cells (w, OF_plColumn4) = stationplData (Cells (w, OF_stationColumn) .Value & Cells (orderStart - 1, OF_plColumn4) .Value)

Cells (w, OF_totalplColumn) = stationplData (stationHash)
Cells (w, OF_totalclearanceColumn) = stationclearanceData (stationHash)

It will end if

It will end if

Following

Debug.Print Timer - start and "add dictionary data to the order file"

clientdataFile.Activate

ActiveWorkbook.Close saveChanges: = False

Application.ScreenUpdating = True

Range (Cells (orderStart - 2, OF_buyAlgoColumn), Cells (orderEnd, OF_plColumn4)). HorizontalAlignment = xlCenter

MsgBox ("Complete purchase algorithm")


End Sub

architecture – What is the best way to scale heavy for loops in the Flask application in AWS?

The service is to process the file that can contain 10 to 50 pages, what is the most recommended way to scale the for loop that I made on each page? Spark? Message queue? AWS Lambda (actually I'm not so sure about this and how it could be used with the Elastic Container Service) or maybe other simpler solutions.

Algorithms: Can all problems $ O (n) $ be solved without nested loops?

There are examples of implementations of algorithms that contain nested loops but of complexity O (n), and some of them have corresponding implementations that do not contain nested loops. So, here's a question: can all these implementations be simplified or converted to an implementation with only higher-layer loops? Namely, they can all problems that have a $ O (n) $ Can the algorithm be solved with an algorithm without nested loops?

Linear algebra – Loops in Matlab to solve a random walk problem

The matlab exercise says "Suppose now that you are playing the game with 100 coins (50 coins for each player to start) with a loaded die, so that the probability of the dice falling to an even number is 0.53 instead of 0.5 .win a coin on a roll.) Write a for loop to create the transition matrix P and store this matrix as OneEmat. "

What is the loop supposed to do? How should I start the loop? How do I get a loop to create a matrix?

Any advice would be appreciated since I am pretty lost on how to start.

While the loops continue until a user has entered a blank line (for example: "")

Remove this question

My effort to solve the problem:

while false:
yes number_entered! = "":
number_entered = float (number_entered)

number = true