Create blue devops pipeline – SharePoint Stack Exchange

We can create a compilation pipe using yaml. But how to call this yaml to create build pipleine in blue devops? Are we supposed to navigate to dev.azure.com and manually enter the yaml file?

I've spent hours looking for a way to automate the creation of compilation pipes, but I couldn't find anything. Any pointer would be greatly appreciated.

c ++ – The signal handler does not work in the open process through the pipeline

Note: this is a general question (not limited to Wolfram command, and you do not have to know about Wolfram).

I consider wrapping Wolfram (CUI calculator as a python interpreter). Without packaging, SIGINT causes the program to suspend the calculation.

$ / usr / bin / wolfram -noinit

In[1] : = While[True, 1] #start infinite loop
#execute `kill -SIGINT` from another shell to send SIGINT
Interrupt> # calculation now interrupted (you can cancel or resume it)

However, with a wrap, SIGINT Causes Wolfram to leave right now.

$ ./wrapper.out

In[1] : = While[True, 1] #start infinite loop
#execute `kill -SIGINT` (the objective is not the wrapper but the` wolfram` itself)
Trapped SIGPIPE. # `wolfram` comes out and the envelope receives SIGPIPE

The complete code of the envelope is here. (The actual code is written better without exit (0) and with a smart pointer, but the simplified code below still causes the problem.)

using namespace std;
#include 
#include 

void signal_handler (int signal) {
yes (signal == SIGINT) {
cout << "Caught SIGINT.  n";
} else {
cout << "Caught SIGPIPE.  n";
}
exit (0); // it's not good since the destroyers are not called
}

int main (int argc, char ** argv) {

// set a signal handler
signal (SIGINT, signal_handler);
signal (SIGPIPE, signal_handler);

FILE * pipe = popen ("/ usr / bin / wolfram -noinit", "w");

while (true) {

chain buf
getline (cin, buf);

fprintf (pipe, "% s  n", buf.c_str ());
fflush (tube);

}

pclose (tube);

}

Why, with a wrapper, is the signal handler of the pipelined process ignored? And how can I maintain the functionality of the original signal driver? 3 popen man he did not give me any clue

.net – Microservices architecture for transformation pipeline / data ingestion project

I am working on the design of a new Data Ingestion Pipeline with the highlights of the new project:

  • Download and update data to / from SharePoint using the SharePoint APIs
  • Download and update data from / to the incident management application / JIRA using the JIRA APIs
  • Download and update data to / from SQL sources using the provided APIs
  • Download and update data to / from external custom applications using APIs

I am considering the architecture of microservices for the previous project, where I will seek to create 4 separate services for each of the above purposes.

And finally, a batch processing client that would run all these API services using C # .NET

But I've been wondering if the implementation of the microservices architecture will be excessive, and all I really need is for a single client to call all of these APIs directly without having to create superior services to the individual ones.

And, in addition, regarding the configuration of the project in Visual Studio, all these services must be in their own separate solutions or, rather, be part of a .NET solution with multiple projects in it.

python: processing and adjusting mixed data types using sklearn Pipeline and ColumnTransformer

I am using the ColumnTransformer in sklearn as part of a pipeline for the multiclassification using mixed data types but obtaining an error during the adjustment of the model.

The general idea is in line with the third suggestion in the main response of this publication. The approach involves a word bag model that provides predicted probabilities introduced in a second classifier with another characteristic ("stacking"). But I also want to average the missing fill values ​​for this numerical characteristic. I have based my code on a hybrid of these examples sklearn one two.

I have been able to obtain the processing of the two types of data working in isolation using the code that corresponds directly to that of those solutions. However, it seems to go wrong when I try to process the two types of data together.

Text data processing only works if I adapt my 2-function training data as a 2d set, which then becomes two separate 1d arrays during column processing (I guess this is because there is only one column for each type of data). While numeric data processing only works if I adapt my 2-feature data as a data frame, which are then converted into 2 separate Pandas series during column processing (or equally works if the processing is in a 2D matrix) .

From what I've read, Tranformers generally only work with 2d matrices and I guess this includes the Transformer Function. So my idea instead was to use Transformer Function to convert the Pandas Series of the text data type to a 1d matrix before vectoring, etc. and then return the predicted odds as a Pandas Series.

The data looks like this.

                        age tag free_text
index
0 "the example text 1 is great" 52 1
1 "Sample text 2 is impressive" NaN 0
2 "Sample text 3 is amazing" 26 0
... ..... ... ..
... .... ... ..

Here is my code

import pandas like pd
import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import FunctionTransformer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

def to_series (X):
back pd.Series (X)

def to_1d_array (X):
returns X.values

pipeline = Pipeline ([

    ('union', ColumnTransformer(
        [
            #Pipeline for standard bag-of-words model for body (first column)
            ('text', Pipeline([('1D array', FunctionTransformer(to_1d_array)),
('vect', CountVectorizer()),
                ('tfidf', TfidfTransformer()),
                ('clf', ClassifierWrapper(MultinomialNB())),
 ('Back to Series', FunctionTransformer(to_series))]) ["free_text"])

# Pipe for filling age (second column)
(& # 39; age & # 39 ;, Pipeline (steps =[('imputer', SimpleImputer(strategy='mean'))])
["age"]),],

# Weight components in FeatureUnion
transformer_weights = {
& # 39; Text 1,
& # 39; num & # 39 ;: 1,
}
)),

# Use an SVC classifier in the combined characteristics
(& # 39; svc & # 39 ;, SVC (kernel = & # 39; linear & # 39;)),])

pipeline.fit_transfrom (data[["free_text", "age"]], data.label.values)

Instead of the model fitting well, the error message appears:
ValueError: the string could not be floated:

According to the error message, it may not be working because the original data type of the column was str and it is being replaced by a float in the form of this predicted probability. I also tried the opposite: convert 1D matrices into 2d matrices and then reuse FunctionTransformer, but as expected it did not work.

Ultimately, you might be overestimating what is possible with ColumnTransformer …..

powershell: Get-SPSite pipeline – SharePoint Stack Exchange

I'm trying to run the following on a very large farm that has more than ten thousand site collections:

Get-SPSite -Limit All | Write-Host $ _. Url

I know that with so many site collections, it would take a long time to list them all, but I was hoping that the output would at least begin to show through the channeling. Instead, it seems that everything is stagnating why Get-SPSite try to load everything in some kind of internal matrix or list before sending objects through the pipeline.

Is there any way to force the pipeline to happen?

unity – LW ​​Render Pipeline does not represent the area, the spotlights, the indirect lighting?

Unity build 2019.1.4f1

I've been working on a scene for quite some time and I will not bore you with all the things I've tried in the past. Currently, I am using the lightweight processing pipe for an indoor scene. The room is covered with a mesh that is supposed to allow only directional light through the windows. It is assumed that the 4 panels of the window allow this light to pass through them. There are 13 area lights that are supposed to emit additional light from the windows, but have no effect. The same goes for 16 spotlights near the ceiling, I can not see them at all, even if I pick them up. It seems that I can only get direct light in this scene.

Why is this? What am I not doing? Why only directional light seems to work, and indirect lighting does not exist? Why is my mixed lighting setup blocked in "subtractive", is it relevant?

All the objects are configured in some variation of Static, and I am pretty sure that all my lights are set to "baked". I have stuck with the configuration again and again and I have run out of ideas. Can anyone look at my project and help me understand why lighting is so terrible? Attached is a link to the postcode of the project and some images that imply how the scene should be viewed.

This first image is one of the many test versions of Blender, and most of what I'm looking for (although it's a bit too dark and the direct light from the windows is not in the scene):
enter the description of the image here

Here is my reference material:

enter the description of the image here

..and here's the bad (dark and low resolution) that was my last Unity bake:

enter the description of the image here

Here are my lighting settings:

enter the description of the image here
enter the description of the image here

Here is the link to my project files:

Project directory

Can someone please give me some feedback on how I can make the lights work properly? Am I using the wrong pipe, or am I doing something incorrectly?

Thank you for your time and contribution, it is very appreciated!

ETL Pipeline Design – Software Engineering Exchange Stack

I am not an intelligent architecture engineer, but I have been entrusted with the task of building our ETL pipeline at work. I am an automatic learning engineer. I have never connected any infrastructure. I would love to learn, but it is discouraging.

The following is not a thing for the job, but for my understanding of the ETL.

Perform an automated process that moves data from an S3 group to a data analysis database in a useful way. Analysts should enrich the data before using it (think about adding aggregate statistics).

Using the services of Azure, we channel the data from S3 to the database through the Data Factory. We also aggregate statistics added here before inserting them into the database.

It's that easy? Should there be more threads here? Is it better to keep a replica of the data in a table and then create our aggregated data enriched in a different table?

I am especially confused about the purpose of ETL and the scope of the transformations that you can apply with ETL.

I am open to all comments, and criticisms here. Everything helps me learn this.

database: configuration of the pipeline to analyze the data stored in the web application DB

Background:

  • So there is a web application (Ruby) with a Postgres DB production (hosted in the cloud)
  • I would like to run some machine learning algorithms in a Python configuration on the production data and (ultimately) implement the model in a production configuration (in the cloud)
  • I just know how to run these algorithms locally on, say, a Numpy array that fits in memory and assuming training data is fixed
  • Let's say that the data set of interest would, in the end, be too large to fit into memory, so the data must be accessed in batches.

My general question is:

What is a good way to configure the pipeline to run the algorithms in the production data?

To be more specific, here is my current reasoning, which may or may not make sense, with more specific questions:

  • Taking into account that the algorithms will need to access the data again and again, the reading speed will be quite important. We can not afford to access it through the network and we can not continue to consult the production database of the web application. What is the best way to store the data and make it available to the machine learning algorithms to process? Copy everything to another relational DB that the Python code can access locally?

  • Finding the correct model is probably easier if it is done locally on a sample of the data that fit in the memory. Once a good candidate is found, we can retrain with all the data we have. Should we do the second step locally as well? Or should you generally try to set up a complete production line that allows you to work with a greater amount of data at this stage already?

  • Let's say you have new data written regularly. If you do the initial training visiting lots of the data you have at the time 0and, after stopping the training, you will probably have to retrain it from scratch using all the information you have later. t? Is retraining something reasonable to automate production?

The suggestions and general sources that help with this type of questions are appreciated.

workflows – Microservices: pipeline flow in which each task is in a different domain

I am creating a workflow composed of several microservices. The domains are divided into different topics. In the example, I can define it as:

  • Event processing domain
  • Domain of configurations
  • Stock inventory domain

Each domain contains its own private database, public APIs, etc. and eventually it runs in Kubernetes.

Responsibilities:

  • Event processing domain: to define the complete workflow of the event life cycle
  • Configuration domain: all events must be compared with a complex configuration mechanism, and decide if this event should be discarded or continue the channeling
  • Domain inventory inventory: for each event, we need to add information from this inventory.

The flow is composed like this:
enter the description of the image here

I am trying to follow several patterns to achieve that:

  • Architecture directed by events, but with commands, through the ESB.
  • The entire workflow is defined in the event processing domain, but the tasks themselves are defined in different domains (since they have access to your database, etc.)

https://www.infoq.com/articles/events-workflow-automation

My main concerns are:

-I know there are some channeling tools to handle those flows (Data Flow / Flink / etc.), but the problem is that all the workflow and the jars must be handled and written in a great tool, which causes 2 problems Main:

  • Responsibilities: each team that is responsible for the domain now has to implement it outside its domain, and without a database and more access to the data, since it is executed in a different context
  • A small monolith is created, all teams must implement and write code in this central

– Those services really manipulate the data that is received, it is not a simple command (do this, to that), so it fits more to a pipe service elsewhere.

I'm trying to choose the best strategy.

  1. A pipe that must be defined / changed easily (add more conditions, change the order, etc.)
  2. Avoid direct communication from service to service and use event-driven architecture
  3. The most challenging problem: the fact that each microservice develops independently

2 tools that I am reviewing –

Any discussion would be appreciated 🙂

Does Hstore and jsonb have the same pipeline internally?

Is hstore implemented identically as jsonb, only with a restriction that the data must be flat? If not, when / why should hstore be used instead of jsonb?