I have an existing jenkins job that starts a shell script to copy my prod environment into qa.
We added a lot of data to prod (gzip dump went from 2gig to 15gig) and, suddenly, my jobs in jenkins started to fail.
We are running postgres 9.5 in aws and jenkins 2,171. all jenkins jobs are executed in the master, which is the same server with 6 executors. No memory problems / cpu / disk space
I tried some things:
statement_timeout In the instance of Postgres it is already
0. Going from bash to sh for some reason helped in some scripts but not in others. In particular, it still has several psql statements removed. The script works well when it is executed from an interactive shell.
We also tried disabling Process Tree Killer https://wiki.jenkins.io/display/JENKINS/ProcessTreeKiller. do not go.
Here is the code of two of the most innocuous commands that should be executed fairly quickly.
$ POSTGRES_HOST_OPTS It only has the name of the database and the port:
echo -e "Executing the POSTGIS command" psql $ POSTGRES_HOST_OPTS -U $ POSTGRES_ENV_POSTGRES_USER_PROD -d postgres -c "CREATE EXTENSION postgis;" echo -e "Creating the temporary user dv3_qa_tmp so that we can change the name of $ POSTGRES_ENV_POSTGRES_USER_PROD user n" psql $ POSTGRES_HOST_OPTS -U $ POSTGRES_ENV_POSTGRES_USER_PROD -d postgres -c "create role dv3_qa_tmp password & # 39; $ PGPASSWORD_QA & # 39; createdb createrole heredit login;"
Here is the jenkins console output:
Waiting for the new instance to be available ... -e Renaming the dv3_prod database to dv3_qa Delicate -e Executing the POSTGIS command Delicate -e Creating the temporary user dv3_qa_tmp so that we can change the user name dv3_prod_user Delicate -e Rename the user dv3_prod_user to dv3_qa_user Delicate Delicate -my All ready
From the jenkins.log file there is something in the file descriptors, but I'm not sure how that relates. I have also tried to redirect stderr that removes this message but does not prevent the commands from being killed.
April 10, 2019 16:23:31 hudson.Proc $ LocalProc join WARNING: Process the filtered file descriptors. See https://jenkins.io/redirect/troubleshooting/process-leaked-file-descriptors for more information. java.lang.Exception in hudson.Proc $ LocalProc.join (Proc.java:334) in hudson.tasks.CommandInterpreter.join (CommandInterpreter.java:155) in hudson.tasks.CommandInterpreter.perform (CommandInterpreter.java:109) in hudson.tasks.CommandInterpreter.perform (CommandInterpreter.java:66) in hudson.tasks.BuildStepMonitor $ 1.perform (BuildStepMonitor.java:20) in hudson.model.AbstractBuild $ AbstractBuildExecution.perform (AbstractBuild.java:741) in hudson.model.Build $ BuildExecution.build (Build.java:206) in hudson.model.Build $ BuildExecution.doRun (Build.java:163) in hudson.model.AbstractBuild $ AbstractBuildExecution.run (AbstractBuild.java:504) in hudson.model.Run.execute (Run.java:1818) in hudson.model.FreeStyleBuild.run (FreeStyleBuild.java:43) in hudson.model.ResourceController.execute (ResourceController.java:97) in hudson.model.Executor.run (Executor.java:429)