delete all jenkins jobs and configurations
only way I see to do it is manually by going into /Users/Shared/Jenkins and killing everything in jobs and configs.
smtp.rb:966:in `check_auth_response’: 535-5.7.1 Please log in with your web browser and then try again. Learn more at (Net::SMTPAuthenticationError)
Recipe for parallelizing pipeline runs of integration tests on multi-node Jenkins
Recipe for parallelizing pipeline runs of integration tests on multi-node Jenkins
Goal: We want to run our test pipeline in parallel (not our tests, our pipeline)
Reason: The whole pipeline takes more than half an hour, and the changes will queue up to run through the pipeline - or you’ll have to run them in batches and lose data about which specific change makes the tests fail. We’re looking at 20-40 checkins a day.
Assumption: pipeline has 4 stages: build(10 min), deploy(5 min), smoke tests(15 min), acceptance tests(90 min). (This is a simplification of the real situation, but it doesn’t matter here). The numbers are real (also horrible).
* If we ran every checkin separately, we’re looking at 2hr * 20 checkins minimum. Obviously there are not this many hours in a day.
* What we should really be doing is making the pipeline stages faster. We’re also trying to do that, but it’s not easy without buy-in and/or time allocation from the “testing team” (in a different country)
* The build includes all unit tests
* Smoke and acceptance tests reach across the network and hit the service interface of our deployed app
Current situation: builds can happen in parallel on different boxes or the same box. Deploys cannot. It’s one deploy per box (or else they will interfere with each other), and we only have one deployable box.
Action: We got more deployable boxes. Yay!
Problem: Now we can deploy to as many boxes as we want, but how do the testing jobs know which deploy to test against? What if we deploy a new version of the app to box B while tests are being run against it? We have to prevent this somehow.
Assumption: using Jenkins 1.456, with plugins: Parameterized Trigger, Build Pipeline, Node and Label Parameter, Startup Trigger (and others, but those are the relevant ones for this activity)
Assumption: the tests accept a “test against this url” variable
Node config
* The basic manipulation here is to use a script in a Jenkins job to update the config of a Jenkins node
* This is done by posting xml to the api of the Jenkins node. Example:
** curl -v -H “Content-Type: text/xml” -d @config.xml http://<jenkins>:8080/computer/%NODE_NAME%/config.xml
Deploy job
* Can only run on nodes labeled “deployable” (thanks to the Node and Label Parameter Plugin)
* Add a step to relabel the box that the deploy job is running on to “tests-are-running-against-this-box” or similar.
* Echo NODE_NAME into a temp file as ‘testTargetHost’ or similar
* Add parametrized trigger of smoke tests which passes out the temp file with testTargetHost
Label cleanup job
* The deployed-to node should have its label changed back to “deployable” when tests are done running.
* Very similar to the relabel step in the deploy, but backwards, i.e. with sed s/tests-are-running-against-this-box/deployable/
In smoke test job
* Parametrize job to take in testTargetHost
* Command your smoke test target to run against the box testTargetHost
* Echo testTargetHost to new temp file so it can be passed as a parameter to the acceptance test job or label cleanup job
* Add parametrized trigger of acceptance tests which passes out the temp file with testTargetHost
* Add parametrized trigger of label cleanup job on FAILURE of smoke tests- pass it testTargetHost from the temp file (so that the node will become deployable again if no tests are running against it anymore)
In acceptance test job
* Parametrize job to take in testTargetHost
* Command your acceptance test target to run against the box testTargetHost
* Echo testTargetHost to new temp file so it can be passed as a parameter to the label cleanup test job
* Add parametrized trigger of label cleanup job on pass or fail of acceptance tests and pass it testTargetHost from the temp file
Notes / lessons learned
* Windows does not come with sed or curl, so make a new windows share on Jenkins master and install them there so you don’t have to install them to all nodes individually
* Windows echo includes spaces, unlike unix echo.
** echo testTargetHost=%NODE_NAME%> testHostInfo.txt :: there is no space
** echo testTargetHost=%NODE_NAME% > testHostInfo.txt :: this space will cause problems
* Make a Jenkins job to kick off the thinBackup plugin daily and check the results into your version control system. Also use the JobConfig History plugin (the backup will catch this data also) - helps with more complex configurations
* Master node is treated differently in the api- script around it like the nuisance it is or don’t dynamically relabel it (my choice)
* Recovering from unplanned shutdown
** After a shutdown in the middle of a pipeline run, the labels will be in a ‘bad’ state, i.e. one or more nodes labeled “tests-are-running-against-this-box” and nothing in progress to kick off the label cleanup job.
** So make a job which runs on startup which relabels badly labeled nodes (see sample code below)
Script examples:
echo testTargetHost=%NODE_NAME%> testHostInfo.txt
curl http://<jenkins>:8080/computer/%NODE_NAME%/config.xml | sed s/deployable/temporarily-removed-from-deploy-pool/ > config.xml
curl -v -H “Content-Type: text/xml” -d @config.xml http://<jenkins>:8080/computer/%NODE_NAME%/config.xml
# label cleanup after restart in middle of pipeline
# note: the loop is windows-specific. It’s just as easy in bash though.
echo Find the list of nodes that need to be retagged
curl http://<jenkins>:8080/label/temporary-label-for-experiments/api/xml | sed “s/>/\n/g” | grep “<\/nodeName” | sed “s/<\/nodeName//” > mislabeled_nodes.txt
for /F “tokens=*” %%A in (mislabeled_nodes.txt) do (
curl http://<jenkins>:8080/computer/”%%A”/config.xml | sed s/temporarily-removed-from-deploy-pool/deployable/ > config.xml
curl -v -H “Content-Type: text/xml” -d @config.xml http://<jenkins>:8080/computer/”%%A”/config.xml
)
dynamically relabel jenkins nodes using the xml api
curl http://<jenkins>:8080/computer/%NODE_NAME%/config.xml | \sed s/deployable/temporarily-removed-from-deploy-pool/ > config.xml
curl -v -H “Content-Type: text/xml” -d @config.xml http://<jenkins>:8080/computer/%NODE_NAME%/config.xml
Jenkins hides the “properties” section of an advanced command. If you put a -Dvalue=variable in the main Ant section, it (at least in my case) does not take effect…
tried to fix the world
couldn’t find the source code
rewriting is going to take a while
prod server password please?
get list of nodes with a particular label from a job in Jenkins
Windows way to get list of nodes with a particular label from a job in Jenkins (assumes that you have curl and sed… which I had to install specially for this and other reasons)
curl http://<yourJenkinsServer>/label/<yourLabel>/api/xml | sed “s/>/\n/g” | grep “<\/nodeName” | sed “s/<\/nodeName//”
Explanation:
curl http://<yourJenkinsServer>/label/<yourLabel>/api/xml # this gets the raw xml. There are no linebreaks in it.
| sed “s/>/\n/g” # break the xml on the end-bracket to format the xml into lines, one of which which has the nodename, followed by something that we can delete with sed
| grep “<\/nodeName” # only lines with node names on them will have this
| sed “s/<\/nodeName//” # delete everything on the line except the node name
Note that in windows, any expression with a bracket in it needs to be surrounded by ” but in unix it only needs a ‘
Note that the \n works in sed (cygwin sed) but not in the sed that comes with osx
A version of this which works on osx:
curl http://<yourJenkinsServer>/label/<yourLabel>/api/xml | sed -e ‘s/>/\
/g’ | grep “<\/nodeName” | sed ‘s/<\/nodeName//’
Note the linebreak above- this is intentional and is because the \n did not work.
Note that this might also be able to be done in Groovy, probably more simply.
how do you script deleting a label from all jenkins nodes?
Probably label manipulation using Groovy
acceptance test of journeys
http://thoughtworks.fileburst.com/assets/thoughtworks-tech-radar-march-2012-us-color.pdf
What does that mean?
Partial explanation http://www.infoq.com/news/2011/02/Technology-Radar-ThoughtWorks
jenkins conditionally parameterize build
do it somehow
why do I have to restart jenkins so often?
because it has out of memory errors
because it is running on windows
just make a job to restart it every night
different ways to tax the rich
A thought came to me when I read the phrasing of http://www.stallman.org/archives/2011-nov-feb.html#20_February_2012_(Child_Care)
Increasing taxes on the rich makes the rich feel negatively about a plan- they do not want to be less rich. But a common cause of wanting to tax the rich more heavily is wanting to create more jobs. I would prefer a tax measure which required a person with significant assets to prove that they have caused more people to be employed gainfully in a given year, or a stronger tax measure will be applied. A ‘did not provide jobs’ tax for the ultra-rich.
Providing proof of having created jobs might be difficult. Also difficult- would the jobs have any value? It might be cheaper for a rich man to pay several people to sit around for minimum wage than to create a new company which would use their employment productively. Also difficult- is there a phrasing of this which would allow it to be put into a law? What underlying principles of democracy does it subvert? I am interested in your opinion on this.
