Jenkins Blogs

Syndicate content
Pipes Output
Updated: 23 min 48 sec ago

Come join the infra team!

Fri, 2014-04-18 14:45

We are looking for volunteers to join the small infra team here at the Jenkins project. We are the butlers of the butler that get Mr.Jenkins going.

We've been managing our servers through puppet, and have been slowly folding pieces one at a time to puppet, but there's still a lot of snowflake services that need proper operationalization.

So to fix them up, PuppetLabs folks generously agreed to help us get going with a deployment of Puppet Enterprise. Tyler has managed to arrange a "rapid deployment" engagement. To kick start the effort, an instructor would come for one week (April 28th-May 2nd) to bring us up to speed on modern Puppet. we'll then spend some time on our own to puppt-ize more, and deploy Puppet Enterprise.

The end goal is to ensure sustainability of our infrastructure, in case of unexpected server loss.

As we are about to get this effort going, we think this is a good time to solicit a few more volunteers. We are looking for someone who could join this two week engagement in San Francisco, and keep their involvement beyond that. This is a part time volunteer work, and you'd get some visibility and exposure to the inner guts of open-source projects, not to mention the satisfaction of getting thanked for your work.

If you are interested, please drop us a note at the infra list.

Active Directory plugin improvements

Wed, 2014-04-16 09:00

One of the few plugins that I still personally maintain is Active Directory plugin. In the past few months, I've been making steady improvements in this plugin, thanks to various inputs and bug reports given to me from the ClodBees customers.

One of the recent fixes was to get the "remember me" feature finally working for Active Directory. This requires a relatively new Jenkins 1.556, but it eliminates the need to having to constantly type the password in.

Then I've rebumped the version of COM4J, which was causing a thread leak when Jenkins runs on Windows. If you are running a Windows deployment with lots of active users, this probably would have contributed to the instability of Jenkins.

And then lastly, a small but crucial improvement was made to the way we search group membership, so that we can avoid recursively searching AD. This should result in a significant speed improvement when you are logging into Jenkins through AD.

The latest version of the plugin as of writing is 1.37. I hope you'll have a chance to update the plugin soon.

Upcoming Jenkins Office Hours: Acceptance Test Harness

Tue, 2014-04-15 11:38

One of the new efforts in Jenkins this year is the acceptance test harness for Jenkins.

We will be doing the Jenkins office hours next week to go over this and sync up and coordinate between people in the community that are trying to work on this.

It'll be April 23rd 11am PT (see what this time is in your time zone) on Google Hangout at If you are intereste in hacking Jenkins or if you are a large user of Jenkins who have acceptance tests, we are looking forward to seeing you there.

For those of you who haven't looked, this test harness allows you to write blackbox tests of Jenkins and its plugins. It was originally used to test LTS releases, but over the time, it acquired a number of features, such as ...:

  • Docker support for launching complex fixtures to test Jenkins with.
  • Pluggability to launch Jenkins under test (JUT) in many different environments
  • Pluggability to provision Jenkins and slaves from EC2 to test large deployments
  • Choice of cucumber or JUnit to write test scripts

We are working on porting over existing test cases, but we'd like to work with users to move their acceptance tests on top of this same harness. The idea is to pool those test cases in the community so that we can test Jenkins and its plugins as we develop them. For this to work, we want tests to have lots of metadata (such as what plugins it touches), and for the harness to have sufficient modularity that different people can run the same scenario against different deployments, including existing instance.

Jenkins 1.532.3 LTS is released

Fri, 2014-04-11 12:48

The final LTS release of the 1.532.x line is out today. You can download it from the usual location. Changelog is here.

Starting with the next 1.554.x LTS, the release model will switch to the train model, where we commit to dates and get whatever we can ship by that date.

You can see the scheduled dates in our event calendar. Backporting window for 1.554.1 is almost closing, so if you want to have your favorite issues nominated for it, please see the process in the Wiki and hurry!

InfoQ CI survey 2014

Fri, 2014-04-11 12:40

InfoQ has been running a CI server survey for more than a month now, and here is the current result:

Jenkins has gotten more than 70% of the votes, once again proving the wide adoption among developers. If you are one of those who picked Cruise Control into the "considering" section, I'd encourage you to look around a bit more.

You can still vote from their website or leave comments if you want.

By the way, the design of two axes make no sense to me; for example, I'd order the adoption axis to "considering -> migrating to -> using now -> moving away from", and the circle seems to imply two axes are somehow interchangeable, when it should probably be just in a checkerboard to indicate those are independent axes.

More scalable slaves

Wed, 2014-04-09 17:18


NIO-based Java Web Start (JNLP) slave handling is coming to 1.560. This will help you run a large number of JNLP slaves more efficiently. A connected JNLP slave used to occupy one thread on the master, but now it occupies none. Combined with the earlier change that eliminated threads from idle executors, now you can connect thousands of slaves.

All you have to do is to use the latest slave.jar from Jenkins 1.560. No other changes are necessary on users' part.

A bulk of this is implemented in remoting 2.38, and a good part of it was implemented about a year ago on the airplane on the way to Europe.

We plan to make CLI connections take advantages of this too, which helps those who use that a lot. That's not in 1.560, but hopefully it'll be in the near future. This change also paves a way for multi-participant bus-topology communication, which I think would be an useful building block for the work-in-progress master-to-master API.

Maven job type performance improvements in Maven plugin 2.2

Fri, 2014-04-04 09:20

I recently had an opportunity to visit a big Jenkins user on site, and one of the things they've told me is that building projects in the Maven job type is substantially slower than doing the same with the freestyle project type.

This is partly expected, because this job type does more for you. For example, it automatically archives your build artifacts, fingerprints all the relevant information, and so on. These are good things, and naturally, it cost time.

But the slow down they are seeing was substantial, and this is a complaint I've heard from others as well. So I started looking into it.

With a help of artificial delay induced to my network interface and several custom scripts to probe into the running processes, I was able to understand what was going on and make some good improvements.

First, in Maven plugin 2.0, we've made a change in the way we archive artifacts from Maven. Previously, the artifacts were copied between the master and the Maven JVM, and for a reason I'll mention later, this was very slow, especially in a network that has a large latency. With Maven plugin 2.0 and onward, artifacts are archived between the master and the slave JVM.

The second problem that I discovered was that the spy program we put inside Maven is causing excessive amount of unnecessary classloading. Some classes have static initializers that too eagerly refer to other classes, which in turn brings in other classes, and so on. Despite the jar file caching that we do, these classloading still sometimes requires precious roundtrips to the master, which costs in the order of 10s of ms. I was able to make various changes in Jenkins core to cut this down, and these fixes will land in Jenkins 1.559 (ETA is April 14th.) The classloading overhead is independent of the size of your Maven build, so this improvement is more for people who have lots of small Maven builds, like Jenkins building Jenkins plugins.

Now, on to the biggest fruit of this investigation I was able to discover and fix. Imagine the Maven JVM has a lot of data to send to the master, say you are archiving test reports or code coverage report. A good implementation would send these data as fast as ppossible to the master, paying respect to the limit of flow control to avoid overwhelming the master.

It turns out that the way we set up this communication channel was far from optimal. Instead of having the Maven JVM push data with flow control, we were relying on the master to pull data. That is, master has to send out a request to the slave to fetch the next batch of data (8KB), then once it receives that data, it sends out another request to fetch the next batch of data, and so on. If your network latency is 10ms, this scheme only lets us send 500KB/sec, even if you have a gigabit ethernet. No wonder it was so slow!

This fix is in in Maven plugin 2.2. See JENKINS-22354 if you want to know more about the actual diffs and such.

Unfortunately, none of these are available for those who are on 1.532.x LTS, but the next 1.554.1 LTS will be able to run the newer Maven 2.2 plugin. So the help is on the way!

Your Java Web Start slaves will be always clean

Tue, 2014-04-01 17:25

If you have slaves that connect through Java Web Start (such as slaves installed as Windows services), we have a good news for you.

In case of a connection loss, this type of slaves has been designed to automatically attempt to reconnect to the master. This makes sense because you want these slaves to remain online all the time, even if your janitor trips over the ethernet cable. Unfortunately, it also means that over the time, these slaves accumulate gunk, such as mutated static states, any left-over threads or memory leaks, or native libraries that are loaded into JVM.

To prevent that, a better approach is to restart the slave JVM (JENKINS-19055) and have the new JVM reconnect, instead of having the same JVM reconnect. That would ensure that the slave always stays clean. I've planned to make this change for a while now, and I'm happy to report that this change is finally landing to the upcoming 1.559.

Restarting JVM is easy on Unix, where I could just exec(3) to itself. We've been doing this for ages on masters, for example when you update a plugin and tell Jenkins to restart.

The hard part is to do this for Windows, where the most of the time was spent. I had to improve windows service wrapper to support self-restarting services, which turned out to be trickier because Windows service control manager doesn't provide "restart" as an atomic operation. It also kills not just the service process itself but all the processes in the group. So I had to double-fork the service wrapper into a separate process group just to restart a service from within itself.

In any case, the end result is that if you have installed a service through GUI, be it on Windows, Unix, or OS X, slaves will restart themselves every time it gets disconnected from the master.

I've also taken the opportunity to make jenkins-slave.exe on the slave self-updating. Every time it connects to the master, it gets the latest version from the master.

If you have installed Web Start slaves as services, make sure to update the local copy of slave.jar on these slaves to 2.37 or later. This "restart on reconnect" feature only kicks in when you are running this very recent version of slave.jar. And yes, we realize it'd be nice for slave.jar to update itself, which is tracked as JENKINS-22454. But that's a work for another day.

Call for Sponsors: 2014 Jenkins User Conferences

Fri, 2014-03-21 11:19

Jenkins User Conference (JUC) season is upon us! It’s a busy year for the Butler – he’s hosting conferences all over and looking for sponsors to help:

  • Boston – June 18
  • Berlin – June 25
  • Herzelia, Israel – July 16
  • Bay Area (California) – October (date TBD)

Mr. Jenkins and the JUC Organizing Committee want to invite you and your company to sponsor a JUC this year. Show your support for the Jenkins community and help keep costs low for attendees*. The funds go to are put to good use: conferences are two full tracks. Lunch, light breakfast, coffee and a coveted Jenkins t-shirt are also included.

Sponsors get all sorts of thanks from the Jenkins community:

  • Your logo on the conference t-shirt and all other conference communication (emails, website, signage, etc.)
  • A blog featuring sponsors
  • Free passes
  • Silver and Gold sponsors get a table to talk to folks and hand out swag
  • Gold sponsors get either a speaking slot, happy hour sponsorship or a dedicated room for demos
  • And more, but most especially, you get to support JenkinsCI.

Just let us know if you’re interested to get the details. We’d love to have you join us.

Friendly reminder: We are looking for speakers for all four cities. Call for Papers ends March 30 for Boston, Berlin and Israel. Submit your abstract now and come share your expertise with the Jenkins community.

We hope to see you at a JUC this year!

Lisa, Alyssa and the JUC Organizing Committee

*PS - Registration just opened for Boston and early-bird tickets are only $59.

Jenkins User Conferences This Year

Wed, 2014-03-19 13:52

Over the past three years, the Jenkins User Conference is held annually in the Bay Area with a few events in different locations around the world. The Jenkins User Conference has established a reputation as a focal point for the Jenkins community to come together to share new ideas and best practices. Each year we have experienced the growth and expansion within the Jenkins community. This year we are taking this platform to other regions of the world, offering regional gatherings of Jenkins users and developers.

At the moment, we are working on the following JUCs and events for 2014:

  • JUC Boston - June 18
  • JUC Berlin - June 25
  • JUC Israel - July 16
  • JUC San Francisco Bay Area - October (TBD)
  • JUC Australia/New Zealand - November/Dec (TBD)

Jenkins Events

  • Copenhagen - September
  • Brazil - November/December (TBD)

There are a few different ways to get involved:

Learn more:

Looking forward to seeing you at a local JUC near you! :o)

Jenkins at FOSDEM 2014

Sat, 2014-01-25 19:20

FOSDEM is probably the biggest open-source developers' conference in Europe.

This year will be the 3rd year for us to be present in FOSDEM. There will be a bunch of community people, handing out flyers and stickers, showing Jenkins, and generally be available to talk to people! This year, we'll bring some Jenkins T-shirts to sell, and hopefully some bobble heads as well. So please be sure to drop by.

And if you are already involved in the Jenkins project and willing to help us man the booth, that'd be awesome! Looking forward to seeing you!