Faster slave classloading


Jenkins comes with the remoting library that it uses to communicate between a master and slaves. This is a pretty awesome library, I think, which served us well.

One of the things this remoting layer does it to transfer the Java byte code on demand from the master to slaves on demand. This approach helps us keep slave deployment simple, as you don't have to keep the master and all the slaves in sync, but it also made the slave start-up slower, because none of the byte code loaded to slaves are kept around. It was all forgotten once the slave gets disconnected.

When slaves are static and stays online for hours, this wasn't a problem at all. But as more and more slaves become elastic (think EC2 or CloudBees DEV@cloud), This delay is becoming more and more noticeable. A similar issue happens when the Maven project type, which uses the same remoting library to talk to the running Maven build.

Jenkins User Conference Israel Summary

Jenkins User Conference in Israel was held this year on a different venue than the last year, because we have grown! I believe Shlomi Ben-Haim of JFrog said in his opening speech that the attendance has grown more than 50%, despite the ticket price increase.

This year, the event was held at a former Kibbutz turned into an event facility. This was rather fit for Jenkins for both emphasizes the community. The auditorium was big, the sky was bright & clear, and it was a wonderful day. JFrog folks even made a few Jenkins drapes (that I eventually brought back with me, so expect to see them)



Office hours this week: Git plugin refactoring

This Wednesday's Jenkins office hours is all about the Git plugin refactoring that's going on.

Git plugin is one of the most popular plugins out there, and it's been around for quite some time. Combine that with the fact that there are so many different ways to use Git, it was inevitable that Git plugin became quite a capable but complex plugin over time. It has more than a dozen options and switches, and it was becoming harder to use and harder to maintain.

As early as 2010, some of us have already been saying that we should refactor this plugin, but none of us have managed. The good news is, I finally started tackling this problem last month while I was in London, and I've made a steadily progress since then and I'm ready for a wider review.

So we'll spend this Wednesday going over the changes.

Reducing the # of threads in Jenkins: SSH slaves

As the usage of Jenkins expands, we started seeing users who run multiple 100s of slaves on one master, and thus it became a lot more important for us to scale well to even larger number of slaves.

While I was looking at the thread dump of a large system, I started noticing that there are a large number of threads lying around pumping InputStream and writing to another OutputStream. On Linux, each thread occupies 2MB just for its stack size, so if we can eliminate some of them, it'd be a good saving.

So this morning, I tackled one source of such waste.

Jenkins has the ability to launch slave on a remote server via SSH for the longest time, and to simplify this, we've been using a pure-Java implementation of SSH client.

To cut the long story short, I was able to eliminate two pump threads per every SSH connection. Furthermore, when it runs on the upcoming Jenkins 1.521, it'll save one more thread per every SSH connection. So if you have 100 slaves connected through SSH, this alone saves up 600MB of memory. That's pretty good for a few hours work!

Jenkins User Conference Israel - Coming June 6

Your favorite Butler will visit Israel on June 6 for the Jenkins User Conference Israel. More than 200 people have already registered to hobnob with other Jenkins users and eat like kings.

The agenda is up here. You’ll find a great list of speakers from Israel, Europe and the US to compliment a plethora of treats. There will be an ice cream break, fruits break, beer break and special chef lunch.