Update: Wiki and issue tracker outage

I recently wrote about the two day outage of our wiki and issue tracker:

While this was a rather lengthy outage, it could have been much worse. We lost none of the data, after all.

OSUOSL have since published their post mortem. I was really wrong about not losing any data:

A further complication was that our backups were pointed at mysql2, which was out-of-date with mysql1, due to the initial synchronization failures. Fortunately, we had the binary logs from the 17th through the 30th. This means that though most data could be restored, some data from between the 15th and the 17th was lost.

For our issue tracker, that means that issues JENKINS-29432 to JENKINS-29468 were lost, as well as comments posted from about July 15 12:20 PM to July 17 2 AM (UTC). We know this thanks to the jenkinsci-issues mailing list where the lost issues and comments can be looked up for reposting.

We unfortunately don't have such a record from our wiki.

Upcoming office hour on Workflow

Jesse Glick will host the next office hour this Wednesday, 11 AM PDT on Workflow.

Workflow has been Jesse's project for the last year or so. If you don't know what Workflow is, check out these talks about it from past JUCes:

This will be a developer-focused session on integrating with Workflow. He'll discuss things like how to make sure your plugin can be used as part of workflows, and best practices for extending the workflow DSL. There's already been a session on Workflow in January, but Jesse hasn't been idle, and there's new stuff to share.

Participate in the Hangout on Air or watch live on YouTube.

JUC Speaker Blog Series: Carlo Cadet, JUC U.S. West

Mobile is Joining the Party At This Year’s Jenkins User Conference

Consider this as a shout out to mobile app developers: You are invited! For the first time, there’s a mobile track at this year’s Jenkins User Conference to discuss the best ways to extend CI/CD to mobile application testing.

Bay Area Jenkins Area Meet-up kick-off gathering today

A week ago we reported that Uday is looking at organizing a regular Jenkins meet-up in Silicon Valley. This has made a progress since then, and this evening we'll get together to figure out logistics for the first meet-up:

August 5th, Wednesday 6:30 PM - 7:30 PM
Starbucks, 750 Castro St, Mountain View, CA 94041

The agenda is:

Wiki and issue tracker outage over the weekend

As you may have noticed, our wiki and issue tracker were unavailable from Thursday to Sunday last week. What happened?

We host parts of our infrastructure at the Open Source Lab at Oregon State (OSUOSL), including the databases for these two services. So far, there's no post mortem by OSUOSL (they expect to post one later this week), so we need to piece together what we know.

The databases for the wiki and issue tracker became inaccessible around midnight/12 AM Thu/Fri night (all times UTC). Due to the large number and size of databases on that server, pulling from backups, restoring from backup and replaying the binlogs took them quite a while. During that time, we put up a maintenance screen on the wiki (and messed up the one for Jira, so there were connection timeouts instead).

The databases were back around 3 AM on Sunday. We disabled the maintenance screens around 6 PM later that day.

While this was a rather lengthy outage, it could have been much worse. We lost none of the data, after all. We thank the OSUOSL team for their efforts getting everything back up over the weekend!