Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
Merge pull request #7 from jenkinsci/master
JENKINS-28648
  • Loading branch information
cast committed May 29, 2015
2 parents df9f666 + 978953d commit 755d4a7
Show file tree
Hide file tree
Showing 11 changed files with 280 additions and 57 deletions.
36 changes: 21 additions & 15 deletions blogpost/lambda.md
@@ -1,14 +1,14 @@
#AWS Lambda Jenkins plugin

Since the release of AWS Lambda in preview mode we wanted to use it to process event based flows.
For one of our larger projects it immediately became clear that we needed asynchronous handling of files, isolated from the main api which we wanted to quickly respond to any client.
Ever since the release of AWS Lambda in preview mode we were passionate to use aws lambda to process event based flows. For one our latest larger projects we at Cronos (Xplore Group and Cloudar) immediately saw the need for asynchronous handling of files, isolated from the main api. Our challenge was that the api could quickly respond to any client.
This is how we did it.

Instead of processing files for several seconds, blocking our users calls and reducing throughput, we decided to put the file on S3 and let AWS Lambda process it asynchronously.
Instead of processing files for several seconds, blocking our users calls and reducing throughput, we decided to put the file on S3 and let AWS Lambda process it asynchronously. Coupled with the AWS Lambda retry mechanism we have a robust system.

After building and testing the Lambda function we wanted to integrate the function deployment into our continuous integration and deployment system using Jenkins. Command line tools including the AWS cli and Kappa where available but we wanted to limit the dependencies of our build servers.
After building and testing the Lambda function we wanted to integrate the function deployment into our continuous integration and deployment system using Jenkins. Command line tools including the AWS cli and Kappa were available but we wanted to limit the dependencies of our build servers.
That's why we decided to develop a Jenkins plugin for AWS Lambda that would allow us to deploy functions without further dependencies.

Currently the plugin can deploy and invoke functions as a build step and post build action. When invoking a function it is possible to couple the output to Jenkins environment variables.
Currently the plugin can deploy and invoke functions as a build step and post build action. When invoking a function it is possible to inject the output as Jenkins environment variables.

Github link: [https://github.com/XT-i/aws-lambda-jenkins-plugin](https://github.com/XT-i/aws-lambda-jenkins-plugin)
Jenkins wiki link: [https://wiki.jenkins-ci.org/display/JENKINS/AWS+Lambda+Plugin](https://wiki.jenkins-ci.org/display/JENKINS/AWS+Lambda+Plugin)
Expand Down Expand Up @@ -53,7 +53,7 @@ You'll also need access to iam:PassRole to attach a role to the Lambda function.
]
}

For invocation you only need InvokeFunction:
For invocation you only need access to InvokeFunction.

{
"Version": "2012-10-17",
Expand All @@ -73,20 +73,22 @@ For invocation you only need InvokeFunction:

##AWS Lambda function deployment

After creating a job you can add a build step or post build action to deploy an AWS Lambda function
After creating a job you can add a build step or post build action to deploy an AWS Lambda function.

![Jenkins Build Step menu](build-step.jpg)

Due to the fact that AWS Lambda is still a rapid changing service we decided not to have select boxes for input.
The AWS Access Key Id, AWS Secret Key, region and function name is always required. All other fields depend on the update mode.
The AWS Access Key Id, AWS Secret Key, region and function name are always required. All other fields depend on the update mode.

If the update mode is Code you also need to add the location of a zipfile or folder.
Folders are automatically zipped according to the [AWS Lambda documentation](http://docs.aws.amazon.com/lambda/latest/dg/walkthrough-s3-events-adminuser-create-test-function-create-function.html)
For the Configuration update mode you need the role, handler and if you want to diverge from the defaults the memory and timeout values.
When choosing the Both update mode, both UpdateFunctionCode and UpdateFunctionConfiguration is updated.
Folders are automatically zipped according to the [AWS Lambda documentation](http://docs.aws.amazon.com/lambda/latest/dg/walkthrough-s3-events-adminuser-create-test-function-create-function.html)

For the Configuration update mode you need the role and handler. If you want to diverge from the defaults add the memory and timeout values.

When choosing the Both update mode, both UpdateFunctionCode and UpdateFunctionConfiguration are performed.

If the function has not been created before the plugin will try to do a CreateFunction call, which needs all fields previously mentioned in addition to the runtime value.
The update mode value is ignored if the function does not exists, but it will take effect for future builds if the function still exists.
The update mode value is ignored if the function does not exists yet, but it will take effect in future builds.

![AWS Lambda Jenkins plugin deployment configuration](deploy.jpg)

Expand All @@ -96,14 +98,15 @@ To invoke a function once again open up the add build step or post build action

![Jenkins Post Build Action menu](post-build.jpg)

Again you need to add the AWS Access Key Id, AWS Secret key, region and function name. Optionally you can add a payload that your function expects.
You need to add the AWS Access Key Id, AWS Secret key, region and function name. Optionally you can add a payload that your function expects.

If you enable the Synchronous checkbox you will receive the response payload that can be parsed using the Json Parameters.
You will also get the log output from Lambda into your Jenkins console output.
You will also get the logs from Lambda into your Jenkins console output.

![AWS Lambda Jenkins plugin invocation configuration](invoke.jpg)

The json parameters allow you parse the output from the lambda function if the json format is used. The parsed value will then be injected into the Jenkins environment using the chosen name.
The json parameters allow you to parse the output from the lambda function. The parsed value will then be injected into the Jenkins environment using the chosen name.
An empty jsonPath field allows you to inject the whole response into the specified environment variable.

![AWS Lambda Jenkins plugin invocation json parameters](invoke-json-parameters.jpg)

Expand Down Expand Up @@ -135,3 +138,6 @@ These environment variables can be used as parameters in further build steps and
On the job build result page you'll get a summary of all deployed and invoked functions and their success state.

![AWS Lambda Jenkins plugin job build result](result.jpg)

We hope this blog post is an inspiration for all of you to get more out of the AWS Lambda and continuous integration and deployment.
At Cronos we’re sure that in the future more challenges will cross our path and we look forward to share them with you.
2 changes: 1 addition & 1 deletion pom.xml
Expand Up @@ -45,7 +45,7 @@
</developers>

<properties>
<aws.version>1.9.35</aws.version>
<aws.version>1.9.39</aws.version>
<jsonpath.version>2.0.0</jsonpath.version>
</properties>

Expand Down
@@ -1,7 +1,11 @@
package com.xti.jenkins.plugin.awslambda.exception;

public class AWSLambdaPluginException extends Exception {
public class AWSLambdaPluginException extends RuntimeException {
public AWSLambdaPluginException(String message) {
super(message);
}

public AWSLambdaPluginException(String message, Throwable cause) {
super(message, cause);
}
}
@@ -0,0 +1,12 @@
package com.xti.jenkins.plugin.awslambda.exception;

public class LambdaDeployException extends AWSLambdaPluginException {

public LambdaDeployException(String message) {
super(message);
}

public LambdaDeployException(String message, Throwable cause) {
super(message, cause);
}
}
Expand Up @@ -3,6 +3,7 @@
import com.amazonaws.AmazonClientException;
import com.amazonaws.services.lambda.AWSLambdaClient;
import com.amazonaws.services.lambda.model.*;
import com.xti.jenkins.plugin.awslambda.exception.LambdaDeployException;
import com.xti.jenkins.plugin.awslambda.upload.UpdateModeValue;
import com.xti.jenkins.plugin.awslambda.upload.DeployConfig;
import com.xti.jenkins.plugin.awslambda.util.LogUtils;
Expand All @@ -29,18 +30,18 @@ public LambdaDeployService(AWSLambdaClient client, JenkinsLogger logger) {
* - No function found:
* - Always call createFunction regardless of UpdateMode value
* @param config configuration to be updated or added
* @param zipFile zipfile containing code to be updated or added
* @param functionCode FunctionCode containing either zipfile or s3 location.
* @param updateModeValue Full, Code or Config, only used if function does not already exists.
* @return true if successful, false in case of failure.
*/
public Boolean deployLambda(DeployConfig config, File zipFile, UpdateModeValue updateModeValue){
public Boolean deployLambda(DeployConfig config, FunctionCode functionCode, UpdateModeValue updateModeValue){
if(functionExists(config.getFunctionName())){

//update code
if(UpdateModeValue.Full.equals(updateModeValue) || UpdateModeValue.Code.equals(updateModeValue)){
if(zipFile != null) {
if(functionCode != null) {
try {
updateCodeOnly(config.getFunctionName(), zipFile);
updateCodeOnly(config.getFunctionName(), functionCode);
} catch (IOException e) {
logger.log(LogUtils.getStackTrace(e));
return false;
Expand All @@ -66,9 +67,9 @@ public Boolean deployLambda(DeployConfig config, File zipFile, UpdateModeValue u
return true;

}else {
if(zipFile != null) {
if(functionCode != null) {
try {
createLambdaFunction(config, zipFile);
createLambdaFunction(config, functionCode);
return true;
} catch (IOException e) {
logger.log(LogUtils.getStackTrace(e));
Expand All @@ -87,12 +88,10 @@ public Boolean deployLambda(DeployConfig config, File zipFile, UpdateModeValue u
/**
* This method calls the AWS Lambda createFunction method based on the given configuration and file.
* @param config configuration to setup the createFunction call
* @param zipFile zipfile that will be uploaded
* @param functionCode FunctionCode containing either zipfile or s3 location.
* @throws IOException
*/
private void createLambdaFunction(DeployConfig config, File zipFile) throws IOException {

FunctionCode functionCode = new FunctionCode().withZipFile(getFunctionZip(zipFile));
private void createLambdaFunction(DeployConfig config, FunctionCode functionCode) throws IOException {

CreateFunctionRequest createFunctionRequest = new CreateFunctionRequest()
.withDescription(config.getDescription())
Expand All @@ -112,15 +111,17 @@ private void createLambdaFunction(DeployConfig config, File zipFile) throws IOEx
/**
* This method calls the AWS Lambda updateFunctionCode method based for the given file.
* @param functionName name of the function to update code for
* @param zipFile zipfile that will be uploaded
* @param functionCode FunctionCode containing either zipfile or s3 location.
* @throws IOException
*/
private void updateCodeOnly(String functionName, File zipFile) throws IOException {
ByteBuffer functionCode = getFunctionZip(zipFile);
private void updateCodeOnly(String functionName, FunctionCode functionCode) throws IOException {

UpdateFunctionCodeRequest updateFunctionCodeRequest = new UpdateFunctionCodeRequest()
.withFunctionName(functionName)
.withZipFile(functionCode);
.withZipFile(functionCode.getZipFile())
.withS3Bucket(functionCode.getS3Bucket())
.withS3Key(functionCode.getS3Key())
.withS3ObjectVersion(functionCode.getS3ObjectVersion());

logger.log("Lambda update code request:%n%s%n", updateFunctionCodeRequest.toString());

Expand Down Expand Up @@ -165,6 +166,45 @@ private Boolean functionExists(String functionName){
}
}

public FunctionCode getFunctionCode(String artifactLocation, WorkSpaceZipper workSpaceZipper){
if(artifactLocation.startsWith("s3://")){
String bucket = null;
String key = null;
String versionId = null;

String s3String = artifactLocation.substring(5);
int versionIndex = s3String.indexOf("?versionId=");
if(versionIndex != -1){
versionId = s3String.substring(versionIndex + 11);
s3String = s3String.substring(0, versionIndex);
}
int separatorIndex = s3String.indexOf("/");
if(separatorIndex != -1){
bucket = s3String.substring(0, separatorIndex);
if(s3String.length() > separatorIndex + 1) {
key = s3String.substring(separatorIndex + 1);
}
}

return new FunctionCode()
.withS3Bucket(bucket)
.withS3Key(key)
.withS3ObjectVersion(versionId);

} else {
try {
File zipFile = workSpaceZipper.getZip(artifactLocation);
return new FunctionCode()
.withZipFile(getFunctionZip(zipFile));
} catch (IOException ioe){
throw new LambdaDeployException("Error processing zip file.", ioe);
} catch (InterruptedException ie){
throw new LambdaDeployException("Error processing zip file.", ie);
}

}
}

/**
* Get ByteBuffer from zip file.
* @param zipFile file to be wrapped.
Expand Down
@@ -1,5 +1,6 @@
package com.xti.jenkins.plugin.awslambda.service;

import com.xti.jenkins.plugin.awslambda.exception.LambdaDeployException;
import hudson.FilePath;
import hudson.util.DirScanner;
import org.apache.commons.lang.StringUtils;
Expand Down Expand Up @@ -35,8 +36,12 @@ private File getArtifactZip(FilePath artifactLocation) throws IOException, Inter
File resultFile = File.createTempFile("awslambda-", ".zip");

if (!artifactLocation.isDirectory()) {
logger.log("Copying zip file");
artifactLocation.copyTo(new FileOutputStream(resultFile));
if(artifactLocation.exists()) {
logger.log("Copying zip file");
artifactLocation.copyTo(new FileOutputStream(resultFile));
} else {
throw new LambdaDeployException("Could not find zipfile or folder.");
}
} else {
logger.log("Zipping folder ..., copying zip file");
artifactLocation.zip(new FileOutputStream(resultFile), new DirScanner.Glob("**", null, false));
Expand Down
Expand Up @@ -26,11 +26,11 @@
* #L%
*/

import com.amazonaws.services.lambda.model.FunctionCode;
import com.xti.jenkins.plugin.awslambda.service.JenkinsLogger;
import com.xti.jenkins.plugin.awslambda.service.LambdaDeployService;
import com.xti.jenkins.plugin.awslambda.service.WorkSpaceZipper;

import java.io.File;
import java.io.IOException;

public class LambdaUploader {
Expand All @@ -47,7 +47,7 @@ public LambdaUploader(LambdaDeployService lambda, WorkSpaceZipper zipper, Jenkin
public Boolean upload(DeployConfig config) throws IOException, InterruptedException {
logger.log("%nStarting lambda deployment procedure");

File zipFile = zipper.getZip(config.getArtifactLocation());
return lambda.deployLambda(config, zipFile, UpdateModeValue.fromString(config.getUpdateMode()));
FunctionCode functionCode = lambda.getFunctionCode(config.getArtifactLocation(), zipper);
return lambda.deployLambda(config, functionCode, UpdateModeValue.fromString(config.getUpdateMode()));
}
}
@@ -1,9 +1,6 @@
package com.xti.jenkins.plugin.awslambda.util;

import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.auth.AWSCredentialsProviderChain;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.internal.StaticCredentialsProvider;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.lambda.AWSLambdaClient;
Expand All @@ -12,8 +9,7 @@ public class LambdaClientConfig {
private AWSLambdaClient client;

public LambdaClientConfig(String accessKeyId, String secretKey, String region) {
AWSCredentialsProvider credentials = new AWSCredentialsProviderChain(new StaticCredentialsProvider(new BasicAWSCredentials(accessKeyId, secretKey)));
client = new AWSLambdaClient(credentials);
client = new AWSLambdaClient(new BasicAWSCredentials(accessKeyId, secretKey));
client.setRegion(Region.getRegion(Regions.fromName(region)));
}

Expand Down

0 comments on commit 755d4a7

Please sign in to comment.