Blog

Demystifying Shippable's deployment keys

Mon, 13 Oct 2014 08:25:00 GMT

When you enable a project on Shippable, you can see that a key gets added to your repository's "Deployment Keys" on GitHub or BitBucket. However, when you visit your subscription dashboard on Shippable, there's another "Deployment Key" listed there that doesn't match what you see in your repository settings. I'm going to try and clear that up for you in this post.

tl;dr

Shippable uses two key pairs:

~/.ssh/project_rsa and ~/.ssh/project_rsa.pub for git clone, git pull and git fetch operations
~/.ssh/id_rsa and ~/.ssh/id_rsa.pub for git submodule operations, and any other commands that use SSH which you include in your YML files

Project keys

Every time you enable a project, we generate an SSH key and sets that on GitHub as the deployment key for the project. This happens only for private projects because we uses the APIs to fetch the YML file and figure out what we need to build. This is the key you see when you go to the Deployment Keys settings for your project. During the build, this key pair is available at ~/.ssh/project_rsa and ~/.ssh/project_rsa.pub.

We use this key to do two things:

All git clone/pull/fetch operations that happen in the git_sync step of your build. Because the filenames are not standard key names, we use ssh-agent so git will use these keys when talking to GitHub or BitBucket.
We also use this key pair to encrypt and decrypt secure environment variables in your project. This means that if you ever disable and re-enable a project on Shippable, you will need to re-encrypt the secure environment variables in your YML file.

Subscription keys (a.k.a "Deployment Keys")

We also generate a second set of keys for every subscription. This is what you see when you expand the "Deployment Keys" accordion on the subscriptions dashboard. We ask that you use this the "deployment key" when you want to integrate with services like Heroku, and also when you need to pull in submodules or dependencies from other private repositories. These keys are placed at ~/.ssh/id_rsa and /.ssh/id_rsa.pub during the build. This way, when you do a git push heroku or git submodule update you don't need to worry about ssh-agent and the likes. Git will use id_rsa by default and do the right thing.

Why do we use two key pairs?

If we didn't do this, you would never be able to pull submodules or other dependencies from other private repositories. On GitHub and BitBucket, a key pair can only be associated with a single project or user account. Once we set the project keys as the deployment key on your repository, you would not be able to re-use the same key on another user account to allow pulling code from other repositories. The only other alternative we had at the time was to add the key automatically to your user account, but we decided that this is too invasive and gives us far more permissions that we actually need. By managing two key pairs and making them both available to the build we're letting you decide when and how to grant the additional access required for submodules and dependencies from other private repositories.

Getting Composer to work with GitHub tokens on Shippable

Fri, 10 Oct 2014 06:28:47 GMT

Composer is a dependency manager for PHP. If you're using it in your projects and running builds on Shippable, you might notice that once in a while your builds fail with a dangerous looking error:

Could not fetch https://api.github.com/repos/twilio/twilio-php/zipball/51ab2929a66d4455ecb470eafdfe2bab798945ff, 
enter your GitHub credentials to go over the API rate limit

The credentials will be swapped for an OAuth token stored in /home/shippable/.phpenv/versions/5.4/composer/auth.json, 
your password will not be stored

To revoke access to this token you can visit https://github.com/settings/applications

Build timed out

This can be a frustrating error to deal with, because Composer is waiting for user input and your build will just wait until it times out or you manually cancel it.

Wait, my build runs fine locally. Why does this happen only on Shippable?

Composer downloads dependencies directly from GitHub. By default it does this using unauthenticated requests and GitHub currently allows only 60 unauthenticated requests per hour. Unless you're using dedicated hosts on Shippable, you're running on our shared hosts. It looks like GitHub applies the rate limit to an IP address so if other builds running on the same host have already made 60 calls to GitHub in the last hour, your build is going to fail.

OK, so how do I get this to work on Shippable?

Composer lets you configure a GitHub OAuth token. Once you do this, you'll be making authenticated requests to GitHub and your rate limit immediately shoots up to 5,000 requests per hour. Because this is done with your unique token, the rate limit is all yours to consume and doesn't get shared with any other builds that might be running. Let's get started with this.

Step 1: Create a GitHub token

Generate a GitHub token. You only need read permission, so it's safe to uncheck all of the scopes for the token.

Step 2: Protect your token with Shippable environment variable encryption

Now that you have a token you want to tell composer to use it during your build. However, you shouldn't just check this in to your repository because it's a private token for your account. This is where Shippable's encrypted environment variables come in handy. Head over to your subscription dashboard on Shippable and click on the "Encrypted env vars" link:

Define a new environment variable. I'm going to call it MY_GITHUB_TOKEN like so:

Step 3: Add the environment variable to your YML file

Paste the output from the encryption dialog exactly as it is to the env section of your shippable.yml file. When your build runs, this value will be decrypted and made available to your build as an environment variable called MY_GITHUB_TOKEN.

Step 4: Configure Composer to use your token

The last step is to tell Composer to use your token. Just add this to your before_install section and you're done:

before_install:
  - composer config -g github-oauth.github.com $MY_GITHUB_TOKEN

That's it. Composer will now use your personal token when fetching dependencies from GitHub.

Looking back at 2013

Thu, 26 Dec 2013 13:12:00 GMT

2013 went by quickly, and what a year it has been. Three massive things that I want to reflect on for this year.

1. Shippable goes from "cool idea" to $2M in funding in less than a year

Avi and I started talking about building a continuous integration service somewhere in December of 2012. Since then, we've written and rewritten everything four times, slowly making progress towards a better product-market fit. We started out by using Jenkins as the core of our platform but we realized very soon that it wasn't going to let us innovate fast enough, at which point we built our own CI back end. That was a lot of fun to build.

We got cross platform builds running using a solid distributed architecture that allowed us to run builds 3x faster than the competition. It was around this time that we got in to TechStars Seattle, which turned out to be a massive learning experience for all of us. They don't call it an accelerator for nothing -- the amount of stuff we got done in the three months at TechStars was phenomenal.

Along the way, it became clear to us that Docker was going to change the way the world thinks about development, testing and deployment software. That's when we decided to start moving our own infrastructure to Docker. Today, our customers builds happen inside Docker containers and we're completely embracing an entire development workflow based around Docker. It's an exciting time to be a developer and I'm thrilled to be a part of this team.

2. I've moved to developing full time in Javascript on NodeJS

I love .NET. C# is, without a doubt, the best language that I've ever worked with, but I'm very worried about Windows. It's becoming painfully clear that there's almost no one other than Microsoft that's driving innovation on the Windows platform. All the cool toys seem to be on the other side of the fence and that makes me very sad. It's just disappointing that Microsoft, with all its abilities, has simply allowed the ecosystem to rot away.

As a developer, it's hard to ignore all the awesomeness that's going on in the open source world. As much as I love C# and Visual Studio, there's something to be said about stuff like NodeJS, Python, Docker and Vim. Granted, some of those things can also be used on Windows, but the truth is Windows simply isn't a first class citizen of the open source world and all the interesting stuff happens on Linux first anyway.

I've now learned to love my set up of tmux and Vim (+ plugins, of course). I'm not as productive as I used to be with Visual Studio yet, but it's not as scary as it seems and I'm getting a little better every day.

3. Oh, and one more thing...

Now this is going to come as a complete surprise to everyone because I've been really, really quiet about it so far. It isn't that I wanted to keep it quiet, but the truth is that this little project has kept me so busy for the past few weeks that I just haven't had the time to go announcing it to the world.

Folks, meet Aradhya.

The bundle of awesomeness that's been keeping Niree and I busy since the 7th of December.

I don't think the enormity of parenting has descended on me yet, but I do know this -- when I'm looking at her, time slows down, everything is alright with the world and it all makes sense.

2014, here we come!

Standing here at the cusp of another new year, I'm filled with hope, enthusiasm and excitement. 2013 was one hell of a ride and I'm sure 2014 will be filled with even more ups, downs and everything in between! Here's wishing everyone a fantastic new year filled with all the best that life has to offer. :)

Pro tip: Don't use a generic email address to promote your business

Sat, 16 Mar 2013 12:43:10 GMT

How often do you see a proposal that goes like this:

Hey there,

At XYZ we do awesome stuff around something. Check us out at xyz.com!

Thanks,
xyz@gmail.com

Really? It only takes 10 minutes to set up email on your own domain. If you really don't have the time to do that, then I'm not too sure I want to give you money to do things for me.

You really should

Tue, 12 Mar 2013 19:20:00 GMT

Procrastination is the root of all laziness. It's easy to come up with a million reasons why you shouldn't do something.

Excercise? Too hard. Tomorrow, maybe.
Eat healthy? Just one more piece of cake.
Save money? Ooooh, XBox!
Start a company? Hmmm, maybe in a few years when I'm really "ready".

So why should you do any of this right now? Simple. You have more time and energy to do these things today than you will have tomorrow.

Let it sink in for a minute. Now go do something.

Asynchronous S3 file uploads with async/await

Thu, 07 Feb 2013 19:34:01 GMT

The async/await pattern in C# really is awesome. Perhaps it is just syntactic sugar over the TPL, but it makes the code so much more beautiful and easier to read. Before async/await came around this would have been an ugly mess of BeginXXX/EndXXX. Instead, now we get to do this:

var s3 = AWSClientFactory.CreateAmazonS3Client(accessKey, secretKey);
PutObjectRequest putRequest = new PutObjectRequest()
{
    FilePath = "your-file.txt",
    Key = "whatever/you/want",
    BucketName = "your-bucket-name",
};

PutObjectResponse response = await Task
    .Factory
    .FromAsync<PutObjectRequest, PutObjectResponse>(s3.BeginPutObject, s3.EndPutObject, putRequest, null);

That is so much easier on the eyes, and that's just the wrapper around the traditional BeginXXX/EndXXX calls that the Amazon SDK gives us. If you're using the newer stuff like HttpClient, it's just one line to async goodness:

HttpResponseMessage response = await client.GetAsync("http://bing.com");

Plain-text passwords: Will it ever end?

Fri, 23 Mar 2012 09:11:00 GMT

Sometimes you just have to wonder how long computer security professionals will have to keep screaming “Don’t store passwords in plain-text” before people take notice. Getting an email like this from MasterCard really freaks me out:

I wonder if they store PINs in plain-text, too. It’s just sad when a credit card company’s policies allow something like this to get pushed in to production.

Yes, SIM Cards Must Die

Fri, 23 Mar 2012 08:56:00 GMT

I like this idea.

The author thinks carriers won’t support it because it will prevent them from locking in customers. I think they should support it because it also prevents competitors from locking in customers.

Read the original post at http://ilyabirman.net/meanwhile/2012/03/23/1/

Opening hours

Sun, 15 Jan 2012 12:56:00 GMT

I really wonder why the Indian Railways, arguably one of the most active e-commerce sites in the country, needs to be offline for an hour every day.

Seriously!?

I run a few web apps of my own and I’ve had only 50 minutes of downtime in an entire month. Yes, I know my app is nowhere near as complex as a railway reservation system. And yes, I know, those of you that are measuring up time in 5 nines are laughing at me, but give me a break – my services are in beta. Still, why does the IRCTC need an hour off every single day? Sure, as a customer, I’m somewhat inconvenienced, but as a developer I’m really curious as to what they’re doing in that hour.

Turn off the complication!

Sat, 07 Jan 2012 23:36:00 GMT

Surely, we’d all be better off if we would just turn off “complications” in our system settings.

Priority #1 for 2012: Exercise

Wed, 04 Jan 2012 05:12:00 GMT

2011 has been a very interesting year for me. It’s the year I found out that my body is in far better shape than I give it credit for. The fact that I’ve done almost nothing to aid that over the last 30 years is, however, somewhat troubling. So, for 2012, I’ve decided that the biggest priority for me is to get more exercise.

The funny thing about exercise is that you don’t really need to do a lot of it to start experiencing the benefits. And, you don’t really need to splurge on a gym, either. Just go take a walk. It really is that simple. If you’re getting absolutely no exercise right now, stepping up to nothing more than a brisk 30 minute walk every day will do wonders for your body. Assuming you’re already eating in a relatively healthy manner, a regular walk is all you really need to start burning off fat, increase your energy levels and generally feel better every day.

Walking for half an hour isn’t difficult. What’s difficult is making it a habit. This video is what got me to finally turn this into a real priority for 2012. I hope it does the trick for you, too.

23 and 1/2 hours: What is the single best thing we can do for our health?

Perspective

Mon, 02 Jan 2012 22:53:00 GMT

When Windows 95 shipped, my primary machine was a 486/DX50 with 8MB of RAM. My test machine was a 386 with 4MB of RAM. The combined computing power and storage capacity of all the machines in my office is now exceeded by your cell phone.

-- Raymond Chen, Microsoft.

Interesting stuff I read last week #2

Sun, 23 Oct 2011 09:08:00 GMT

Programming/technology

Are 9 to 5 programmers looked down upon?

There are many kinds of ugly

What kind of attacks against home router’s NAT do exist?

What were the reasons why Windows never had a decent shell?

Official Usability, User Experience and User Interface Guidelines from companies

Google+ gets a +1 for browser security <—Good checklist for web projects

Just some other awesome CSS3 buttons

Essential jQuery plugin patterns

Ask HN: Open-source alternatives of Gmail?

List.js: Sortable, searchable HTML lists

Lytro camera <—Wow, I want one of these

State of .NET image resizing

Startups

How to build a web startup – lean launchpad edition

Why are hand-written signatures still so commonly used?

The coffee shop fallacy <—”Enjoying the product is the luxury of the customer, not the producer.”

Want something? Make it easy to say yes

Beautiful vs. Usable

Mon, 17 Oct 2011 06:53:00 GMT

I like Chrome. I really do. It’s a lovely browser. Fast, clean, minimal. But every so often I run into minor irritations like this, which is why I still use Firefox as my main browser.

Take a look at what Chrome’s tabs look like once you have a lot of stuff going on:

Now compare that to Firefox:

The way I see it, at this point Chrome’s tabs are as useless as they are beautiful. Firefox shows you less tabs in one go, but at least I can use what I see and I can scroll through the tabs with my mouse’s scroll wheel.

Interesting stuff I read last week

Sun, 16 Oct 2011 08:04:00 GMT

Here are some of the more interesting things I’ve read over the past week.

Programming/technology

Hacked! <— Important stuff. Turn on two-factor authentication in gmail if you haven’t done it already.
When to use C over C++, and C++ over C?
Can touching any hardware inside a running PC be dangerous for you?
Hiding your data in plain sight – USB hardware hiding
Expresso 3.0 -- The premier regular expression development tool
Undo History - Why limit it?
Why is not all internet traffic encrypted?
SSL And The Future Of Authenticity
How do mashups work with same-orgin policy?
Dizzying but invisible depth
There is only one Cloud Icon in the Entire Universe

Health/medicine

Miscellaneous

Disconnect, Disable and De-stress: How I'm learning to sit down, think clearly and get stuff done

Tue, 11 Oct 2011 12:51:00 GMT

The recent Onion article on the death of Steve Jobs really struck a chord with me:

Jobs will be remembered both for the life-changing products he created and for the fact that he was able to sit down, think clearly, and execute his ideas

If you really think about, that's a superhero power – the ability to sit down, think clearly and execute. I've always known that I have trouble finishing things. For as long as I can remember, I've always been a starter. I love the rush of embarking on a new project. The initial brainstorming and roughing out the first prototypes really give me a high. Unfortunately, after that I just fizzle out. The actual process of moving things from prototype to real world product always seems to bore me. Somewhere along the way, I generally lose interest and shelve things. As a result, I now have around 10,327 half-finished things.

That sucks.

Reading that piece on The Onion, however, made me take a good hard look at myself and I decided to make a few changes. One of the things I quickly noticed during my introspection was that I'm distracted very easily. Whether I'm writing code, watching television, reading a book or even trying to fall asleep, my mind is always buzzing with a million different thoughts. I'm not quite sure when that started happening, but now I find that it takes a serious effort for me to focus on a single train of thought for an extended period of time.

There are just too many distractions. Too many devices crying for attention. Too many emails to process. Too many blogs to read. And the worst thing is that the default on every device or app is to let you know instantly when something happens. The list is just endless:

Email
RSS
Twitter
Facebook
Hacker News
etc., etc., etc.

This stuff is Kryptonite.

So, as the first step to regaining my superpowers, I've now decided to turn off every notification on every device or app that I own. That's right, everything. Hell, even my phone now makes noises only for incoming calls. No more "you've got mail", twitter alerts, Facebook messages or blog alerts to get in my way. Zero alerts. Period. No exceptions. Nada. Not on the phone, laptop or desktop.

And you know what? The world suddenly seems like a more peaceful place. It's only been two days since I started doing this and already I can feel my thought process improving. Not only am I getting more work done, but I'm doing a better job at it than before.

Try it, people. Turn off the notifications. There's important stuff to be done. Don't let a new tweet or email take your eyes off the ball.

Essential Geek Skills: Automating incremental snapshot backups on Windows with rsync

Sun, 09 Oct 2011 03:26:00 GMT

I just finished setting up a fully automated incremental snapshot backup regime on my Windows 7 machine. Every night, my script creates an exact replica of my data and retains 3 days of snapshots. This means that I can go “back in time” for three days in case I need to recover something quickly. More importantly, the data isn’t hidden away in some obscure, proprietary file format – it’s sitting there just like a bunch of normal files. This makes it really easy to recover from a catastrophic failure of my main drive.

Note: The process I’m about to describe is for geeks only. If you’re not a geek, maybe you should just buy a decent backup application and save yourself the trouble of doing this manually. If, on the other hand, you’ve got an hour to spare and you’re OK with dealing with things like Cygwin, ntrights and the Task Scheduler, read on!

The core idea is to use rsync to make a backup of your data and use hardlinks to create the snapshots, as described by Mike Rubel.

What you’ll need:

Cygwin (we’ll be using rsync to make the backups)
Windows 2003 Resource Kit (We need a copy of ntrights.exe to do this on Windows 7 Home Premium)
A spare hard disk that’s at least the same size as the drive you want to backup

Step 1: Install Cygwin

Get the installer from the Cygwin site. Be sure to install the rsync package by selecting it from the installer:

Step 2: Prepare your backup disk

Format your disk and create the following files:

1) backup.sh

This file creates the actual backup and manages moving around the snapshots. Just be sure to replace /cygdrive/d/* with your source (the location with the data you want to backup) and /cygdrive/e in the following script to point to your backup drive.

rm -rf /cygdrive/e/backup.3
mv /cygdrive/e/backup.2 /cygdrive/e/backup.3
mv /cygdrive/e/backup.1 /cygdrive/e/backup.2
cp -al /cygdrive/e/backup.0 /cygdrive/e/backup.1
rsync -a /cygdrive/d/*  /cygdrive/e/backup.0/

2) cygrun.bat

This is the file that will startup a bash shell and run our backup script. We’ll be calling this from a scheduled task. The credit goes to Zenovations.com for this script:

@echo off
rem set HOME=c:\
if "%DEF_PATH%"=="" set DEF_PATH=%PATH%
set PATH=c:\cygwin\bin;%DEF_PATH%
set myargs=%*
if "%myargs%" == "" goto noarg
rem echo %myargs%
bash --rcfile "%HOME%/.bashrc" -i -c "%myargs%"
c:
rem pause
sleep 1
goto exit
:noarg

rxvt -e /usr/bin/bash --login -i

:exit
exit

Step 3: Create a scheduled task

Nothing special about this, just set up a standard scheduled task. I’ve set mine to trigger at 4:00 AM every day. The action should look like this:

Step 4: Making it work on Windows 7 Home Premium

If you choose to run this task as a non-administrator user, you will need to ensure that your user account has the “Logon as a batch job” privilege. Unfortunately, Windows 7 Home Premium doesn’t give you an easy way to do this. So get yourself the Windows 2003 Resource Kit and use the ntrights.exe tool that comes with it to assign the privilege from the command line. Credits to Daniel for this tip.

ntrights -u COMPUTER\User +r SeBatchLogonRight

And that, my dear readers, should be it. You should (hopefully) have a fully automated, scheduled, incremental snapshot backup running of your important data.

FileSystemWatcher is (almost) always a bad idea

Thu, 29 Sep 2011 02:38:00 GMT

Using the FileSystemWatcher to watch a folder for changes usually turns out to be a bad idea. From missing notifications to IOExceptions when trying to copy files, I’ve just given up on using FileSystemWatcher to do anything useful. I find that using Quartz.NET to set up a simple scheduled task that polls the folder occasionally is a more reliable and robust solution. Just remember to use stateful jobs to ensure your tasks don’t overlap.

Would you use a Facebook app for project management?

Tue, 24 May 2011 07:49:00 GMT

I asked this question a few days ago on OpinionAided and to my buddies on Facebook. The results of my stunningly non-scientific research are in, and the world apparently thinks it’s a bad idea. Here are the results:

OpinionAided

Yes	12
No	28
Neutral	6
Total votes	46

Facebook

Yes	0
No/not very useful	5
Maybe	5
Total votes	10

The comments on the Facebook poll shows me that the biggest concern is the desire to keep social and work profiles separate. Personally, I agree with this to a large degree. However, I think an app like this would be useful for planning “social projects”, like a camping trip or something like that with your friends and family. There are projects that need management that aren’t necessarily related to work.

What's the distance between elegant and ugly?

Wed, 16 Feb 2011 23:28:00 GMT

4 pixels, apparently.

I have some JavaScript code that uses jQuery to get the height of a web page to determine whether we should do some auto-scrolling as we add new content to the page. I do this by comparing the height of the document and the window like this:

var $win = $(window);
var currentPos = $win.height() + $win.scrollTop();

if (currentPos == $(document).height()) {
    // The user is at the bottom of the page.
    shouldScroll = true;
} else {
    shouldScroll = false;
}

Unfortunately, Internet Explorer 8 acts weird when it comes to comparing the height of window and document. When there's no scroll bar on the page, the height of the document is always 4 pixels more than the height of the window. You can see this happening even on the jQuery documentation for height. Chrome, Firefox and IE9 will report 125px for both document and window height. IE8 will report the window height as 121px.

So, $win.height() + $win.scrollTop() is always 4 pixels less than $(document).height() which causes my scrolling feature to kick in, even when there's no scrolling to do. As a result, I had to turn that elegant piece of code into this monstrosity just because of IE 8:

// This first check is an ugly hack to deal with the fact that
// IE 8 reports document height to be window height + 4 pixels
// in the absence of a scroll bar.
if ($win.scrollTop() == 0) {
    if (currentPos + 4 >= $(document).height()) {
        // User is at the bottom of the page
        scrollResult = true;
    } else {
        scrollResult = false;
    }
} else {
    // Just the following code would be sufficient if it weren't
    // for IE 8.
    if (currentPos == $(document).height()) {
        // The user is at the bottom of the page.
        scrollResult = true;
    } else {
        scrollResult = false;
    }
}