You've reached The Infinite Loop, a blog about programming, entrepreneurship and other interesting things.

  • Demystifying Shippable's deployment keys

    When you enable a project on Shippable, you can see that a key gets added to your repository's "Deployment Keys" on GitHub or BitBucket. However, when you visit your subscription dashboard on Shippable, there's another "Deployment Key" listed there that doesn't match what you see in your repository settings. I'm going to try and clear that up for you in this post.

    tl;dr

    Shippable uses two key pairs:

    1. ~/.ssh/project_rsa and ~/.ssh/project_rsa.pub for git clone, git pull and git fetch operations
    2. ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub for git submodule operations, and any other commands that use SSH which you include in your YML files

    Project keys

    Every time you enable a project, we generate an SSH key and sets that on GitHub as the deployment key for the project. This happens only for private projects because we uses the APIs to fetch the YML file and figure out what we need to build. This is the key you see when you go to the Deployment Keys settings for your project. During the build, this key pair is available at ~/.ssh/project_rsa and ~/.ssh/project_rsa.pub.

    We use this key to do two things:

    1. All git clone/pull/fetch operations that happen in the git_sync step of your build. Because the filenames are not standard key names, we use ssh-agent so git will use these keys when talking to GitHub or BitBucket.
    2. We also use this key pair to encrypt and decrypt secure environment variables in your project. This means that if you ever disable and re-enable a project on Shippable, you will need to re-encrypt the secure environment variables in your YML file.

    Subscription keys (a.k.a "Deployment Keys")

    We also generate a second set of keys for every subscription. This is what you see when you expand the "Deployment Keys" accordion on the subscriptions dashboard. We ask that you use this the "deployment key" when you want to integrate with services like Heroku, and also when you need to pull in submodules or dependencies from other private repositories. These keys are placed at ~/.ssh/id_rsa and /.ssh/id_rsa.pub during the build. This way, when you do a git push heroku or git submodule update you don't need to worry about ssh-agent and the likes. Git will use id_rsa by default and do the right thing.

    Why do we use two key pairs?

    If we didn't do this, you would never be able to pull submodules or other dependencies from other private repositories. On GitHub and BitBucket, a key pair can only be associated with a single project or user account. Once we set the project keys as the deployment key on your repository, you would not be able to re-use the same key on another user account to allow pulling code from other repositories. The only other alternative we had at the time was to add the key automatically to your user account, but we decided that this is too invasive and gives us far more permissions that we actually need. By managing two key pairs and making them both available to the build we're letting you decide when and how to grant the additional access required for submodules and dependencies from other private repositories.

  • Getting Composer to work with GitHub tokens on Shippable

    Composer is a dependency manager for PHP. If you're using it in your projects and running builds on Shippable, you might notice that once in a while your builds fail with a dangerous looking error:

    Could not fetch https://api.github.com/repos/twilio/twilio-php/zipball/51ab2929a66d4455ecb470eafdfe2bab798945ff, 
    enter your GitHub credentials to go over the API rate limit
    
    The credentials will be swapped for an OAuth token stored in /home/shippable/.phpenv/versions/5.4/composer/auth.json, 
    your password will not be stored
    
    To revoke access to this token you can visit https://github.com/settings/applications
    
    Build timed out
    

    This can be a frustrating error to deal with, because Composer is waiting for user input and your build will just wait until it times out or you manually cancel it.

    Wait, my build runs fine locally. Why does this happen only on Shippable?

    Composer downloads dependencies directly from GitHub. By default it does this using unauthenticated requests and GitHub currently allows only 60 unauthenticated requests per hour. Unless you're using dedicated hosts on Shippable, you're running on our shared hosts. It looks like GitHub applies the rate limit to an IP address so if other builds running on the same host have already made 60 calls to GitHub in the last hour, your build is going to fail.

    OK, so how do I get this to work on Shippable?

    Composer lets you configure a GitHub OAuth token. Once you do this, you'll be making authenticated requests to GitHub and your rate limit immediately shoots up to 5,000 requests per hour. Because this is done with your unique token, the rate limit is all yours to consume and doesn't get shared with any other builds that might be running. Let's get started with this.

    Step 1: Create a GitHub token

    Generate a GitHub token. You only need read permission, so it's safe to uncheck all of the scopes for the token.

    Step 2: Protect your token with Shippable environment variable encryption

    Now that you have a token you want to tell composer to use it during your build. However, you shouldn't just check this in to your repository because it's a private token for your account. This is where Shippable's encrypted environment variables come in handy. Head over to your subscription dashboard on Shippable and click on the "Encrypted env vars" link:

    Encrypting environment variables on Shippable

    Define a new environment variable. I'm going to call it MY_GITHUB_TOKEN like so:

    enter image description here

    Step 3: Add the environment variable to your YML file

    Paste the output from the encryption dialog exactly as it is to the env section of your shippable.yml file. When your build runs, this value will be decrypted and made available to your build as an environment variable called MY_GITHUB_TOKEN.

    Step 4: Configure Composer to use your token

    The last step is to tell Composer to use your token. Just add this to your before_install section and you're done:

    before_install:
      - composer config -g github-oauth.github.com $MY_GITHUB_TOKEN
    

    That's it. Composer will now use your personal token when fetching dependencies from GitHub.

  • Looking back at 2013

    2013 went by quickly, and what a year it has been. Three massive things that I want to reflect on for this year.

    1. Shippable goes from "cool idea" to $2M in funding in less than a year

    Avi and I started talking about building a continuous integration service somewhere in December of 2012. Since then, we've written and rewritten everything four times, slowly making progress towards a better product-market fit. We started out by using Jenkins as the core of our platform but we realized very soon that it wasn't going to let us innovate fast enough, at which point we built our own CI back end. That was a lot of fun to build.

    We got cross platform builds running using a solid distributed architecture that allowed us to run builds 3x faster than the competition. It was around this time that we got in to TechStars Seattle, which turned out to be a massive learning experience for all of us. They don't call it an accelerator for nothing -- the amount of stuff we got done in the three months at TechStars was phenomenal.

    Along the way, it became clear to us that Docker was going to change the way the world thinks about development, testing and deployment software. That's when we decided to start moving our own infrastructure to Docker. Today, our customers builds happen inside Docker containers and we're completely embracing an entire development workflow based around Docker. It's an exciting time to be a developer and I'm thrilled to be a part of this team.

    2. I've moved to developing full time in Javascript on NodeJS

    I love .NET. C# is, without a doubt, the best language that I've ever worked with, but I'm very worried about Windows. It's becoming painfully clear that there's almost no one other than Microsoft that's driving innovation on the Windows platform. All the cool toys seem to be on the other side of the fence and that makes me very sad. It's just disappointing that Microsoft, with all its abilities, has simply allowed the ecosystem to rot away.

    As a developer, it's hard to ignore all the awesomeness that's going on in the open source world. As much as I love C# and Visual Studio, there's something to be said about stuff like NodeJS, Python, Docker and Vim. Granted, some of those things can also be used on Windows, but the truth is Windows simply isn't a first class citizen of the open source world and all the interesting stuff happens on Linux first anyway.

    I've now learned to love my set up of tmux and Vim (+ plugins, of course). I'm not as productive as I used to be with Visual Studio yet, but it's not as scary as it seems and I'm getting a little better every day.

    3. Oh, and one more thing...

    Now this is going to come as a complete surprise to everyone because I've been really, really quiet about it so far. It isn't that I wanted to keep it quiet, but the truth is that this little project has kept me so busy for the past few weeks that I just haven't had the time to go announcing it to the world.

    Folks, meet Aradhya.

    Aradhya Krishna

    The bundle of awesomeness that's been keeping Niree and I busy since the 7th of December.

    I don't think the enormity of parenting has descended on me yet, but I do know this -- when I'm looking at her, time slows down, everything is alright with the world and it all makes sense.

    2014, here we come!

    Standing here at the cusp of another new year, I'm filled with hope, enthusiasm and excitement. 2013 was one hell of a ride and I'm sure 2014 will be filled with even more ups, downs and everything in between! Here's wishing everyone a fantastic new year filled with all the best that life has to offer. :)

  • Pro tip: Don't use a generic email address to promote your business

    How often do you see a proposal that goes like this:

    Hey there,

    At XYZ we do awesome stuff around something. Check us out at xyz.com!

    Thanks,
    xyz@gmail.com

    Really? It only takes 10 minutes to set up email on your own domain. If you really don't have the time to do that, then I'm not too sure I want to give you money to do things for me.

  • You really should

    Procrastination is the root of all laziness. It's easy to come up with a million reasons why you shouldn't do something.

    • Excercise? Too hard. Tomorrow, maybe.
    • Eat healthy? Just one more piece of cake.
    • Save money? Ooooh, XBox!
    • Start a company? Hmmm, maybe in a few years when I'm really "ready".

    So why should you do any of this right now? Simple. You have more time and energy to do these things today than you will have tomorrow.

    Let it sink in for a minute. Now go do something.

  • Asynchronous S3 file uploads with async/await

    The async/await pattern in C# really is awesome. Perhaps it is just syntactic sugar over the TPL, but it makes the code so much more beautiful and easier to read. Before async/await came around this would have been an ugly mess of BeginXXX/EndXXX. Instead, now we get to do this:

    var s3 = AWSClientFactory.CreateAmazonS3Client(accessKey, secretKey);
    PutObjectRequest putRequest = new PutObjectRequest()
    {
        FilePath = "your-file.txt",
        Key = "whatever/you/want",
        BucketName = "your-bucket-name",
    };
    
    PutObjectResponse response = await Task
        .Factory
        .FromAsync<PutObjectRequest, PutObjectResponse>(s3.BeginPutObject, s3.EndPutObject, putRequest, null);
    

    That is so much easier on the eyes, and that's just the wrapper around the traditional BeginXXX/EndXXX calls that the Amazon SDK gives us. If you're using the newer stuff like HttpClient, it's just one line to async goodness:

    HttpResponseMessage response = await client.GetAsync("http://bing.com");
    
  • Plain-text passwords: Will it ever end?

    Sometimes you just have to wonder how long computer security professionals will have to keep screaming “Don’t store passwords in plain-text” before people take notice. Getting an email like this from MasterCard really freaks me out:

    MasterCard

    I wonder if they store PINs in plain-text, too. It’s just sad when a credit card company’s policies allow something like this to get pushed in to production.

  • Yes, SIM Cards Must Die

    I like this idea.

    ilyabirman.net

    The author thinks carriers won’t support it because it will prevent them from locking in customers. I think they should support it because it also prevents competitors from locking in customers.

    Read the original post at http://ilyabirman.net/meanwhile/2012/03/23/1/

  • Opening hours

    I really wonder why the Indian Railways, arguably one of the most active e-commerce sites in the country, needs to be offline for an hour every day.

    image

    Seriously!?

    I run a few web apps of my own and I’ve had only 50 minutes of downtime in an entire month. Yes, I know my app is nowhere near as complex as a railway reservation system. And yes, I know, those of you that are measuring up time in 5 nines are laughing at me, but give me a break – my services are in beta. Still, why does the IRCTC need an hour off every single day? Sure, as a customer, I’m somewhat inconvenienced, but as a developer I’m really curious as to what they’re doing in that hour.

  • Turn off the complication!

    Surely, we’d all be better off if we would just turn off “complications” in our system settings.

    image