Dr Nic

Creating new BOSH releases with bosh-gen and bosh-solo

Below is a 45-minute walk-thru tutorial of creating and deploying a BOSH release for a Redis VM. It introduces a brand new tool: bosh-solo.

A BOSH release describes a complete running system from the ground up – compiled packages from source, templated configuration files, and monit to start/stop processes. BOSH itself then allows you to deploy your release across 1 or more VMs with optional persistent disks on the target infrastructure of your choice.

With bosh-solo you can develop and test BOSH releases without BOSH itself. Develop an entire working system quickly before you deploy to production.

“Just wanted to share a small personal achievement, last night I was able to
take a release I had created (actually, the redis-on-demand from your
tutorial) and bring it up using bosh-solo, connect to it, set/get, etc.

“I’ve gotta say, this is VERY slick. Awesome work on this.” – Brian McClain, creator of BoshDB.

BOSH: What, How, When

I’m still very bullish on BOSH. I’ve been experimenting with it internally at work, looking to see how it could duplicate or improve upon our current infrastructure, automation and release management. I’ve also watched the commits that have come out in the last month and I’m excited by the project velocity and direction.

A few weeks ago I was fortunate to be invited to LinkedIn to the SV Forum group to give an introduction to BOSH. What it is, why it was created, how to use it and when you might use it.

This talk also includes cutaways to Vadim Spivak from the Cloud Foundry & BOSH core teams, who has many very interesting things to say about BOSH and the future of BOSH.

BOSH: What, How, When from Dr Nic on Vimeo.

This talk is one hour. If you only have 10 minutes, there is a good introduction upfront. Or you could read the slides first to decide if you want to watch the talk.

I’ve rewatched this talk. Given all the things I’ve learnt about using it and living it, for the most part I still agree with myself and what I said. That’s handy.

It was two days after RedHat had released OpenShift, so I had a few bonus comments to make about OpenShift towards the end of the talk. I’m also some what liberal with making jokes about any other technology that comes into my mind at the time. A politically correct person might have said different things.

At the time of the talk, I was not able to give a live demo of BOSH. I think such a demo would be valuable to get a complete “what is BOSH?” understanding, since it was a 5 minute demo that sold me on it.

Thanks to LinkedIn for recording the talk and for letting me share the video, and to LinkedIn’s Dan Lujan for doing the great post-production work. It’s a great room they have there for giving talks, and with all the cameras in the room they got some great shots of Vadim answering questions.

Creating a BOSH from scratch on AWS

BOSH architecture

A lot of devops projects revolve around managing instances/VMs once they already exist. For example, Chef and Puppet do configuration management of instances once the instances have been created. The “monitoring sucks” work is concerned with monitoring the contents/jobs/processes of instances and the instances themselves. Libraries such as fog and jclouds dedicate themselves to provisioning instances and other cloud resources.

BOSH attempts to do something that I haven’t seen in OSS yet. It wants to allow a devops team to describe the entire software stack – from the base operating system image used to boot new instances (stemcell), to the software packages installed on instances (packages), to the processes that are run on each instance (jobs). It wants to allow a devops team to describe the runtime deployment of your stack – which instance types are used, how many of them, how much disk is attached, and how networking properties such as IPs are configured.

More than that, BOSH then wants to allow you to change the configuration – change instance types (scale vertically), how many of them (scale horizontally), upgrade packages, and even upgrade the base image – and deploy all those changes/deltas to the running system.

That’s what it wants to do.

And then I saw a demo of it doing it. This demo was running against vSphere. There was a running deployment of instances, which I could also see in vCenter. A simple YAML configuration file was changed – he changed the instance specification to add more CPUs to each instance – and ran bosh deploy.

Then I watched vCenter light up as it reflectively showed that disks were being detached; instances were being torn down; new, bigger CPU instances being booted, and the disks reattached.

I needed one of these BOSH things immediately. It has some good high-level and low-level documentation

But first, I needed a BOSH. And I needed a BOSH that could talk to AWS instead of vSphere.

Below is a screencast of creating your own BOSH on AWS. The written instructions are also available.

Thanks to a whole bunch of CloudFoundry folk who wrote BOSH and also helped me figure out how to set it up: Oleg, Vadim, skaar, Martin, and Mahesh. Also to Mark, Dave, Matt and James for sharing and caring when I visited the team.

What’s next? How do I use BOSH?

Once you get a BOSH created, I’ve started some other tutorials for how to use BOSH as I figure it out. Perhaps it could become a handbook or guide in the future. If you find any bugs, or have suggestions, please drop a ticket in the Issues.

At the time of writing, there are the following tutorials/guides: