Dr Nic

Creating new BOSH releases with bosh-gen and bosh-solo

Below is a 45-minute walk-thru tutorial of creating and deploying a BOSH release for a Redis VM. It introduces a brand new tool: bosh-solo.

A BOSH release describes a complete running system from the ground up – compiled packages from source, templated configuration files, and monit to start/stop processes. BOSH itself then allows you to deploy your release across 1 or more VMs with optional persistent disks on the target infrastructure of your choice.

With bosh-solo you can develop and test BOSH releases without BOSH itself. Develop an entire working system quickly before you deploy to production.

“Just wanted to share a small personal achievement, last night I was able to
take a release I had created (actually, the redis-on-demand from your
tutorial) and bring it up using bosh-solo, connect to it, set/get, etc.

“I’ve gotta say, this is VERY slick. Awesome work on this.” – Brian McClain, creator of BoshDB.

BOSH: What, How, When

I’m still very bullish on BOSH. I’ve been experimenting with it internally at work, looking to see how it could duplicate or improve upon our current infrastructure, automation and release management. I’ve also watched the commits that have come out in the last month and I’m excited by the project velocity and direction.

A few weeks ago I was fortunate to be invited to LinkedIn to the SV Forum group to give an introduction to BOSH. What it is, why it was created, how to use it and when you might use it.

This talk also includes cutaways to Vadim Spivak from the Cloud Foundry & BOSH core teams, who has many very interesting things to say about BOSH and the future of BOSH.

BOSH: What, How, When from Dr Nic on Vimeo.

This talk is one hour. If you only have 10 minutes, there is a good introduction upfront. Or you could read the slides first to decide if you want to watch the talk.

I’ve rewatched this talk. Given all the things I’ve learnt about using it and living it, for the most part I still agree with myself and what I said. That’s handy.

It was two days after RedHat had released OpenShift, so I had a few bonus comments to make about OpenShift towards the end of the talk. I’m also some what liberal with making jokes about any other technology that comes into my mind at the time. A politically correct person might have said different things.

At the time of the talk, I was not able to give a live demo of BOSH. I think such a demo would be valuable to get a complete “what is BOSH?” understanding, since it was a 5 minute demo that sold me on it.

Thanks to LinkedIn for recording the talk and for letting me share the video, and to LinkedIn’s Dan Lujan for doing the great post-production work. It’s a great room they have there for giving talks, and with all the cameras in the room they got some great shots of Vadim answering questions.

Creating a BOSH from scratch on AWS

BOSH architecture

A lot of devops projects revolve around managing instances/VMs once they already exist. For example, Chef and Puppet do configuration management of instances once the instances have been created. The “monitoring sucks” work is concerned with monitoring the contents/jobs/processes of instances and the instances themselves. Libraries such as fog and jclouds dedicate themselves to provisioning instances and other cloud resources.

BOSH attempts to do something that I haven’t seen in OSS yet. It wants to allow a devops team to describe the entire software stack – from the base operating system image used to boot new instances (stemcell), to the software packages installed on instances (packages), to the processes that are run on each instance (jobs). It wants to allow a devops team to describe the runtime deployment of your stack – which instance types are used, how many of them, how much disk is attached, and how networking properties such as IPs are configured.

More than that, BOSH then wants to allow you to change the configuration – change instance types (scale vertically), how many of them (scale horizontally), upgrade packages, and even upgrade the base image – and deploy all those changes/deltas to the running system.

That’s what it wants to do.

And then I saw a demo of it doing it. This demo was running against vSphere. There was a running deployment of instances, which I could also see in vCenter. A simple YAML configuration file was changed – he changed the instance specification to add more CPUs to each instance – and ran bosh deploy.

Then I watched vCenter light up as it reflectively showed that disks were being detached; instances were being torn down; new, bigger CPU instances being booted, and the disks reattached.

I needed one of these BOSH things immediately. It has some good high-level and low-level documentation

But first, I needed a BOSH. And I needed a BOSH that could talk to AWS instead of vSphere.

Below is a screencast of creating your own BOSH on AWS. The written instructions are also available.

Thanks to a whole bunch of CloudFoundry folk who wrote BOSH and also helped me figure out how to set it up: Oleg, Vadim, skaar, Martin, and Mahesh. Also to Mark, Dave, Matt and James for sharing and caring when I visited the team.

What’s next? How do I use BOSH?

Once you get a BOSH created, I’ve started some other tutorials for how to use BOSH as I figure it out. Perhaps it could become a handbook or guide in the future. If you find any bugs, or have suggestions, please drop a ticket in the Issues.

At the time of writing, there are the following tutorials/guides:

Instant new Rails applications with the App Scrolls

When I start a new project I want to start how I plan to finish.

If I intend to write integration tests then I want to start them immediately. If I intend to set up continuous integration then I want CI setup immediately. If I intend to deploy the app somewhere then I want to deploy it immediately. If I intend to use some background workers, I want to know that they are setup immediately.

I want it all setup as easily as I can generate a new Rails app. They shouldn’t be extra work later. I’ll be busy then.

Assuming you want the same simplicity and speed, then let me share with you the App Scrolls!

gem install appscrolls
scrolls new mydemoapp -s twitter_bootstrap unicorn postgresql resque github eycloud

This one command will:

  • create a new Rails application called “mydemoapp”
  • install Twitter Bootstrap (you choose fixed or floating) and some other Rails basic tweaks
  • sets up Unicorn and Postgresql in the application
  • sets up Resque and Redis for background workers
  • hosts the project on GitHub (public or private repo)
  • deploys the application on Engine Yard Cloud

These are the magical App Scrolls!

Can you run the command above, or do you need a video to see it happen? Let me know.

Scrolls are interactive

$ scrolls new myapp -s twitter_bootstrap github resque postgresql unicorn eycloud
...
   question  Create a GitHub repository?
          1)  Public
          2)  Private
      github  Enter your selection: 1
...
      resque  Enter a secret string for the route /resque/YOUR-SECRET-STRING: mysecret
...
    question  Which Twitter Bootstrap layout?
          1)  Fluid
          2)  Fixed
  twitter_bootstrap  Enter your selection: 2
...
    question  Select application cluster configuration?
          1)  Basic - 1 app VM (5 CPU-based processes) & DB Master VM
          2)  Pro   - 3 app highly-available VMs (15 CPU-based processes) & DB Master VM
          3)  Solo  - 1 VM for app processes, DB and other services
     eycloud  Enter your selection: 1

Deploy automatically

I perfer to host on Engine Yard Cloud and I work there. So I added support to Engine Yard Cloud to all the scrolls and it will automatically boot a new environment and deploy your new application. Thanks to David Calavera’s unofficial ey_cli tool and the unofficial github-ruby CLI for making this possible!

Which Scrolls are supported?

Lots of magical scrolls are available!

administration: active_admin
assets: jquery, prototype
deployment: eycloud, eycloud_recipes_on_deploy, git, github, passenger, unicorn
persistence: mysql, postgresql, redis, sqlite3
stylesheet: twitter_bootstrap
templating: simple_form
testing: capybara, cucumber, rspec, test_unit
worker: delayed_job, resque
other: env_yaml, guard, rails_basics, split

I might blog about individual combinations later. My favourite is guard, which magically combines with other scrolls you choose.

Heroku or CloudFoundry?

If you’re a fan/employee of Salesforce/Heroku or VMWare/CloudFoundry or other platform, can I please ask for your help to make the current set of scrolls work? I’ll pair with you to get you started if you need it; though I think the code base and the scrolls are relatively self explanatory.

Isn’t this like Rails Wizard?

Yes! Before the App Scrolls, there was Michael Bleigh’s Rails Wizard. He created the first version during the 2010 Rails Rumble [Intridea blog post], and iterated on it a few times. I especially like that he extracted the project into two codebases – the CLI/scrolls and the web version.

I renamed it because ultimately I think it could support more than Rails applications and it definitely should become more than a “new application wizard”. And because I’ve been playing Elder Scrolls. Fortunately, Michael convinced me not to name the project after the computer game. Thanks to Thom Mahoney for suggesting “App Scrolls” and saving me from a naming nightmare.

A new name and a new repository? I’m still in two minds as to if I should have done that. Oh well, its done now. App Scrolls is a cool name.

Why “Scrolls”?

Rails Wizard used the word “recipe” to describe each component or service that would be added to a new application. Talking with others at Engine Yard, the word “recipe” got confusing. There were wizard recipes and chef recipes. When you wanted to use Resque you needed both. “Scrolls!” seemed a fun alternative name.

Wizards use scrolls, not recipes. Every D&D person knows that.

Where are all the old scrolls?

For the initial release, I’ve moved many contributed “recipes” from the Rails Wizard into the scrolls/zzz archive folder. There are so many and I don’t yet know how to ensure they all work as expected. I intend to find a good way to allow rubygems to self-host their scroll.

Continuous Integration?

I mentioned that I like CI to be setup immediately for new applications. That’s next. Jenkins, Travis CI (when they release private application support), and what other CI systems should be supported?

I hope you use the App Scrolls for your next big or small Rails app and that its useful to you!

How to stop killing people with your public speeches

Let’s do the math.

If you give a speech to 200 people for 30 minutes you are consuming 100 hours of human life.

Giving an hour-long talk to a thousand people? That’s six weeks of human life devoted to your talk.

*gulp*!

Let’s assume 6 weeks of human life is at stake. It is not a loan and you cannot give it back. One hour after you finish speaking, you’ve used up 6 weeks of human life.

If you’re bad enough for long enough you kill a whole person.

Mathematically, if 1000 people witness your speech and 0 of them change any aspect of their life then you should not have given the talk. You used up 6 weeks of human life and achieved nothing.

Let’s all pledge together:

  • I am valuable
  • I will continue to share my ideas and my work in public
  • I will continue to improve the craft of sharing publicly (for example, speech craft and blogging)
  • I will respect the time of other people as a gift
  • I will change lives

This pledge needs a name; anyone?

You are an incredible individual. You have done things, seen things, diagnosed things, solved things that no one else has done. No one else has the accumulated knowledge and reasoning that you have. You were the one who conclusively deduced that all solutions to World Peace involved chocolate biscuits.

You are also unique in your methods of communication. You have different stories, different mental models, and different approaches to communication. You do that thing with your voice and your posture. As a child you were beaten up for being different. As an adult it makes you extremely valuable.

There are human beings who need to have their lives changed by you. Your value is when other people behave differently because of you.

What to do next?

There is no check box, pass or fail, “crossing the finish line” in public speaking. I’d rather watch you play Guitar Hero. Aim higher. Take responsibility for your communication. Look at the human beings in front of you and rock their world.

How?

Like the craft of software development, there is the craft of communication.

Practise the craft of communication. Limit the minutes/hours/days/weeks that you use of other people’s lives. Write your speeches and then practise them to increasingly larger audiences.

“I’m not very good at speeches”

Write small speeches for small audiences. As you improve, increase the size of the audience. If it’s necessary to achieve your goal, increase the duration of your talk. Though how often can you communicate 80% of your message in just 5 to 15 minutes?

If you are given 45 minutes at a conference to speak, you are permitted by me to speak for 15 minutes only. If you know you only have 15 minutes, perhaps invite one or two other people to share during the remaining time. Or give back the remaining time to the audience. Ask them to use it to think about or explore the idea you’ve shared with them.

“Where can I find small audiences?” If you’re practising a large audience speech, then pull a handful of people into a room and they can alpha test your talk. As a small group they’ll probably do something annoying like give feedback. You’re probably self-conscious enough that your brain is full of negative self-feedback, so you may or may not want their feedback. But let them give it to you. “Thanks, that’s a good idea” is the reward you give them back.

Next, find another small audience. The jokes won’t be as funny in small audiences. Large audiences laugh louder and stronger. Do the talk again.

Follow up with the first group – did they do anything different in their lives after your talk? Did they talk about your ideas or thoughts during the week? If your speech is good then some small nugget will carry with them, regardless that they are a friend/peer/family member.

Don’t worry too much if you don’t rock their world: friends and peers tend to not learn as much from you. “You can’t be a guru in your home town.” That’s not the job of a friend. Friends are there to laugh at you and put you in embarrassing situations for their own amusement. Nonetheless, it’s interesting to see if your small friendly audiences thought about your talk in the days afterwards.

Even in a small room with friends, unload on them with energy and passion. Practise your communication as much as you practise your content.

“Where can I learn speech craft?”

Toastmasters.

You learnt to drive with your parents and a driving instructor. You learnt software development by writing small demo apps and pair programming. And yet some people have the audacity to stand on stage in front of 100s of people and be terrible. Those people are not you. You will not suck. You are going to learn the craft of speaking because you want to use people’s time well and you want to change their life.

Toastmasters is a club format. There is no “school” concept. Rather a group of 20 people meet twice a month and each time they all suck less. They stop stuttering. They tell interesting stories. Everyone learns something fascinating from other members who have different backgrounds and professions.

The practise format is interesting and successful. You give three types of speeches: 5-7 minute prepared talks, 1-3 minute impromptu talks, and 2-3 minute improptu evaluation talks about the previous speakers. That’s right, you get immediate evaluation from another club member!

Every club is different so visit a few of them to find a club that you like. Some a filled with old farts who value meeting protocol over having a good time. Some clubs like doing competitions. Some clubs meet weekly. Some clubs meet for breakfast or lunch or in the evening.

It’s relatively cheap.

You will learn the craft of speaking and leadership. And conversely, though not explicitly promised in any Toastmasters literature, you’ll stop killing people.

Is it really that important?

Personally, I think its a bit unfair of a presenter to be bad in front of a large audience for a long duration and say “I’m sorry if I was bad. I’m trying to get better.”

But! It is far far worse is the competent presenter who does no preparation, does a terrible job, and merely hopes it goes better next time. There are human lives at stake. One of them might be mine.

As a public speaker you have a gift – human lives. You have their time and their open minds. Aim high. Change lives. You are special. I hope to experience you speaking awesomely at a conference near me soon!

Thanks

Thanks to Zach Holman for our chat on the plane back from Ruby MidWest and for proof reading and fixing the post. He is a great speaker and will definitely convince you of whatever snake oil he’s selling rock your world. Just look at these slides!