Suppose your neo database contains suppliers for custom t-shirt printing. Some suppliers have a minimum order quantity, and you want to quickly lookup suppliers that would accept a given quantity. This is easy with the Embedded Neo4J server in Java:

1
2
3
4
5
6
7
8
9
10
11
12
13

  // Add to an index
  Index<Node> suppliers = graphDb.index().forNodes("suppliers");
  suppliers.add(someSupplier, "minimum-order", new ValueContext(5).indexNumeric());

  // Test query with a match
  QueryContext queryContext = QueryContext.numericRange("minimum-order", null, 8);
  IndexHits<Node> potentialSuppliers = suppliers.query(queryContext);
  assertEquals(1, potentialSuppliers.size());
  assertEquals(someSupplier, potentialSuppliers.getSingle());
  // Test query without a match
  queryContext = QueryContext.numericRange("minimum-order", null, 4);
  potentialSuppliers = suppliers.query(queryContext);
  assertEquals(0, potentialSuppliers.size());

However, those numeric ranges are not supported by Neo4J REST server. It’s probably best to write a plugin for this, but if you cannot use custom plugins (for example, I don’t think the Heroku Neo4J Add-On allows custom plugins) then the Neo4J Gremlin Plugin may be your best choice.

Here’s how you would do the same thing if you’re using Ruby’s Neography:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

class NeoQuery
  GREMLIN_NUMERIC_INDEX_TEMPLATE = <<eos
    import org.neo4j.index.lucene.*;
    neo4j = g.getRawGraph();
    tx = neo4j.beginTx();
    idxManager = neo4j.index();
    cuIndex = idxManager.forNodes(index_name);
    node = neo4j.getNodeById(Long.parseLong(node_id));
    cuIndex.add(node, key_name, new ValueContext(value).indexNumeric());
    tx.success();
    tx.finish();
eos

  GREMLIN_NUMERIC_QUERY_TEMPLATE = <<eos
    import org.neo4j.index.lucene.*;
    neo4j = g.getRawGraph();
    idxManager = neo4j.index();
    cuIndex = idxManager.forNodes(index_name);
    cuIndex.query(QueryContext.numericRange(key_name, null, value, false, false));
eos

  @@neo = Neography::Rest.new(ENV["NEO4J_URL"] || "http://localhost:7474")

  def self.index_numeric(index_name, node, key_name, value)
    @@neo.execute_script(GREMLIN_NUMERIC_INDEX_TEMPLATE, {
      :index_name => index_name,
      :key_name => key_name,
      :node_id => node,
      :value => value})
  end

  def self.query_numeric(index_name, key_name, value)
    @@neo.execute_script(GREMLIN_NUMERIC_QUERY_TEMPLATE, {
      :index_name => index_name,
      :key_name => key_name,
      :value => value})
  end
end

# Add to index
NeoQuery.index_numeric('suppliers', your_node.neo_id, :minimum_order_size, 5)
# Test query with a match
potential_suppliers = NeoQuery.query_numeric('suppliers', :minimum_order_size, 8)
potential_suppliers.size.should == 1
Neography::Node.load(potential_suppliers[0]).neo_id.should == your_node.neo_id
# Test query without a match
potential_suppliers = NeoQuery.query_numeric('suppliers', :minimum_order_size, 4)
potential_suppliers.size.should == 0

Disclaimer: I have not tested behavior in a clustered environment and you should consider security before using execute_script.

How often do you commit your changes? How often do you upgrade your dependencies?

Commit Often

There is a common answer to the first question:

Everyone Commits To the Mainline Every Day

Martin Fowler Continuous Integration

Code is integrated and tested after a few hours-a day of development at most.

Kent Beck Extreme Programming Explained

We recommend that you aim to commit changes to the version control system at the conclusion of each separate incremental change or refactoring. If you use this technique correctly, you should be checking in at the very minimum once a day, and more usually several times a day.

Jez Humble and David Farley Continuous Delivery

This makes sense: small frequent merges are easier to integrate than large, occasional ones. The less often you commit (to a shared “trunk” branch) the more painful integration becomes. You find problems too late, and spend a lot of time in integration or merge hell: performing painful merges or grepping through large changesets to find the needle in the haystack that broke the app.

Upgrade Often

The same logic applies to upgrading dependencies. If you upgrade frequently, each individual upgrade is usually quick and painless. Yet there is less consensus about how often to upgrade.

I’m a fan of small, frequent upgades. I consider any outdated library to be tech debt. You don’t always need to address it right away but you should review it and make an informed decision. This is the difference between Reckless/Inadvertant debt and Prudent/Deliberate debt.

I’m not a fan of Long-Term Support (LTS) releases either. It means you’re setup up for a major upgrade, you’re missing out on faster/safer/cooler software. Your environment probably isn’t as static as you pretend either - why would you use a locked version of Selenium/WebDriver unless you’ve turned off Firefox/Chrome updates.

Uninformed Pessimism

Can a team simultaneously be “bleeding edge” and a “late adopter”?

I’ve seen it. I was on a team that had used an a beta release of a library, then skipped the next few stable releases. When we finally upgraded a method was missing. I dug through the release notes for several versions, looking for a deprecation notice with a recommended alternative. I never found one - because the method never made it out of the beta!

The team did not choose to do this - they were simple uninformed. They’d been inadvertently reckless. This is a trap you can fall in even if you don’t use beta versions. If you don’t have a good dependency report that shows what you’re using and what upgrades are available then you are uninformed. You could try to manually assemble a report, but that is impractical on large projects (most Java projects), or projects with lots of small, frequently released libraries (most Ruby projects).

Informed Pessimism

The first step towards better dependency management is becoming is “Informed Pessimism”: generate and regularly review a report of available upgrades. Many package managers have a report or command you can use:

• mvn versions:dependency-updates-report
• bundle_outdated
• gem outdated
• yum list updates
• apt-get -s dist-upgrade

Bundler usually beats Maven, but in this case the Maven plugin generates a nice report I can display from a CI server. Here’s a live examle.

Cautious Optimism

Alex Chaffee proposed Cautious Optimism as the ideal. A Cautious Optimism build system will attempt to upgrade dependencies as soon as possible.

Cautious Optimism isn’t for everyone, but if you have good dependency management and a Continuous Delivery setup you can trust your pipeline to catch problems introduced by an upgrade. If a problem is found, you lock the dependency to the last-known good version until a solution is found. Some tools that are useful in implementing Cautious Optimism:

• Jenkins Maven Dependency Update trigger
• mvn versions:use-latest-versions
• bundler-auto-update

Matrix Builds

If your team can release frequently it can probably upgrade frequently. The only exceptions may be upgrades to large frameworks - Java/Spring, Ruby/Rails, etc. It may be prudent to delay an upgrade even if its possible - just because it means a long downloads and a lot of knew features to study.

It is possible to get the feedback without actually committing to an upgrade. I view this as using your CI/CD setup to answer two questions:

• Is the app releasable? (Test with fixed dependencies)
• Is the app upgradable? (Test with latest dependencies)

I’d probably do the upgradable check less often. The releasabilty checks should be every commit, but you can probably check for upgradability nightly.

Conclusion

Most Agile teams believe in the “commit often” motto. You should remember that everyone is supposed to commit often. Unless everyone has integrated recently, someone on the team may still be headed towards merge hell.

I consider third-party library developers to be part of the extended team. Unfortunately these developers cannot push their own changes and cannot resolve the conflicts. The core team needs to be proactive about reviewing and pulling upgrades as often as practical. Every day may not be practical, but try to avoid skipping releases.

One key goal of continuous deployment is to reduce the risk of releasing software. Counter-intuitively, increased throughput and increased production stability are not a zero-sum game, and effective continuous deployment actually reduces the risk of any individual release. In the course of teaching continuous delivery and talking to people who implement it, I’ve come to see that “doing it right” can be reduced to four principles:

• Low-risk releases are incremental.
• Decouple deployment and release.
• Focus on reducing batch size.
• Optimize for resilience.

Read the rest of this article on InformIT (free, no registration required)

Translations: 中文

Continuous delivery is a software development strategy that optimizes your delivery process to get high-quality, valuable software delivered as quickly as possible. This approach allows you to validate your business hypothesis fast and then rapidly iterate by testing new ideas on users. Although Continuous Delivery focuses on engineering practices, the concept of continuous delivery has implications for the whole product-delivery process, including the “fuzzy front end” and the design and analysis of features.

Read the rest of this article on InformIT (free, no registration required)

In working with development teams at organizations which are adopting Continuous Delivery, I have found there can be friction over practices that many developers have come to consider as the right way for Agile teams to work. I believe the root of conflicts between what I’ve come to think of as traditional agile and CD is the approach to making software “ready for release”.

Evolution of software delivery

A usefully simplistic view of the evolution of ideas about making software ready for release is this:

• Waterfall believes a team should only start making its software ready for release when all of the functionality for the release has been developed (i.e. when it is “feature complete”).
• Agile introduces the idea that the team should get their software ready for release throughout development. Many variations of agile (which I refer to as “traditional agile” in this post) believe this should be done at periodic intervals.
• Continuous Delivery is another subset of agile which in which the team keeps its software ready for release at all times during development. It is different from “traditional” agile in that it does not involve stopping and making a special effort to create a releasable build.

Continuous Delivery is not about shorter cycles

Going from traditional Agile development to Continuous Delivery is not about adopting a shorter cycle for making the software ready for release. Making releasable builds every night is still not Continuous Delivery. CD is about moving away from making the software ready as a separate activity, and instead developing in a way that means the software is always ready for release.

Ready for release does not mean actually releasing

A common misunderstanding is that Continuous Delivery means releasing into production very frequently. This confusion is made worse by the use of organizations that release software multiple times every day as poster children for CD. Continuous Delivery doesn’t require frequent releases, it only requires ensuring software could be released with very little effort at any point during development. (See Jez Humble’s article on Continuous Delivery vs. Continuous Deployment.) Although developing this capability opens opportunities which may encourage the organization to release more often, many teams find more than enough benefit from CD practices to justify using it even when releases are fairly infrequent.

Friction points between Continuous Delivery and traditional Agile

As I mentioned, there are sometimes conflicts between Continuous Delivery and practices that development teams take for granted as being “proper” Agile.

Friction point: software with unfinished work can still be releasable

One of these points of friction is the requirement that the codebase not include incomplete stories or bugfixes at the end of the iteration. I explored this in my previous post on iterations. This requirement comes from the idea that the end of the iteration is the point where the team stops and does the extra work needed to prepare the software for release. But when a team adopts Continuous Delivery, there is no additional work needed to make the software releasable.

More to the point, the CD team ensures that their code could be released to production even when they have work in progress, using techniques such as feature toggles. This in turn means that the team can meet the requirement that they be ready for release at the end of the iteration even with unfinished stories.

This can be a bit difficult for people to swallow. The team can certainly still require all work to be complete at the iteration boundary, but this starts to feel like an arbitrary constraint that breaks the team’s flow. Continuous Delivery doesn’t require non-timeboxed iterations, but the two practices are complementary.

Friction point: snapshot/release builds

Many development teams divide software builds into two types, “snapshot” builds and “release” builds. This is not specific to Agile, but has become strongly embedded in the Java world due to the rise of Maven, which puts the snapshot/build concept at the core of its design. This approach divides the development cycle into two phases, with snapshots being used while software is in development, and a release build being created only when the software is deemed ready for release.

This division of the release cycle clearly conflicts with the Continuous Delivery philosophy that software should always be ready for release. The way CD is typically implemented involves only creating a build once, and then promoting it through multiple stages of a pipeline for testing and validation activities, which doesn’t work if software is built in two different ways as with Maven.

It’s entirely possible to use Maven with Continuous Delivery, for example by creating a release build for every build in the pipeline. However this leads to friction with Maven tools and infrastructure that assume release builds are infrequent and intended for production deployment. For example, artefact repositories such as Nexus and Artefactory have housekeeping features to delete old snapshot builds, but don’t allow release builds to be deleted. So an active CD team, which may produce dozens of builds a day, can easily chew through gigabytes and terabytes of disk space on the repository.

Friction point: heavier focus on testing deployability

A standard practice with Continuous Delivery is automatically deploying every build that passes basic Continuous Integration to an environment that emulates production as closely as possible, using the same deployment process and tooling. This is essential to proving whether the code is ready for release on every commit, but this is more rigorous than many development teams are used to having in their CI.

For example, pre-CD Continuous Integration might run automated functional tests against the application by deploying it to an embedded application server using a build tool like Ant or Maven. This is easier for developers to use and maintain, but is probably not how the application will be deployed in production.

So a CD team will typically add an automated deployment to an environment will more fully replicates production, including separated web/app/data tiers, and deployment tooling that will be used in production. However this more production-like deployment stage is more likely to fail due to its added complexity, and may be may be more difficult for developers to maintain and fix since it uses tooling more familiar to system administrators than to developers.

This can be an opportunity to work more closely with the operations team to create a more reliable, easily supported deployment process. But it is likely to be a steep curve to implement and stabilize this process, which may impact development productivity.

Is CD worth it?

Given these friction points, what benefit is there to moving from traditional Agile to Continuous Delivery worthwhile, especially for a team that is unlikely to actually release into production more often than every iteration?

• Decrease risk by uncovering deployment issues earlier,
• increase flexibility by giving the organization the option to release at any point with minimal added cost or risk,
• Involves everyone involved in production releases - such as QA, operations, etc. - in making the full process more efficient. The entire organization must identify difficult areas of the process and find ways to fix them, through automation, better collaboration, and improved working practices,
• By continuously rehearsing the release process, the organization becomes more competent at doing it, so that releasing becomes autonomic, like breathing, rather than traumatic, like giving birth,
• Improves the quality of the software, by forcing the team to fix problems as they are found rather than being able to leave things for later.

Dealing with the friction

The friction points I’ve described seem to come up fairly often when Continuous Delivery is being introduced. My hope is that understanding the source of this friction will be helpful in discussing it when it comes up, and working through the issues. If developers who are initially uncomfortable with breaking with the “proper” way of doing things, or find a CD pipeline overly complex or difficult understand the aims and value of these practices, hopefully they will be more open to giving them a chance. Once these practices become embedded and mature in an organization, team members often find it’s difficult to go back to the old ways of doing them.

Edit: I’ve rephrased the definition of the “traditional agile” approach to making software ready for release. This definition is not meant to apply to all agile practices, but rather applies to what seems to me to be a fairly mainstream belief that agile means stopping work to make the software releasable.

Heroku is an easy way to host your apps. It is simple and runs in the cloud - so you avoid the need for servers and a lot of infrastructure automation scripting. Unfortunately, there aren’t many good Continuous Integration or Continuous Delivery options that can run on Heroku12, so if you’re a firm believer in CI/CD you will still need some servers outside of Heroku.

So, how do you add Heroku deployments into you pipeline if you are running a CI server outside heroku? This Jenkins setup is working for me:

Heroku Setup

If you don’t already have multiple environments in Heroku, you’ll need to set that up. Check out the Heroku guide on Managing Multiple Environments for an App.

tl;dr: Heroku environments are really distinct apps. They each have their own set of plugins and collaborators, so make two apps with the same plugins. I wouldn’t share the collaborators - the team can push to CI, but only CI should push to production.

Here’s an sample of a two-environment setup:

Heroku setup

1
2

heroku create --stack cedar --addons scheduler your-app
heroku create --stack cedar --addons scheduler --remote production your-app-prod

Jenkins Setup

Here’s what you need to do on the Jenkins side:

Plugins

Install Jenkins GIT plugin

Create the Post Deploy Job

Create a new job named your-app-postdeploy. It should run whatever is necessary to complete a deployment after a git push to Heroku. Probably something along the lines of:

Jenkins Execute shell Setting

1

heroku rake db:migrate db:seed --app your-app-prod

Setup both Git repos as SCMs

Setup the build

Setup whatever your CI would normally do if you weren’t using Heroku environments. If your CI tests include an integration phase that hits http://your-app.heroku.com then you should probably include the same steps as your post-deploy job (with –app your-app).

Setup the merge and push

Setup the post-deploy trigger

Summary

This should get you Continuous Deployment from Jenkins to Heroku. There are a couple caveats:

• You may get a merge conflict if someone manually pushes changes to production that were not pushed through your-app/master. You shouldn’t do that anyways.
• This is Continuous Deployment, not Continuous Delivery. You would need to make some changes to support a manual gate before production. The only opportunity this provides for a manual gate (between deployment and post-deployment) is not a viable option.
• Your application may be broken between the deploy and post-deploy. This is usually just a few seconds. You could briefly enable Heroku maintenance mode if the user experience is an issue.

Footnotes

1. Travis-CI is one. It only seems to support public GitHub repos and is still in alpha. ↩
2. Tddium is another. It is a paid service and Heroku does not currently recommend allowing it to push to production. ↩

Deploying Django applications on Heroku

18 January 2012 – Melbourne, Australia

For a long time Ruby developers enjoyed painless deployment to Heroku

The Python landscape was limited to Google App Engine for quite some time (and I do mean limited but I’ll save that for another time). Once Heroku was acquired by Salesforce it seems that the market for cloud hosting of Python applications has exploded. Now we have plenty of choices such as epio gondor appHosted DjangoZoom (some of these are still in private beta). There is also dotCloud that seems to support just about everything. It’s a good time to be a Python developer.

After their acquisition Heroku has released new features faster than ever. Their Cedar stack now officially supports Ruby, Node.js, Clojure, Java, Python and Scala. Let’s take a look at how we can deploy a fairly typical, albeit simple, Django application to Heroku.

Prerequisites: pip and virtualenv

Before we get started we’ll need to install pip and virtualenv

If you have setuptools installed you should be able to install both using:

sudo easy_install pip
pip install virtualenv

If you need more detailed instructions please take a look at installing pip.

Create git repository

You should already know how to create a new git repository.

git init myawesomeproject
cd !$

In case you are wondering, !$ expands to the last argument to the last command (myawesomeproject in this case).

Create virtual-environment

If you have been using virtualenv for a while you might be used to creatting virtual environments in a folder called ve, env or similar. For the best experience when working
with Heroku you should however create the virtual environment directly at the root of your checkout.

virtualenv --no-site-packages .
source bin/activate

You have now created and activated your virtual environment. You will need to run bin/activate every time you are working on this project. While we’re at it let’s also ignore the virtualenv artifacts. Put the following in your .gitignore file

/bin
/include
/lib
/share

While you’re at it you should also consider adding *.pyc to .gitignore.

Install dependencies and freeze

For a simple Django application you will only need Django and psycopg2 (to talk to Postgres). Install them using pip and then freeze the exact versions used to a file called requirements.txt. Heroku will use requirements.txt to automatically install your dependencies when you push.

pip install Django psycopg2
pip freeze > requirements.txt

When you add new requirements to your project you can pip install them directly and regenerate requirements.txt with pip freeze.

Create Django application

Now you can create a Django project

django-admin.py startproject myproject

and make it awesome…

Handling database migrations

You are going to quickly need to migrate your database schema. Fortunately you can use Django South to handle data and schema migrations. Install it with pip

pip install south

and re-create your requirements file.

pip freeze > requirements.txt

Add it to your INSTALLED_APPS in settings.py and start converting your applications using convert_to_south

myawesomeproject/manage.py syncdb
myawesomeproject/manage.py convert_to_south your_application

Let’s also tell South that our current database schema is up to date, by fake applying the initial migration.

myawesomeproject/manage.py migrate --fake your_application

Now every time you make a change to your Django models, you can create new migrations and apply them.

myawesomeproject/manage.py schema_migration --auto your_application
myawesomeproject/manage.py schema_migration migrate your_application

You can learn more about South, including using it for data migrations, by checking out the tutorial and documentation

Handling static files

In a more traditional hosting setup you might use Apache or Nginx to handle serving static files. When deploying to Heroku though you should consider hosting your static files in S3. Luckily Django can easily support a variety of storage backends, and the django-storages package allows you to easily use S3.

First, create a bucket in S3, using either the AWS Console or your favorite tool. Then, modify your settings.py and add the following values:

import os

AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = '<YOUR BUCKET NAME>'

STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

STATIC_URL = 'http://' + AWS_STORAGE_BUCKET_NAME + '.s3.amazonaws.com/'
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'

Notice that we are using environment variables to store the AWS access key and secret key. While we are on this topic, if you are planning to open source the Django application you are deploying, consider also storing your SECRET_KEY in an environment variable.

SECRET_KEY = os.environ.get('DJANGO_SECRET_KEY')

We are now ready to Deploy to Heroku.

Creating an environment in Heroku

Let’s start by installing the Heroku gem

gem install heroku

followed by creating a new Cedar stack in Heroku.

heroku create --stack cedar

Optionally, you might want to map your own domain name to your Heroku stack.

heroku addons:add custom_domains
heroku domains:add www.example.com
heroku domains:add example.com

You can find information on managing custom domains in Heroku here

Let’s add the necessary environment variables (do the same for SECRET_KEY if necessar)

heroku config:add AWS_ACCESS_KEY_ID=youraswsaccesskey
heroku config:add AWS_SECRET_ACCESS_KEY=yourawssecretkey

For extra security you should use the Identity & Access Management (IAM) service to create a separate user account with the following policy

 {
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::BUCKETNAME",
        "arn:aws:s3:::BUCKETNAME/*"
      ]
    }
  ]
}

This way if the credentials stored in Heroku are ever compromised, the attacker will only have access to the files stored in the bucket of this application.

Deploying application to Heroku

This is as easy as running running git push.

git push heroku master

Your application is now deployed, but you still need to configure the database

heroku run myawesomeproject/manage.py syncdb
heroku run myawesomeproject/manage.py migrate

You should now have a working application, but we have not yet deployed our static files.

heroku run myawesomeproject/manage.py collectstatic

At this point you should have a fully functional Django application deployed to Heroku with static files hosted in S3. If you are having problems you can investigate the logs with heroku logs. You can also consider turning on DEBUG temporarily, but don’t forget to turn this off. To make it easier to turn DEBUG on and off consider adding the following to your settings.py

DEBUG = bool(os.environ.get('DJANGO_DEBUG', ''))
TEMPLATE_DEBUG = DEBUG

Now you can turn debug on and off using heroku config:add DJANGO_DEBUG=true and turning it off with heroku config:remove DJANGO_DEBUG

Emacs + paredit under terminal (Terminal.app, iTerm, iTerm2)

15 January 2012 – Melbourne, Australia

I prefer to use Emacs in a full-screen terminal window. One problem that has plagued me until today though has been the lack of proper Control and Meta arrow combinations when working at the terminal. Especially when working with paredit which frequently involves the use of C-left, C-right, C-M-left, C-M-right and less frequently of M-up and M-down. To get paredit to work properly I kept switching to Cocoa Emacs (in case you didn’t know, you can install Cocoa emacs with brew install emacs --cocoa if you are using homebrew )

Today I decided to get to the bottom of this problem at any cost. First, I suspected that Control arrow combinations were not being sent properly by my terminal, in my case iTerm2. You can also fix this however for Terminal.app or the original iTerm.

iTerm2 key bindings

Select Profiles > Open Profiles… from the menu bar, or press Command-O and take a look at the default profile. Click on the Keys section. While you are here verify you have Left Option and Right Option as +Esc. For the arrow key fixes though you will need to add a series of key shortcuts. The easiest way to get started is select Load Preset… > xterm Defaults.

This will map C-up, C-down, C-right, and C-left to send the following escape sequences:

C-up    : Esc-[1;5A
C-down  : Esc-[1;5B
C-right : Esc-[1;5C
C-left  : Esc-[1;5D

It will also define Shift arrows and Control-Shift arrows, but we don’t care about those at the moment. These are not quite sufficient, but before we go any futher, let’s make sure we can get these to work in Emacs.

Check Control-arrow bindings within Emacs

Open up a new terminal window and then open emacs at the terminal with emacs -nw. Now, with paredit turned off, try C-left and C-right, which should most likely move a word at a time left and right respectively. To verify Emacs is picking up the correct keys you can also try C-h k for Describe key followed by the key combination. For example, C-h k C-left should display

<C-left> runs the command backward-word, which is an interactive
compiled Lisp function in `simple.el'.

It is bound to <C-left>, <M-left>, M-b, ESC <left>.

(backward-word &optional ARG)

Move backward until encountering the beginning of a word.
With argument ARG, do this that many times.

[back]

As long as TERM is set to xterm the above bindings should work automatically in Emacs. Should you have to define your own bindings for these escape sequences you could do so with

(define-key input-decode-map "\e[1;5A" [C-up])   
(define-key input-decode-map "\e[1;5B" [C-down]) 
(define-key input-decode-map "\e[1;5C" [C-right])
(define-key input-decode-map "\e[1;5D" [C-left]) 

Note that input-decode-map is only defined starting with Emacs 23.

Paredit

At this point you should have C-left working outside of paredit. Now turn on paredit mode (M-x paredit-mode) and try C-left and C-right again. Chances are you will see [1;5D and [1;5C. If this happens only in paredit mode, then the culprit is most likely the bidning of M-[. You can figure this out by trying describe key again. If you try C-h k C-left you will most likley see

M-[ runs the command paredit-bracket-wrap-sexp, which is an
interactive Lisp function in `paredit.el'.

It is bound to M-[.

(paredit-bracket-wrap-sexp &optional N)

Wrap a pair of bracket around a sexp

M-[

(foo |bar baz)
  ->
(foo [|bar] baz)

Huh?

Where does M-[ come from?

Each time you press Control + left arrow the terminal will send the following sequence as defined above: ESC [ 1 ; 5 D. Emacs starts interpreting this sequence, but it gets an early match on ESC [ which is the same as M-[ and invokes paredit-bracket-wrap-sexp. We need to turn off this behavior, which we can do by putting the following in ~/.emacs

(require 'paredit)
(define-key paredit-mode-map (kbd "M-[") nil)

Once you load the above code, try C-left again in paredit. If that works, you are ready for the next step.

Add Meta-arrow and Control-Meta-arrow to iTerm2

The xterm Defaults only provided us with certain key bindings. Go back to the profile key bindings under iTerm2 and add bindings for the following:

M-up      : Esc-[1;4A
M-down    : Esc-[1;4B
M-right   : Esc-[1;4C
M-left    : Esc-[1;4D

C-M-up    : Esc-[1;8A
C-M-down  : Esc-[1;8B
C-M-right : Esc-[1;8C
C-M-left  : Esc-[1;8D

To do this, click on the + sign, type the key sequence, then under Action: select Send Escape Sequence and type in the escape seequence starting with [1;. You can look as an example at the values for Control-left and friends that were added when you loaded the xterm Defaults map.

Why these values? I have no idea. I looked at the escape sequences for plain arrow keys, Shift-arrow keys and Control-arrow keys and I decided to experiment a little in the neighboring spaces, using C-h k to figure out which key sequence is bound to what I want. If you find a better explanation please let me know.

iTerm

If you are using iTerm you can add key bindings as follows:

Bookmarks > Manage Profiles > Keyboard Profiles > xterm and under Key map settings: add (by clicking on the + sign)

Key: cursor left
Modifier: Control
Action: send escape sequence

[1;5D

Add the remaining key bindings using the above format and the values from the iTerm2 table.

Terminal.app

In Terminal.app you will need to add a few key bindings by going to Preferences > Settings > Keyboard. The end result will be the same as iTerm2 but the interface is slightly different.

Key: cursor left
Modifier: Control
Action: send string to shell

\033[1;5D

where \033 represents Escape. For the other keys refer to the map for iTerm2 above.

Puppet and Chef are both good at managing SSH known_hosts. This is the primary example for Puppet’s Exported Resources, and Chef does it easily via search.

However, neither solution works well in a “masterless” setup. The Chef solution requires a full Chef Server setup - CouchDB, AMQP, and Solr. Puppet isn’t quite as bad - you just need a database to run masterless and still use Exported Resources - like Loggly does. This negates some of the masterless benefits, though, and Loggly lists lots of caveats.

If you happen to be using Capistrano for any part of your project, here is a fast, simple way to manage known_hosts without requiring a database.

1
2
3
4
5

task :setup_known_hosts do
        find_servers.each do |h|
          run "#{sudo} bash -c 'ssh-keyscan -t rsa #{h} >> /etc/ssh/ssh_known_hosts'"
        end
end

Usage is simple, just:

1

cap setup_known_hosts

or

1

cap setup_known_hosts HOSTS=<your_hosts>

This was a good fit for us. We were using Capistrano for bootstrapping, and Capistrano Multistage Extension to define environments. I just added this task as part of bootstrapping, so cap production bootstrap would allow all my production servers to talk with each other - but no one else.

I’m finally getting a blog started. I’m dual booting Windows and Ubuntu, and had to fix rubypython on Windows (so Octopress can use Pygments for syntax highlighting). I took this pull request and modified it slightly to work with newer Python installers.

This is how I fixed syntax highlighting lib/rubypython/pythonexec.rb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

if FFI::Platform.windows?
  # Do this after trying to add alternative extensions, 
  # since windows install has a python27.a and can cause 
  # troble.
  #
  # Some Windows python installers install the DLL in the python directory
  # others install in the Windows system directory.  So we should search both.
  path = File.dirname(@python)
  windir = ENV['WINDIR']
  # Windows Python doesn't like ' with inner " so we have to switch it around. 
  winversion = %x(#{@python} -c "import sys; print '%d%d' % sys.version_info[:2]").chomp
  dll = "python#{winversion}.dll"
  locations << File.join(windir, "System32", dll)
  locations << File.join(windir, "SysWOW64", dll)
  locations << File.join(path, dll)
  locations << File.join(path, "libs", dll)
end

The iteration is a cornerstone of agile development. It provides a heartbeat for the team and its stakeholders, and a structure for various routine activities that help keep development work aligned with what the customer needs. However, the way many teams run their iterations creates serious pitfalls which can keep them from delivering software as effectively as they could.

The orthodox approach to the iteration is to treat it as a timebox for delivering a batch of stories, which is the approach most Scrum teams take with sprints (the Scrum term for an iteration). In recent years many teams have scrapped this approach, either using iterations more as a checkpoint, as many ThoughtWorks teams do, or scrapping them entirely with Kanban and Lean software development.

For the purpose of this post, I will refer to these two general approaches to running iterations as the “orthodox” or “timeboxed-batch” iteration model on the one hand, and the “continuous development” model on the other hand. Although orthodox iterations have value, certainly over more old-school waterfall project management approaches, continuous development approaches which do away with timeboxing and avoid managing stories in batches allow teams to more effectively deliver higher quality software.

Orthodox iteration model: (or “timeboxed-batch” model). Each iteration works on a fixed batch of stories, all of which must be started and finished within a single iteration. Continuous development model: Stories are developed in a continuous flow, avoiding the need to stop development in order to consolidate a build containing only fully complete stories. The anatomy of the orthodox iteration

In the classically run iteration or sprint, the Product Owner (PO) and team choose a set of stories that it commits to deliver at the end of the iteration. All of the stories in this batch should be sufficiently prepared before the iteration begins. The level and type of preparation varies between teams, but usually includes a level of analysis including the definition of acceptance criteria. This analysis should have been reviewed by the PO and developers to ensure there is a common understanding of the story. The PO should understand what they can expect to have when the story is implemented, and the technical team should have enough of an understanding of the story to estimate it and identify potential risks.

The iteration begins with an iteration kickoff meeting (IKO) where the team reviews the stories and confirms their confidence that they can deliver the stories within the iteration. The developers then choose stories to work on, discussing each story with the PO, Business Analyst (BA), and/or Quality Analyst (QA) as appropriate, then breaking it down into implementation tasks. Implementation takes place with continual reviews, or even pairing with these other non-developers, helping to keep implementation on track, and minimizing the amount of rework needed when the story is provisionally finished and goes into testing.

The QA and BA/PO then test and review each story as its implementation is completed. This is in addition to the automated testing which has been written and repeatedly running following TDD and CI practices. Only once the story is signed off do the developers move on to another one of the stories in the iteration’s committed batch.

As the end of the iteration approaches, developers and QAs should be wrapping up the last stories and preparing a releasable build for the showcase, which is typically held on the final day of the iteration. In the showcase, the team demonstrates the functionality of the completed stories to the PO and other stakeholders, the stories are signed off. The team holds a retrospective to consider how they can work better, then on the next working day they hold the IKO to start the following iteration.

When the iteration ends the team has a complete, fully tested and releasable build of the application, regardless of whether the software actually will go into production at this point.

The start and end dates of the iteration are firmly fixed. If there are stories (or defect fixes) which aren’t quite ready at the end of the iteration, the iteration end date is never slipped. Instead, the story is not counted as completed, so must be carried over to the next iteration.

The benefits of the orthodox iteration

This style of iteration offers many benefits over traditional waterfall methodologies. A short, rigid cycle for producing completely tested and releasable code forces discipline on the team, keeping the code in a near-releasable state throughout the project, and avoiding the temptation to leave work (e.g. testing) for “later”, building up unmanageable burdens of work, stress, and defects to be dealt with under the pressure of the final release phase.

The timeboxed iteration also forces the team to learn how to define stories of a manageable size. If stories are routinely too big to complete in one iteration this is a clear sign that the team needs improve the way it defines and prepares stories.

This demonstrates another benefit of the iteration, which is frequent feedback. The team is able to evaluate not only the quality of their code and its relevance to the business by getting feedback quickly, they are also able to evaluate how effectively they are working, and try out ideas for improving continually throughout the project.

Fundamental problems with the orthodox itertation

The timeboxed-batch approach to iterations has value, particularly for teams inexperienced with agile. However, it has fundamental problems. At core, this approach is waterfall written small, with many of the same flaws, albeit with a small enough cycle that issues can be dealt with more quickly than with a full waterfall project.

To understand why this is so, let’s flesh out the idealized anatomy of the iteration from above with some of the things which often happen in practice.

• At the start of the iteration, no development is taking place, because everyone is working on preparing the new batch of stories. The BA’s are extremely busy now, because they have a full working set of stories to hand over to developers (i.e. however many stories the team can work on at once, that’s how many stories the BA’s must hand over all at the same time). The QA’s are less busy now, although they may be helping the BA’s out, and planning their testing for the iteration’s stories.
• Actually, I lied. Development is taking place, and QA’s are extremely busy. Testing and bugfixing stories left over from the previous iteration is still going on. See points below to understand why. As a result, preparation and starting of the stories for this iteration is sluggish because of work carried over from the previous iteration. This actually isn’t necessarily bad, since it helps to stagger the story preparation work, preventing the BA’s from becoming a bottleneck. However, depending on whether carryover work was factored into the number of stories chosen for the current iteration (business pressures often make this difficult), this may mean the team is already at risk of failing to meet its commitment.
• Once the previous iteration’s work has settled down, QA’s have little to do until the end of the iteration approaches, at which point they come under enormous pressure. Developers are humping to get their stories done in time, leaving QA’s with a pile of stories to be tested in time for the showcase. Any defects they find increase this pressure even more, with very little time to get the fixes in and then re-tested (maybe needing even more fixing and re-testing!)
• If developers complete a story with a bit of time left in the iteration, they aren’t able to start new stories because the stories for the following iteration won’t be ready to work on until the IKO.
• In the end, some stories don’t get fully tested during the iteration. They may be tested in the following iteration, after having already been signed off as “complete” by the unsuspecting Product Owner. If so, developers need to be pulled away to fix the defects found, or else the defects are added to a backlog to be fixed “later” (also known as “probably never”). Other developed code is left completely untested or under-tested, with the vague hope that any defects will be found in later testing phases, or that maybe there aren’t any important bugs anyway. In fact, these defects will be found, but not at a time more convenient to the team.
• If any serious issues are raised by stakeholders during the showcase there is not time to fix them until the next iteration, which means it will take the full length of an iteration before a truly releasable build is created.

The root problem of the orthodox iteration

At the end of the day, the orthodox iteration suffers from two problems which are inherent in its very definition: it organizes work into batches, and it enforces a timebox.

Batching work is the antithesis to flow. The Lean approach to working aims to maximize the flow of work for the members of a team, which in software development translates to getting stories flowing easily through creation, analysis, implementation, validation, and release. When a developer finishes one story and it is signed off, she should have another story ready for her to pick up and start on. This shouldn’t need to wait on an arbitrary ceremony, and certainly shouldn’t have to wait for everyone else on the team to finish their stories and get them all signed off.

The batching focus of orthodox iterations doesn’t only cause developers to block, it also turns BA’s, QA’s, and the PO into bottlenecks. As described above, the start and end of the iteration each put a full working set of stories in the same state at once, all needing the same activity carried out on them at once.

Imagine an assembly line which starts up to assemble twenty cars, then stops while they are all inspected at once. Only once all of the cars are inspected and their defects fixed does the line start up again to begin assembling another twenty cars.

Timeboxing is also a source of problems for iterations. The main problem is the arbitrary deadline creates pressure to get stories “over the line” so they can count towards the velocity for the iteration. Unless management is enlightened (or uninterested) enough to avoid focusing on fluctuations of velocity from iteration to iteration (and even the most enlightened managers I’ve worked with do get worked up over velocity) this leads to the temptation to rush and cut corners, or to play games with stories.

Rushing obviously endangers the quality of the code, which almost certainly leads to delays down the line when the defects surface. Playing games, such as closing unfinished stories and opening defects to complete the work, or counting some points towards an unfinished story, undermines the team’s ability to measure and predict its work honestly. These bad habits will catch up one way or another.

Expecting code to be complete at the end of the iteration, fully tested, fixed, and ready for deployment, is unrealistic unless the iteration is structured with significant padding at the end. This padding must come after all reviews, including the stakeholder showcase, to allow time to make corrections, unless those reviews are mere rubber stamp sessions, with no genuine feedback permitted. This then means the team will be underutilized during the padding time. Otherwise, if there is so much rework done during this period that the entire team is fully engaged, then the risk of introducing new defects is too high to be confident in stable code by the end.

The alternative is to break the strict timeboxed-batched iteration model by interleaving work on the next iteration with the cleanup work from the previous iteration. This turns out to not be such a bad idea, and leads to evolving away from the timeboxed-batch iteration model towards the continuous development model.

The continuous development model

The continuous development model may be purely iteration-less, e.g. Kanban, or it may still retain the iteration as a period for measuring progress and for scheduling activities such as showcases. Once development is underway stories are prepared, developed, and tested using a “pull” approach, being worked on as team members become available, so that stories are constantly flowing, and everyone is constantly working on the highest value work available at the moment. This requires some different approaches to managing work flow than are used with other approaches. For more information, look into Kanban and Lean software development.

Since joining a year or so ago I’ve found that although no two ThoughtWorks projects run in exactly the same way, there is a strong tendency to use iterations which look a lot like Kanban, but retaining a one or two week iteration. Iterations are used to report progress (including velocity), and to schedule showcases and other regular meetings, but stories are not moved through the process in batches. Teams don’t start and stop work as a whole other than the start and end of a release. If the showcase is two days away, nothing stops a developer pair from starting on a new story knowing full well it will be incomplete when the codebase is demoed to the stakeholders, and possibly even deployed to later stage environments.

Although we do make projections and aim to have certain stories done by the next showcase, the team doesn’t promise to deliver a specific batch of stories. If it makes sense, stories can be dropped, added, or swapped as needed. This gives the business more flexibility to adapt their vision of the software as it is developed. It also reduces the pressure to mark a given story as “done” by a hard deadline, since there is no disruption from letting work carry on over the end of an iteration.

I’ve seen a Scrum team become ornery and rebellious when a PO made a habit of asking to swap stories after a sprint had started, even though work hadn’t been started on the particular stories involved. This was made worse because bugfixes were scheduled into sprints alongside stories, meaning that any serious defect found in production completely disrupted the team. Another factor that aggravated the situation was that the stories for each sprint were agreed before the end of the previous iteration. So if the showcase raised ideas for improvements to the functionality completed in iteration N, new stories could only be started in iteration N + 2 at the soonest. This hardly created a situation where the PO or the business felt the development team was responsive to business needs.

Also see Oren Teich’s post Go slow to go fast, which points out the problems with deadlines, and that iterations are simply a shorter deadline.

Challenges and rewards of continuous development

There are certainly challenges in moving to continuous development over the timeboxed-batch model. There is more risk of stories dragging on across multiple iterations. This can be mitigated by monitoring cycle time and keeping things visible, so that the team can discuss the issue and make changes to their processes if it becomes a problem.

For teams which are new to agile and still struggle to create appropriately sized stories, the timeboxed model may be more helpful to build the discipline and experience needed before being able to move to a continuous model. However, for experienced teams, timeboxing and batching stories simply has too many negative effects.

Continuous development, with a looser approach to iterations, maximizes the productivity of the team, avoids pitfalls that put quality at risk, and offers the business and the team more flexibility.

In my previous article on the server lifecycle I mentioned ConfigurationDrift, a term that I’ve either coined, or I’ve forgotten where I originally heard, although most likely I got it from the Puppet Labs folks.

Configuration Drift is the phenomenon where running servers in an infrastructure become more and more different as time goes on, due to manual ad-hoc changes and updates, and general entropy.

A nice automated server provisioning process as I’ve advocated helps ensure machines are consistent when they are created, but during a given machine’s lifetime it will drift from the baseline, and from the other machines.

There are two main methods to combat configuration drift. One is to use automated configuration tools such as Puppet or Chef, and run them frequently and repeatedly to keep machines in line. The other is to rebuild machine instances frequently, so that they don’t have much time to drift from the baseline.

The challenge with automated configuration tools is that they only manage a subset of a machine’s state. Writing and maintaining manifests/recipes/scripts is time consuming, so most teams tend to focus their efforts on automating the most important areas of the system, leaving fairly large gaps.

There are diminishing returns for trying to close these gaps, where you end up spending inordinate amounts of effort to nail down parts of the system that don’t change very often, and don’t matter very much day to day.

On the other hand, if you rebuild machines frequently enough, you don’t need to worry about running configuration updates after provisioning happens. However, this may increase the burden of fairly trivial changes, such as tweaking a web server configuration.

In practice, most infrastructures are probably best off using a combination of these methods. Use automated configuration, continuously updated, for the areas of machine configuration where it gives the most benefit, and also ensure that machines are rebuilt frequently.

The frequency of rebuilds will vary depending on the nature of the services provided and the infrastructure implementation, and may even vary for different types of machines. For example, machines that provide network services such as DNS may be rebuilt weekly, while those which handle batch processing tasks may be rebuilt on demand.

One of the cornerstones of a well-automated infrastructure is a system for provisioning individual servers. A system that lets us reliably, quickly, and repeatably create new server instances that are consistent across our infrastructure means we spend less time fiddling with individual servers. Instead, servers become disposable components that are easily swapped, replaced, and expanded as we focus our attention on the bigger picture of the services we’re providing.

The first step in achieving this is making sure server instances are built using an automated process. This ensures every server is built the same way, that improvements can be easily folded into the server build process, and that it is a simple matter to spin up new instances and to scrap and replace broken ones. Automating this process also means your team of highly skilled, well-paid professionals don’t need to spend large amounts of their time on the brainless rote-work of menu-clicking through OS installation work.

I first used automated installation by PXE-booting physical rack servers in 2002, following the advice I found on the then-current infrastructures.org site, and in later years applied the same concepts with virtualized servers and then IaaS cloud instances.

The machine lifecycle

I think of this as the machine lifecycle (which I tend to call the ‘server lifecycle’ because that’s what I normally work with, although it’s just as applicable to desktops). This involves a number of activities required to set up and manage a single machine instance, such as partitioning storage, installing software, and applying configuration.

These activities are applied during one or more phases of the machine lifecyle. There are three phases: “Package Image”, “Provision Instance”, and “Update Instance”. There are a number of different strategies for deciding which activities to do in each phase.

The various activities may be applied during one or more phases, depending on the strategy used to manage the machine’s lifecycle. Some strategies carry out more activities during the packaging phase, for instance, while other approaches might have a simpler packaging phase but do more in the provisioning and/or updating phase.

Machine lifecycle phases

Image packaging phase

In the image packaging phase, some or all elements of a machine instance are pre-packaged into a machine image in a way that can be reused to create multiple running instances.

This could be as simple as using a bare OS installation CD or ISO from the vendor. Alternately, it could be a snapshot of a fully installed, fully configured runnable system, such as a Symantec Ghost image, VMWare template, or EC2 AMI. Either way, these images are maintained in a Machine Image Library for use in the instance provisioning phase.

Different machine lifecycle strategies use different approaches to image packaging. ManualInstanceBuilding and ScriptedInstanceBuilding both tend to use stock OS images, which involves less up-front work and maintenance of the Machine Image Library, since the instances are take straight from the vendor. However, work is still needed to create, test, and maintain the checklists or scripts used to configure instances when provisioning.

On the other hand, CloningExistingMachineInstances and TemplatedMachineInstances both create pre-configured server images, which need only minor changes (e.g. hostnames and IP addresses) to provision new instances. This is appealing because less work is done to provision a new instance, but the drawback is that creating and updating images takes more work. Admins tend to make updates and fixes to running instances which may not make it into the templates, which contributes to ConfigurationDrift, especially if changes are made ad-hoc.

CloningExistingMachineInstances, which usually takes the shape of copying an existing server to create new ones as needed, tends to make ConfigurationDrift worse, as new servers inherit the runtime cruft and debris (log files, temporary files, etc.) of their parents, and it is difficult to bring various servers into line with a single, consistent configuration. TemplatedMachineInstances are a better way to keep an infrastructure consistent and easily managed.

The tradeoffs between scripted installs vs. packaged images depends partly on the tools used for scripting and / or packaging, which in turn often depends on the hosting platform. Amazon AWS requires the use of templates (AMIs), for example. In either case, exploiting automation more fully in the provisioning phase favours the case for keeping the packaging phase as lightweight as possible.

Instance Provisioning Phase

In the provisioning phase, a machine instance is created from an image and prepared for operational use.

Examples of activities in this phase include instantiating a VM or cloud instance, preparing storage (partitioning disks, etc.), installing the OS, installing relevant software packages and system updates, and configuring the system and applications for use.

There are two main strategies for deciding which activities belong in the packaging versus the provisioning phases. One is RoleSpecificTemplates, and the other is GenericTemplate.

With RoleSpecificTemplates, the machine image library includes images that have been pre-packaged for specific roles, such as web server, application server, mail server, etc. These have the necessary software and configuration created in the packaging phase, so that provisioning is a simple matter of booting a new instance and tweaking a few configuration options. There are two drawbacks of this approach. Firstly, you will have more images to maintain, which creates more work. When the OS used for multiple roles is updated, for example, the images for all of those roles must be repackaged. Secondly, this pattern gives you less flexibility, since you can’t easily provision an instance that combines multiple roles, unless you create - and then maintain - images for every combination of roles that you might need.

With the GenericTemplate pattern, each image is kept generic, including only the software and configuration that is common to all roles. The role for each machine instance is assigned during the provisioning phase, and software and configuration are applied accordingly then. The goal is to minimise the number of images in the machine image library, to reduce the work needed to maintain them. Typically, a separate template is needed for each hardware and OS combination that can’t be supported from a single OS install. The JeOS (Just Enough Operation System) concept takes this to the extreme, making the base template as small as possible.

The GenericTemplate pattern does require a more robust automated configuration during provisioning, and may mean provisioning an instance takes longer than using more fully-built images, since more packages will need to be installed during install.

Instance Updating Phase

Once a machine instance is running and in use, it is continuously updated. This includes activities such as applying system and software updates, new configuration settings, user accounts, etc.

Many teams carry out these updates manually, however it requires a high level of discipline and organization to maintain systems this way, especially as the number of systems grows. The number of machines that a team can be managed is closely dependent on the size of the team, so the ration of servers to sysadmins is low. In practice, teams using manual updates tend to be reactive, updating machines opportunistically when carrying out other tasks, or in order to fix problems that crop up. This leads to ConfigurationDrift, with machines becoming increasingly inconsistent with one another, creating various problems including unreliable operation (software that works on one machine but not another), and extra work to troubleshoot and maintain.

Automated management of infrastructure is vital for delivering highly effective IT services. But although there are plenty of tools available to help implement automation, it’s still common to see operations teams manually installing and managing their servers, which leads to a high-maintenance infrastructure, which soaks up the team’s time on firefighting and other reactive tasks.

I’ve met many smart and skilled systems administrators in this situation. These folks know automation can make their life easier, but they can’t afford to take time away from turning cranks, greasing wheels, and unjamming the gears to keep their infrastructure puffing along in order to focus on improving their situation.

I’m convinced this is largely due to habit. Even though these teams understand that automation would be useful to them, when the pressure is on (and the pressure is always on), they roll up their sleeves, ssh into the servers and knock them into shape, because that’s the fastest way to get stuff done. Manual infrastructure management is what they’re used to. I find that most of these teams haven’t had personal experience of well-automated infrastructures, and don’t tend to believe it’s something they can realistically implement for their own operations.

Sysadmins who have worked in teams with mature, comprehensive automation, on the other hand, can’t go back. Sure, they might log into a box to diagnose and fix something that needs fixing right now, but they can’t relax until they’ve baked the fix into their automated configuration, and made sure that their monitoring will alert them ahead of time if the problem happens again.

Breaking out of manual infrastructure management and setting up an effective automation regime is difficult. Although there are loads of tools out there to make it work, it helps to understand good strategies for implementing them. I recommend looking over the material on the infrastructures.org site. It hasn’t been updated in a few years, so doesn’t take much of the advances since then into account, including virtualization, cloud, and newer tools like Chef and Puppet, but there is still rich material there.

Another must-read which more up to date is Web Operations by John Allspaw, Jesse Robbins, and a bunch of other smart peeps.

I’m also planning to share a few of the practices I’ve seen and used for automation in upcoming posts.

有位童鞋问了这样一个问题："我现在的团队也面临这样的问题，更糟糕的是我们还没有测试套件。目前只是面临企业内网，马上也面临外网的应用了。痛苦。。。那个大侠给个好建议！"其实，文章中已经给出了答案，即一直保持“主干开发为主的短周期分支策略”。当然，很多团队无法达到这种境界。原因可能非常简单。比如，无法拒绝客户要求，否则就有可能失去重要客户。开发时的代码质量太差，当需要发布时，在分支上修复Bug的时间需要很长，而在主干上开发新功能时仍旧不注重质量。传统的瀑布开发方式使两次发布之间的时间间隔太长（每次发布都需要三个月以上）。