Danilo writes about, “Done is not done” and I’m reminded of being careful about using the right metric. I remember being on a project where people had individual velocity. This is a really bad thing unless you want to prevent collaboration. I’d add it to Danilo’s list.

Dan North wrote an interesting post about the perils of estimation, questioning our approach to inceptions, release planning, and setting expectations about scope. This made me think about the implications of those factors once a project starts, and I came up with some anti-patterns on the usage of velocity to track progress. This is my first attempt at writing about them.

Before we start, it’s important to understand what velocity means. My simple definition of velocity is the total number of estimation units for the items delivered in an iteration. Estimation unit can be whatever the team chooses: ideal days, hours, pomodoros, or story points. The nature of items may vary as well: features, use cases, and user stories are common choices. Iteration is a fixed amount of time where the team will work on delivering those items. Sounds simple? Well… there’s one concept that is commonly overlooked and that’s the source of the first anti-pattern: what does delivered means?

One of the most common anti-patterns I’ve seen is not having a clear definition of done. Lean thinking tells us that nothing is really done until it’s delivering value, which in software means: code running in production and being used by real users. Although I know very few teams who can deploy code to production at the end of every iteration (some even do more than once per iteration), once a story is considered done, it could be potentially shipped, if the business decides so. There shouldn’t be a lot of extra work after that.

Another bad implication of this anti-pattern is that some teams decide to change the definition of done and count half-completed work to show progress. Some of the symptoms to help diagnose if your team is suffering from this anti-pattern are:

• The team starts tracking dev-complete stories
• “It’s done, but [we need to write the acceptance test/it's not integrated with the other system/...]“
• “It’s done, but not done-done”
  
  
• It takes a lot of extra work to get the story deployed to production
• After finished, the story goes into the next team’s backlog
• Hearing terms like “development team velocity” or “test team velocity”
• Counting half-points or percentages because “if we don’t count it will look like we haven’t worked”

The solution? Remember that velocity is just a number that provides information for the team to understand and improve it’s process. Forget that you’re tracking it and focus on the entire Value Stream and on what’s really value-added to get things into production. Anything else is just waste. If it’s not done, it’s not done. Accept it, move on, and don’t overcomplicate, because it will only add noise and mask what could have been important information to the team.

[This is fiction. Any resemblance to individuals living or otherwise is purely coincidental.  Really.]

Joan’s phone starting ringing insistently. Joan thought for a moment, since she was watching her favorite reality TV show, and that was her time to just disconnect. In spite of her preferences, Joan decided to answer the phone.

“Joan?” She heard her friend Nancy’s voice, and her heart skipped a beat. Nancy was sobbing. “Nancy? What’s wrong honey?”

“They fired me, Joan! They fired me!”

“Joanie…” sobbing “…they said that I just wasn’t living up to their expectations.”

“Oh, Nancy…”

Now let’s talk about Joan’s possible reactions…

Each of us has a different reaction, and each of us offers a different response based on that reaction*. For the moment, I want to talk about three types of reaction and response: sympathy, empathy, and identification.

Reaction: Sympathy

Response: “Oh, Nancy… that’s terrible. You must feel miserable.  I can only imagine how that feels. Would you like to come over and talk?”

Reaction: Empathy

Response: “Oh, Nancy… I feel terrible. I can’t believe it! I’ll come over and let’s talk about what we can do.” Joan cries.

Reaction: Identification

Response: “Oh, Nancy. Those bastards! After all you’ve done for them, and how hard you’ve worked. You gave your all to that company, and this is how they treat you? I’m devastated.”

Nancy cries.

Joan cries.

Up to this point, sympathy, empathy, and identification sound a lot alike. In all three versions, Joan has an emotional reaction that leads to a behavior – her response. In each case, her response is subtly different. Note that in the following discussion, I am not making a judgment about better versus worse, or good versus bad… I’m working on achieving understanding and recognizing that each type of reaction and response deserves and requires a different response from me.

In being sympathetic, Joan’s response is separate. Joan is clear that what is happening to Nancy is about Nancy, not about Joan. While Joan may feel sad or angry, it is on behalf of her friend. From Nancy’s perspective, there is a little bit of distance between them. Joan’s feelings are moderate.

In being empathetic, Joan’s response is collective. Joan feels what she believes Joan feels, including the pain, indignation, and so forth. For Joan, what is happening is also happening to her, emotionally. From Nancy’s perspective, it’s like a resonation, which may increase the level of her feelings. To a certain extent, Joan’s reaction becomes an extension of Nancy’s reaction. Joan’s feelings are intense, although she recognizes that they are about Nancy.

In identifying with Nancy, Joan takes on Nancy’s feelings and reactions. Joan’s response is intense and personal, as though she were the one who had been fired. Nancy may be taken aback by the intensity of Joan’s reaction, as Joan takes on some of Nancy’s emotional response. Joan behaves as if she were the one who had been fired, and will react to others as if she were the victim as much as Nancy.

To see how this works, let’s add Joan’s husband Mark to the story…

“Joan? What’s going on?”

Sympathy

“It’s Nancy. She got fired today. I feel so bad for her. She’s so upset.”

“That sucks. What’s she going to do?”

“I don’t know yet. I may have to spend some time with her.  I hope that’s okay with you.”

Empathy

“It’s Nancy. She got fired today. I’ve got to go over there to be with her right now!” Sobbing

“That sucks. What’s she going to do?”

“I don’t know yet, but I just know how horrible she feels and that I have to go be with her. It’s so painful! Doesn’t this upset you?”

Identification

“It’s Nancy. She got fired today. I’ve got to go over there to be with her right now!” Sobbing

“That sucks. What’s she going to do?”

“They treated her like dirt! How can you be so calm?  Don’t you care? They were unfair and cruel. I don’t know what we’re going to do, but we’re going to do something to show them!”

Note that Joan’s response to Mark escalates from Sympathy to Empathy to Identification. In the latter, Joan feels that what has happened to Nancy has happened to her, and thus she expects the same kind of reaction from Mark that she’d expect if she had been fired.

This post is long enough.  Now I’m going to go off and think about the differences in responses to each of the three.

Sympathy

• an inclination to support or be loyal to or to agree with an opinion; “his sympathies were always with the underdog”; “I knew I could count on his …
• sharing the feelings of others (especially feelings of sorrow or anguish)
• a relation of affinity or harmony between people; whatever affects one correspondingly affects the other; “the two of them were in close sympathy”
  
  wordnet.princeton.edu/perl/webwn
• http://dictionary.reference.com/browse/sympathy

Empathy

• understanding and entering into another’s feelings
  
  wordnet.princeton.edu/perl/webwn
• http://dictionary.reference.com/browse/empathy

Identification

• the attribution to yourself (consciously or unconsciously) of the characteristics of another person (or group of persons)
  
  wordnet.princeton.edu/perl/webwn
• a process by which one ascribes to oneself the qualities or characteristics of another person.
• A person’s association with or assumption of the qualities, characteristics, or views of another person or group.
  
  http://dictionary.reference.com/browse/identification

*Reaction vs. Response

For the purposes of this discussion, I’m defining “reaction” as the emotional or physical effect that occurs without thinking, and “response” as the chosen action or thought that occurs after the reaction. That is, if I put my hand in a fire, pulling my hand out is a reaction – I don’t think about it – while swearing about it is a response.

Tweet This Post 

Related posts

• You don’t know me… (0)
• You can’t change me (4)
• Who is You? (4)
• What’s in it for me? (0)
• What is he thinking? (3)

I downloaded this Pomodoro Timer from the Apple website.

It has support for AppleScript events, so I created a script that automatically sets my Adium status to away (with an auto-reply) while I am working on a Pomodoro and automatically sets it to Available when I have finished.

After much messing around, trying to work out just how the AppleScript pseudo-natural-language works, I ended up with this to set the away status:

-- This goes on one line in Pomodoro Setup -> AppleScript -> Start
tell application "Adium"
  to set the status of every account whose status type is available
  to the first status whose title is "Pomodoro In Progress"

and this to set it back to available:

-- This goes on one line in Pomodoro Setup -> AppleScript -> Reset and End
tell application "Adium"
  to set the status of every account whose status type is away
  to the first status whose title is "Available"

I've been doing a bit more reading of the Fake source code and one interesting thing which I came across which I hadn't seen was an active pattern which was making use of the ':?' operator to match the input type against .NET types.

  let (|File|Directory|) (fileSysInfo : FileSystemInfo) =
    match fileSysInfo with
      | :? FileInfo as file -> File (file.Name)
      | :? DirectoryInfo as dir -> Directory (dir.Name, seq { for x in dir.GetFileSystemInfos() -> x })
      | _ -> failwith "No file or directory given."

I thought maybe this was just a wild card operator to say that we don't care what the value is as long as it matches 'FileInfo' or 'DirectoryInfo' respectively but I couldn't see it defined on the list of operators on the Microsoft Research website.

A bit of googling led me to Matthew Podwysocki's post about pattern matching which explained the purpose of the operator (about 1/3 of the way down):

What the above example does is check for the corresponding .NET types by using the ':?' operator especially reserved for this behavior.

I've been playing around with a simple 'add' function to try and understand F#'s type inference and one thing I noticed is that if you just define it with minimal code you end up with a function which takes in 2 integers and returns an integer as the result:

let add a b = a + b
 
val add: int -> int -> int

I had thought that the signature and result of that function might remain generic due to the fact that there are more types than just 'int' with which you can make use of the addition operator.

For example, it is possible to add two string together but in fact you need to be more explicit about that:

let add (a:string) (b:string) = a + b
 
val add: string -> string -> string

From what I can tell if we wanted to write a generic add function we would need to do something like this - I originally tried just returning 'new A + new B' from each of the pattern matches but the return type of add3 then becomes 'string' since the first path in the pattern matching returns a 'string'.

    let add3 a b =
        match (box a,box b) with
            | (:? string as newA),(:? string as newB) -> newA +  newB |> box
            | (:? int as newA),(:? int as newB) -> newA + newB |> box
            | (:? decimal as newA),(:? decimal as newB) -> newA + newB |> box
            | _ -> failwith "you can't add these together"

Which is slightly verbose and has a type of "'a -> 'b -" obj' - I haven't been able to work out whether it's possible to create a generic function like this without needing to cast the result down to 'obj'.

I thought it might be possible to get rid of the boxing by making use of the downcast operator:

You can also use the downcast operator to perform a dynamic type conversion. The following expression specifies a conversion down the hierarchy to a type that is inferred from program context.

I tried surrounding the 'newA + new B |> box' code with a call to 'downcast' but that just resulted in the following error message when trying to make use of the function:

Value restriction. The value 'it' has been inferred to have generic type
	val it : '_a
Either define 'it' as a simple data term, make it a function with explicit arguments or, if you do not intend for it to be generic, add a type annotation.

I'd be intrigued to see if anyone has worked out how to do this as I'm out of ideas.

Mark Needham’s post on micro types sparked an idea in my head that recently came to fruition.

I was pondering on the Narrative Fixture code, and the fact that, although most of the internals are sensibly typed objects, at the FitNesse layer, we do a lot of passing of strings.

The idea (at least, my interpretation) with Tiny Types is that the string obviously means something, and that we can probably classify what that meaning is (we have a sensibly named variable for it, after all) , and so why not convert it to that meaningful thing as soon as possible.

This has the nice effect of making some vague things (such as typing on collections) much more explicit.

As an example, in the Casting Director we have:

Map<String, ActorPlayingThe<?>> dressingRoom = new HashMap<String, ActorPlayingThe<?>>();

and, after a little bit of refactoring, we end up with:

Map<CharacterName, ActorPlayingThe<?>> dressingRoom = new HashMap<CharacterName, ActorPlayingThe<?>>();

Personally, I’m a fan of low ceremony, high essence languages, but while working in a high ceremony environment, we can leverage that ceremony to provide us with a nice summary of our thinking, ready for the next time we return to the code.

Our latest technical book club session was a discussion of the logging section in Michael Nygard's Release It.

I recently listened to an interview with Michael Nygard on Software Engineering Radio so I was interested in reading more of his stuff and Cam suggested that the logging chapter would be an interesting one to look at as it's often something which we don't spend a lot of time thinking about on software development teams.

These are some of my thoughts and our discussion of the chapter:

• An idea which Nick introduced on a project I worked on last year was the idea of having a 'SupportTeam' class that could be used to do any logging of information that would be useful to the operations/support team that looked after our application once it was in production.

  This is an approach also suggested by Steve Freeman/Nat Pryce in Growing Object Oriented software (in the 'Logging is a feature' section) and the idea is that we will then focus more on logging the type of information that is actually useful to them rather than just logging what we think is needed.

  One thing which Dave pointed out is that it's often difficult to get access to the operations team to try and get their requirements for the type of logging and monitoring they need and so often ends up being something that's done very late on. On projects I've worked on there has often been a story card for logging and I think this is a good way to go as they are a stakeholder of the system so logging shouldn't just be dealt with as a nice extra.

• Something which I hadn't considered until reading this book is the idea of making logs human readable and machine parseable as well. The default format of most of the logging tools is not actually that useful when you're trying to scan through hundreds of lines of data and it was intriguing how a little indentation could improve this so dramatically with the added benefit of making it much easier to create a regular expression to find what you want.
• One thing I'm interested in understanding is how we work out what's too much logging and what's too little since it seems that it seems that the answer to this question is fairly context sensitive. For example on a recent project we logged all unhandled exceptions that came from the system as well as any exceptions that happened when retrieving data from the service layer. In general the data we've had available has been enough to solve problems but we could probably have done more, just working out what would be useful doesn't seem obvious.
• I think it was Alex who pointed out that it's often useful to have an explicit step in the build to remove any debug logging from the code so that it doesn't end up in production by mistake. This seems like a pretty neat idea although I haven't seen it done yet - it also leads towards the idea that logging is for the operations team which I think is correct although it is often suggested that logging is actually for developers since it is assumed that they would be the ones to eventually solve any problems that arise.
• The idea of having message codes for specific errors messages seems like a really cool idea for allowing easy searching of log files - we've done this on some projects I've worked on and not on others. I guess the key here is to ensure we don't end up with too many different error codes otherwise it's just as confusing as not having them at all.

Business people want estimates. They want to know how much it’s going to cost them to get a solution, and they want to know how likely it is to come in on time and on budget. And of course quality is not negotiable.

Agile teams I encounter are at best nervous about estimates and at worst simply evasive. “You don’t need estimates if you’re doing Agile,” they say. “It will be ready when it’s done. We’re constantly adding value so we don’t need to commit to a date.”

We’re missing the point of release planning

My favourite exchange goes something like:

“We’ve done an inception and broken down the entire project into stories and measured it, and it’s come in at 400 stories, estimated at 865 story points.”

“865 what?”

“865 story points.”

“So how big is a story point?”

“We don’t know yet, we’ll let you know in a few weeks.”

At a governance and funding level the business could care less about story points. They don’t actually care about stories except that we shove them in their faces with our release plans. They care about solving a problem. They came to us and asked us a) how much will it cost to solve the problem, and b) how confident are we about that number?

So how do we approach that? We go through some sort of inception process that looks something like this:

1. Identify some personas

2. Identify some process flows

3. Start breaking the flows down into stories

4. Lots and lots of stories

5. Lots and lots and lots and lots of stories

6. Spike some technical ideas that came out of the stories

7. Estimate the stories

8. Roll up all the estimates and call that our project estimate

The part where we estimate the stories is a real chore (c’mon – we’re estimating 400 stories here), so we cut corners. We do a first pass as t-shirt sizes (small, medium, large) and then take a representative sample (sounds suitably scientific) and do a “detailed” estimate of those. This involves a bunch of people estimating lots of important-sounding metrics: minimum, likely and maximum size, clarity, volatility (eh?) and whatever else, and then multiply it up to provide a WOOOOAAAAAHHHH! hang on a minute! What were we trying to do again?

All they wanted to know is: How long will it take, and how confident are you about that number?

Redefining success – in a bad way

By introducing a comprehensive story list – with or without the notion of story points – we have unwittingly reframed the project. The business started out by defining success as solving the problem, but now we have redefined success as delivering this list of stories. However we frame it, that’s what the business will believe. The project will start and and the business stakeholders will start counting down the list of stories until you get to zero.

So now we have the worst of both the Agile and plan-driven worlds: the business expects delivery of a fine-grained list of requirements (whether we call it a Product Backlog or a Master Story List), and we have only taken a half-hearted attempt at it compared to the big up-front analysis we used to do. From here on we are on the back foot, constantly negotiating with the business to manage scope, when it’s our own fault they even care about the story-level detail. They see the story backlog and mentally turn it 90 degrees and think of it as a Gantt chart. Happy days!

Back to project basics

There are two observations to make here. Firstly the business wants accuracy and we’re giving them precision. If you tell me it will take 4.632 months and it takes 8 months, that’s worse than useless. If you tell me it takes “about six months” and it takes seven months, I should still be onto a winner. (If the return on investment is small enough that the extra month stops the proposition being viable, I’d have been better off investing in something else in the first place. Spending $60,000 to realise a return of $70,000 is risky to say the least.) I’m simplifying here of course because the real RoI varies over time, and its value may be particularly time-sensitive.

Secondly we know that requirements vary on an agile project over time, for good reason. Hopefully we are learning as we go, which means we will discover new requirements and decide others are no longer worth pursuing. If we assume about a third of the requirements will be delivered as described (this is generous in light of the Standish Chaos reports), another third will be delivered but with changes, and the last third won’t be delivered at all – but replaced by other features – then we have just wasted all that time and effort in the inception coming up with detailed, high-_precision_ data for a pile of stories we will never deliver.

To compound this, it turns out that estimation is fractal. The more fine-grained you break down the requirements, the more “edges” you will discover. This means that the more detailed you estimate, the more the total will tend towards infinity, simply due to rounding errors and the fear factors that we multiply into fine grained estimates.

Use the inception for deliberate discovery

So what should we be doing during an inception instead of doing the fine-grained story breakdown? Taking it back to first principles we simply want a rough idea of size and an understanding of certainty. There is uncertainty in everything, so the purpose of the inception is to understand the potential landscape we are delivering in.

When we start the inception we know nothing. We want to come out of the inception knowing as much as we could reasonably expect to learn in the time we have allocated. This discovery is along several axes: technical areas such as the technology stack, potential architectures, integration points and external services; domain questions such as how well we understand the problem and whether it is more about research than solving a clearly-articulated problem; people and process challenges like the path to production, identification of stakeholders, how co-located or distributed the team is and how much of this kind of delivery they have done before. For some of these areas, breaking broad requirements into finer-grained detail is a great way to discover more. But not for all of them, and certainly not at the expense of other discovery activities.

There are good arguments from both the Kanban and Real Options folks about deferring the decomposition of feature sets into features and stories until the “last responsible moment”. This means the information is freshest and you aren’t holding an inventory of atrophying information. You might want a couple of weeks of story-level detail – to promote a consistent flow and avoid starving your process – and beyond that a few features identified that will be broken down into the next candidate stories, but beyond that you shouldn’t be worrying about that level of granularity. The experienced members of the team should be estimating feature sets of the order of person-weeks (or better yet, pair-weeks), not going down to the level of individual pair-days. The less experienced team members should be using the exercise as a learning opportunity.

So please, let’s move beyond this cargo cult approach to inception where we slavishly trot out hundreds of stories with their associated estimates, and remember that we are engaging in a process of deliberate discovery.

Its purpose is firstly to convey to our stakeholders and ourselves an order-of-magnitude sense of size – to quote the Pragmatic Programmers, is it larger than a breadbox and smaller than a house? – and secondly to present the risk landscape in which to understand that estimate.

Hang around my colleagues at ThoughtWorks and you soon get the impression that the only good Enterprise Service Bus (ESB) is a dead ESB. Jim Webber refers to them as Egregious Spaghetti Boxes. So it's not uncommon to hear tales of attempts to get them out of systems that don't need them.

Battle was joined at one client and it brought to mind my younger days playing D&D. Webber swings but misses as the ESB is AC 2, Evan gets a hit and rolls 2d8 for 6 damage. Erik finally kills it by casting "Summon Request Stream Map".

So what was Erik's decisive spell? Essentially the idea was to take a simple request and show how the data for the request and response made their way through the layers of the application. Erik printed out all the code that you needed to read to understand how this would work - which ran to several pages. He also produced this diagram.

It's currently fashionable in agile circles to do Value Stream Mapping as a way to uncover waste in a software development process. I think of this as a request stream map because it similarly takes a request and shows how it moves through the layers allowing us to visualize what's going on and think about the cost and value of the layers.

Layering is an essential tool for building software applications. But like most essential things in life, excess can be almost as much of a problem as too little. A visualization like this (or the multiple pages of code) can help you find where "just enough" is.

One hazard, however. If you do need to transform data from one form to another, it's usually better to a few little transformations than one big transformation. You want to avoid unnecessary transformations not compress the ones you need.

Last week at a ThoughtWorks home-office gathering in New York we had a presentation by one of Microsoft's Architecture Evangelists about their in-the-works cloud offering, Azure. I have to say I was pretty impressed. (If not for the whole Windows thing, I'd probably be trying it out instead of writing about it right now. If you aren't as bothered by Windows as I am, you can register and try it yourself.) Here are some fairly random impressions, mostly based on comparison to the two other best-known cloud contenders.

Azure conceptually leans more towards Google AppEngine's "scale me up; scale me down; hide the details" approach than Amazon EC2's easy-to-manage-infrastructure-as-a-service model, though it currently sits somewhere between them. There's not (yet, at least) any automatic scaling up and down, which surprised me. However what you manually configure is just the number of instances of each application, which is far simpler than manually spinning up new instances of EC2 images and getting them into your cluster (or what-have-you). (Although if you've set up your infrastructure right, your EC2 scaling activities could be just as simple. EC2 just gives you the freedom to do it badly.)

There are two-and-a-half types of application: web apps, worker apps, and web+worker apps. The worker concept is very cool and not something you can do with AppEngine. (The closest thing is their support for scheduled tasks, but these are limited to scheduled URL hits no more often than once a minute with the same restrictions as a normal web request.) Apps all run on something pretty close to the normal .NET framework (minus normal file system access and presumably some other areas--I forgot to ask about threading.)

There are three kinds of storage.

• A file-system-like BLOB storage API, which involves saving individual blocks and then telling the system in what order to stitch them together. It seemed clunky, but I could see writing a simple abstraction over it for small stuff (where you'd usually only need one block) and for large stuff maybe being able to take advantage of already having some blocks uploaded when something gets interrupted and can resume later.
• A BigTable-like "table" storage API. It isn't as rich as AppEngine's BigTable-backed Datastore in that entity properties are restricted to very simple types (no lists, for example) and there's no built-in support for relationships of any kind (even the simple hierarchy of Datastore). (Note there's nothing to stop you from storing keys of other entities as property values and rolling your own awkward relationships.) Similar to Datastore on AppEngine, you have to think about how you want your data partitioned and make some under-informed guesses about slow queries across multiple partitions vs parallel queries each in their own partition. (On Azure, the key for each entity includes an app-generated token naming the partition in which it should reside, which is a bit simpler than Datastore's parentage-based entity groups.) The story on transactions (being limited to a single entity group) seems to be similar as well. There's also currently no indexing by anything other than key, though it sounded like they'll allow you to add at least one additional index per table before they go proper-live.
• Most interestingly, damn-near plain old SQL Server storage in the form of SQL Data Services, with just about every feature of MS SQL Server any normal application would use (except distributed transactions). I believe this is a separate service that you'll need to sign up for and eventually pay for separately. Regardless, it's a key element of the offering, because it provides a huge boost to the suggestion that you can migrate existing .NET applications into the cloud.

There are also persistent queues, which web apps can use to communicate to worker apps asynchronously. (I guess this qualifies as a fourth kind of storage.) They feature guaranteed delivery (with possible repeated delivery) of messages. This is very cool and not something AppEngine offers. (Even if you rolled your own queue-ish thing in datastore, there's no facility to consume messages intelligently.)

There are REST APIs (and SOAP as well, I think) for just about everything, so you have a lot of options for architectures (using heterogeneous technologies) that run partly on-premisis and partly in the Azure cloud.

There are a bunch of other services you can hook into, including ".NET Services" (low-level technical stuff including a Service Bus, Access Control, and a Workflow Service) and "Live Services" (higher-level end-user-meaningful services such as Contacts, Windows Live ID, and assorted Bing stuff).

There's also, surprisingly, the ability to run unmanaged code in your .NET application, AND some form of support for non-.NET stuff via IIS's FastCGI support.

Anyway, it looks like it'll be a compelling offering, competitive with EC2-hosted Windows, and thanks to SQL Data Services much closer to allowing you to put a normal .NET web app into the cloud than AppEngine can do for Java web apps. For any organization willing to trust Microsoft with its infrastructure, Azure is a good reason to be excited. For the rest of the world, carefully executed EC2 still looks like the best bet for the Enterprise in the cloud.

The release of Java support for Google AppEngine means more than just Java: it means lots of cool JVM languages! My personal favorite is Clojure, so when I and several colleagues got an opportunity to try out a pre-release version (thanks to a partnership between Google and ThoughtWorks) I immediately started trying it out. It turns out to be pretty easy and pretty great.

I'm going to walk you through using Clojure on Google AppEngine by stepping you through the creation of a Clojure/Compojure version of the Guestbook application from the Getting Started section of the AppEngine docs. As we go I'll introduce some bits of appengine-clj, a library I extracted along the way to keep anything that felt like boiler-plate code out of my app.

First you'll need to sign up (if you haven't already) and download the SDK.

Creating Your Application

To start, create a project directory and the basic project directory structure documented in the getting started section of Google's documentation. Here's an overview of what to create and what will go where.

project-root/
  src/
    [Clojure source code]
  war/
    [static files]
    WEB-INF/
      [config files]
      classes/
        [compiled classes from ../../src]
      lib/
        [jar dependencies]

Copy clojure.jar, clojure-contrib.jar, and compojure.jar into WEB-INF/lib. You'll also need to add appengine-api-XXX.jar from the SDK. I'm using fairly recent trunk versions of Clojure, clojure-contrib, and Compojure. (The easiest way to make sure you have compatible versions of these is to pull the latest Compojure source from github and then download their deps.zip and build Compojure against that.) If you want to use appengine-clj you can build it yourself from source or grab a prebuilt jar from the downloads section on Github.

Hello, World!

The entry point to your application will be a servlet class. Create a Clojure source file in your src directory and include a :gen-class directive to extend HttpServlet. Use Compojure's defroutes and defservice to create a HelloWorld.

(ns guestbook.servlet
  (:gen-class :extends javax.servlet.http.HttpServlet)
  (:use compojure.http compojure.html))

(defroutes guestbook-app
  (GET "/"
    (html [:h1 "Hello, World!"])))

(defservice guestbook-app)

defroutes above creates a routing function that will respond to an HTML GET for the path "/", and the html function converts vectors to a string of HTML. If you're not familiar with Compojure, start here on the Compojure wiki.

Next create a web.xml with a servlet-mapping sending /* to your servlet class. Since Compojure handles URL routing, your application will have just this one mapping.

<?xml version="1.0" encoding="ISO-8859-1"?>
<web-app 
    xmlns="http://java.sun.com/xml/ns/javaee" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
    version="2.5">
  <display-name>Clojure Guestbook</display-name>
  <servlet>
    <servlet-name>gb</servlet-name>
    <servlet-class>guestbook.servlet</servlet-class>
  </servlet>
  <servlet-mapping>
    <servlet-name>gb</servlet-name>
    <url-pattern>/*</url-pattern>
  </servlet-mapping>
</web-app>

Create an appengine-web.xml, putting your application ID into the application element.

<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
  <application>[your application ID]</application>
  <version>1</version>
  <static-files />
  <resource-files />
</appengine-web-app>

Create an ant build.xml file that compiles your application to the classes directory.

<project name="guestbook-clj" basedir="." default="compile">
  ... here you'll need to define project.classpath ...
  ... see GitHub for the full working example file ...
  <target name="compile" depends="...">
    <java classname="clojure.lang.Compile" classpathref="project.classpath" failonerror="true">
      <classpath path="${src.dir}" />
      <sysproperty key="clojure.compile.path" value="${classes.dir}" />
      <arg value="guestbook.servlet" />
    </java>
  </target>
</project>

I've left out some of the details, but you can find a full working version on Github.

At this point you should be able to run your "Hello, World" iteration of the application locally using the development appserver and deploy to appspot using the appcfg executable (or an Ant task if you prefer). Now it's just a matter of building your application using the tools Google AppEngine, Clojure, and Compojure make available to you.

The User Service

AppEngine has a simple API for dealing with user accounts. Let's greet logged in users by name. You'll need to import com.google.appengine.api.users.UserServiceFactory.

(ns ...
  (:import
    (com.google.appengine.api.users UserServiceFactory)))

...
  (GET "/"
    (let [user-service (UserServiceFactory/getUserService)
          user (.getCurrentUser user-service)]
      (html [:h1 "Hello, " (if user (.getNickname user) "World") "!"]))))

But we have to let users log in to see this work. The UserService also exposes methods for creating login and logout URLs.

...
  (GET "/"
    (let [user-service (UserServiceFactory/getUserService)
          user (.getCurrentUser user-service)]
      (html
        [:h1 "Hello, " (if user (.getNickname user) "World") "!"]
        [:p (link-to (.createLoginURL user-service "/") "sign in")]
        [:p (link-to (.createLogoutURL user-service "/") "sign out")]))))

Now you should be able to log into your application and be greeted by name. On the dev appserver, the login page will let you provide any username and check a box to indicate whether you should be logged in as an administrator for your application. On the appspot servers, you'll get a proper-looking Google Accounts login page. The argument to createLoginURL and createLogoutURL is the path or URL the user should be redirected to after logging in or out.

(ns guestbook.servlet
  ... you no longer need to import UserServiceFactory ...
  (:require
    [appengine-clj.users :as users]))

(defroutes guestbook-app
  (GET "/"
    (let [user-info (request :appengine-clj/user-info)
          user (user-info :user)]
      (html
        [:h1 "Hello, " (if user (.getNickname user) "World") "!"]
        [:p (link-to (.createLoginURL (user-info :user-service) "/") "sign in")]
        [:p (link-to (.createLogoutURL (user-info :user-service) "/") "sign out")]))))

(defservice (users/wrap-with-user-info guestbook-app))

It's about the same amount of code, it just looks a little more clojurey now. (I'm sorry: that's not a word.)

Datastore

Next let's collect guestbook entries and put them in the Datastore. AppEngine for Java has support for a couple of standard Java persistence APIs, JDO and JPA. But we'll use the lower-level datastore API, which seems a better fit for a dynamic language like Clojure (not to mention it doesn't require us to implement Java classes to persist).

The Java API for datastore is pretty simple, but conceptually it's different enough from a SQL database that it definitely takes some getting used to. (I for one am still figuring it out.)

Here's the sixty-second rundown of the bare essentials. The basic unit of persistence is the Entity, which has a Map of String-keyed properties. An Entity has a kind, which is a string denoting the type. (But keep in mind there's no schema here, so you can give any entity any properties.) An Entity is identified by a Key which is something more than a normal DBMS identifier because it can hold an Entity's association with a parent Entity. Besides using the Key, you can retrieve Entities with a Query, which searches either a single kind of entity or descendent entities of a single ancestor (or both, depending on which constructor you use) and can apply simple filtering and sorting.

The natural Clojurized form of an Entity seemed to be a map, so what I've started pulling out into appengine-clj.datastore is functions that allow Clojure code to work with an immutable map of keyword-keyed properties (plus :key and :kind) and have the library take care of translating into Entity objects. Currently there are just create and find methods, since that was all the basic guestbook needed. (But you know that Internet. I'll need a delete function before the week is out.)

Using appengine-clj.datastore, functions for creating and retrieving guestbook greetings are extremely simple.

(ns guestbook.greetings
  (:require [appengine-clj.datastore :as ds])
  (:import (com.google.appengine.api.datastore Query)))

(defn create [content author]
  (ds/create {:kind "Greeting" :author author :content content :date (java.util.Date.)}))

(defn find-all []
  (ds/find-all (doto (Query. "Greeting") (.addSort "date"))))

Note the creation of a com.google.appengine.api.datastore.Query (in find-all) that pulls back all Greetings and orders them by date. I've considered a couple of approaches for cleaning up creation of a query from Clojure, but I haven't decided between something that looks fairly idiomatic vs something that reads just like GQL. For the time being I'm sticking with the Java-interop style since even that is nicely terse and readable. Take a look at the tests for appengine-clj.datastore for more examples.

Speaking of which, this is a good time to mention that writing tests for datastore code is easy with appengine-clj.test-utils, which provides functions to set up an in-memory datastore. The dstest macro used there creates a fresh datastore for each test. If you're using a different testing framework or prefer different scoping, you can call ds-setup and ds-teardown yourself. (Do keep in mind that this is the development version of datastore, so we'll all need to keep an eye out for differences between that and the real Datastore service.)

HTML and Form Handling

Now that we've got our persistence straight (and tested), let's create a UI so users can sign the guestbook. At this point it's just plain Compojure code. We'll create one route to show the guestbook and a form to enter a greeting at "/" and another route for saving the greeting with a POST to "/sign".

Here's our function for signing the guestbook.

(defn sign-guestbook [params user]
  (greetings/create (params :content) (if user (.getNickname user)))
  (redirect-to "/"))

You can see there's very little to it. It takes the request parameters and a user, calls our greetings/create function, and redirects back to the guestbook.

The function for showing the guestbook is quite a bit more to swallow, since it includes our entire user interface.

(defn show-guestbook [{:keys [user user-service]}]
  (let [all-greetings (greetings/find-all)]
    (html [:html [:head [:title "Guestbook"]]
      [:body
        (if user
          [:p "Hello, " (.getNickname user) "! (You can "
            (link-to (.createLogoutURL user-service "/") "sign out")
            ".)"]
          [:p "Hello! (You can "
            (link-to (.createLoginURL user-service "/") "sign in")
            " to include your name with your greeting when you post.)"])
        (if (empty? all-greetings)
          [:p "The guestbook has no messages."]
          (map (fn [greeting]
            [:div
              [:p (if (greeting :author) [:strong (greeting :author)] "An anonymous guest") " wrote:"]
              [:blockquote (h (greeting :content))]])
            all-greetings))
        (form-to [POST "/sign"]
          [:div (text-area "content" "")]
          [:div (submit-button "Post Greeting")])]])))

It takes the user-info map, which it destructures to grab the user and UserService. It calls our greetings/find-all function to get the items to show and then uses Compojure's html helpers to create the document. For any real application you'd want to break the view down into smaller pieces to avoid such a huge nested chunk of vectors (or consider using another templating library like Enlive), but for this example I think it's easier to understand what's going on with the whole page in one function.

Finally here are the routes that wire it all together.

(defroutes guestbook-app
  (POST "/sign"
    (sign-guestbook params ((request :appengine-clj/user-info) :user)))
  (GET "/"
    (show-guestbook (request :appengine-clj/user-info))))

Here I'm using the :appengine-clj/user-info map that's been assoc'd to the request by the Ring middleware.

See the entire servlet file on GitHub, including some enhancements for styling and to see some other code to exercise Clojure features on AppEngine.

The Big Caveat

Two unusual aspects of the Google AppEngine environment create pretty major constraints on your ability to write idiomatic Clojure.

First, an AppEngine application runs in a security context that doesn't permit spawning threads, so you won't be able to use Agents, the clojure.parallel library, or Futures.

Second, one of the most exciting features of AppEngine is that your application will be deployed on Google's huge infrastructure, dynamically changing its footprint depending on demand. That means you'll potentially be running on many JVMs at once. Unfortunately this is a strange fit for Clojure's concurrency features, which are most useful when you have precise control over what lives on what JVM (and simplest when everything runs on one JVM). Since shared references (Vars, Refs, and Atoms) are shared only within a single JVM, they are not suitable for many of their typical uses when running on AppEngine. You should still use Clojure's atomic references (and their associated means of modification) for any state that it makes sense to keep global per-JVM, since there may be multiple threads serving requests in one JVM. But remember JVMs will come and go during the lifetime of your application, so anything truly global should go in the Datastore or Memcache.

More to Come

• I'll try and expand on this in the future with more write-ups, including a discussion of special handling for static files (which as of the version of the SDK I'm using works great on the appspot servers even with a /* servlet mapping but not on the local dev appserver, where servlet mappings win out over static files).
• If you'll sign my silly little guestbook on the appspot servers, I'd like to publish information on how many requests I got and how they performed.
• Google also provides Java APIs for caching, image manipulation, making HTTP requests, and email. I haven't even scratched the surface of those yet.
• With fresh support in AppEngine for scheduled tasks and upcoming support for task queues, there's more Clojure fun to be had.

Enjoy!

There may be a niche in the market, but is there a market for the niche?



How do you create a successful proposition?  If the answer was obvious there wouldn’t be so many failures out there in the market place.

It is easy to commence on a journey of product development with a hunch and clearly there is no substitute for validating ideas in the flesh.  That something at ThoughtWorks we do; helping clients test and learn, rapidly building ideas into tangibles that can be piloted at low cost and low risk before investing in significant build and spend.  However, sometimes a little more rigour is required before you commit to commencing a project in earnest.

That rigour needs to be focused.  What often happens is this rigour turns into a research phase that turns into a project itself.  It need not be this way.  There are certain things you can do, certain questions to ask as you set out on the journey of creating a new, compelling customer proposition.  What follows then is a strawman model to help test potential propositions before moving forward with them.  There are three components to the model, the customer, the environmental context and the organisation or company.

All too often propositions are rooted in the organisation.  They make assumptions about the demand or usage. This model attempts to broaden the analysis and focus upon the customer and the why the proposition will be attractive to them.  The model supports questions that may be asked to help shape thinking, test hypotheses and validate thinking.

I do not propose that this should become a major research exercise  (for example market sizing is a huge effort in itself), rather a tool for asking the right questions, and if the answers are hard to come by, maybe that suggests more thought is required in refining the proposition.

So here goes, a model that provides a framework for considering new customer value propositions.  It’s just an initial idea and I’d welcome feedback and suggestions.

Customer

Before you get too carried away with the proposition, a good starting point would be the customer.  Who are they and what do they do.  Let’s remember that your customer is not everybody.  Your proposition in unlikely to be appealing 24/7.  The challenge is to segment your target market and identify the triggers for action.

The persona: Who do?

Personas are a useful tool for bringing the customer to life.  Much has been written about them, but they are a useful tool for extracting broad data into specific stories that describe individuals. Realise that it is unlikely you will design for everybody. Start with the market that you are targeting, how large is it and what is its propensity to spend? Then within that target market segment the target customer base into different profile customers (personas). You need to understand which persona, which customer profile is most important – prioritise them and focus on the highest value.  This may mean deciding between high volume, low margin mass market and low volume, high margin niche appeal.  This decision needs to be made as early as possible to ensure the proposition remains focused and doesn’t try to be all things to all people, satisfying none.

Values, needs, wants and desires

People are not empty vessels waiting to consume and be filled with your proposition.  Their behaviour is driven by their values, needs, wants and desire.  These may be fundamentally rational (to satisfy a basic human goal) or emotional (to demonstrate status). They are cultural and time based.  Thinking in these terms helps you understand how the proposition will appeal to the customer at different levels.  Let’s take an example of this; a new mobile phone.

Before we think about what the product must do, what are the values that the persona associates with the phone. Is our target market a technophile or a technophobe? Jan Chipchase who works for Nokia includes ethnography in his research to understand how people use their phones; women carry them in their handbags, men in their pockets or their belts.

The basic need that the phone must meet to satisfy the customer, she must be able to make and receive calls.  If the product is unable to meet these needs it is not fit for purpose and the phone proposition will inevitably fail.

Just making phone calls meets the need but there are additional wants that should be satisfied for the product to be more compelling.  It’s a hassle to remember the number of every person she rings, the customer wants to be able to store numbers and see the number of the person who is calling.

Having the ability to see a photograph of her daughter as a screen saver on her phone is neither a need not a want.  The phone is useful and usable without that.  But the customer desires to personalise her phone by having a picture of her daughter on it.  Desirability is the key differentiator of the iPhone.  It doesn’t need to compete on features, it is a cool device that people talk about.  And here is a key decision you need to make on your proposition journey.  Are you looking to compete on parity or whether you want to make a difference.

Questions

• What is the basic need that the proposition is trying to fulfil?
• What counts as hygiene?
• What does the customer need to be satisfied?
• What does the customer want in addition to being just satisfied
• What do other competive products do to maintain feature parity (if you feel you really need to compete on features alone – bad move!)
• Few people would argue they don’t want simplicity and clarity in their interactions with products.  How could your product to make life easier for the customer?
• What will make the customer feel good in themselves about owning the product?
• What other products are “cool” or desirable to your target market.  How can you leverage the essence of those products?

Context

So now we are beginning to understand who the customer is, it is time to nest the proposition in terms of their context.  The old maxim that a half drunk bottle of water in a desert is worth its weight in gold, but on the streets of a city is worthless trash, should be remembered.  Even the best of propositions will deliver little value if they not only consider the customer, but also the context in which they apply: time, demand and usage.

Trigger

So the next step in the model is to ask why, when and how will the customer be attracted to the proposition. What is the trigger that drives the customer to move from awareness (assuming you have that) to action?  There is no point in a financial services company trying to sell me a car loan if I am wealthy enough to own my own car, or I do not drive.  Understand what triggers the customer to be interested in the proposition, when and why this happens.  How can your proposition be at front of mind when the trigger is set.

Questions

• What lifestyle / lifestage events will trigger?
• Internal events personal to the customer; leaving school, getting a first job, getting married, moving house, retiring etc
• External events that they have no control over (think about sports sponsorship and tying a proposition to that sport, or tying a proposition to a celebrity e.g. Michael Jackson..)

Environment

It is very unlikely that the proposition will be wholly unique.  What is the competitive landscape, what noise will it need to be heard above to capture the consumers attention.  Whilst you may review the immediate competitors to see where threats and opportunities lie, what can you learn from other, unrelated products or domains?  How can you fuse together concepts from outside your immediate focus to bring new innovation to your product?  Scenario planning may come in useful, playing out different outcomes for different timelines other than that which you plan for.

Questions

• What is the competitive landscape?
• What can you learn about similar but unrelated propositions?
• Have you considered the political, environmental social and technical influences using the old PEST analysis?
• Have you considered different scenarios and how your proposition would play out under them; what unplanned disruptors could get in the way, or how could your proposition done differently disrupt the market?

The experience engine

Enough of the customer and externalities, what will the proposition look like and why will the target customer go with it? There are three engines within the organisation that drive the proposition, the experience, delivery and value engines.  So…

Utility

To be any good, the product has got to offer basic utility.  It has to do what it says it is going to do.  Sadly, too many products and customer propositions end there.  A utility product will match the consumers needs.  This is where most enterprise software sits…

• What are the key customer needs that the proposition must fulfil?
• What is the basic core functionality that must be met, what are the features that must be offered to gain traction in the market place?
• What features that are typical on competitor products that we could do without?

Quality

I could call this next box usability (as this follows the UXD model) but I think it goes beyond just usability.  What is the quality of not only the immediate interface, but also with the supporting functions?  For example, if you have a call centre to back up the proposition, how many layers of IVR are you forced through?

• Have you considered usability?
• Is the packaging aesthetically pleasing?
• The “happy path” customer journey may be well framed, but what about the “sad path”?  What about when things go wrong, what about when customers don’t act in the way you expect of predict them to act?

Brand

It is easy to get carried away with a new idea before thinking about what it means to the brand.  Typically there will be a strategic roadmap and whilst the proposition may be attractive it may not fit into where the brand is going.

• Is the proposition complementary to the overall brand direction or does it require a new brand and identity?
• Does the proposition support / leverage the brand?
• Does the brand already ‘do it’ under another guise (are you reinventing a wheel that has already been tried somewhere, sometime in the organisation’s history?)
• How will it be marketed?

Community

Finally, what is the ‘buzz’ that the proposition will create, what will get people talking and sharing it and how will you create this buzz.

• Is there a social network component built in that gets people talking and connected?  How will it get people talking in external networks?
• What will cause people to recommend it to others?
• How can customers become part of its evolution?
• What of the proposition will get people passionate, what will drive them away?

Delivery engine

People

A successful proposition needs not only a talented, passionate and committed team to deliver it to market, it also needs a similar team to run it and support it when it is live.  It is a common failing for a rogue “skunkworks” team to emerge in an organisation and develop what appears a compelling proposition, only to have it knocked back and closed down by the “Business as Usual” processes inherent in the organisation

• Who do you need to make the proposition successful?  What is the team?
• Who will create the proposition and who will lead it?  Is it IT led or business led?
• What are the cross-organisational boundaries that the proposition crosses and how will these be eliminated?
• Who will take ownership of the proposition once it crosses over into the market?

Process

• What are the processes that will be required to sustain the proposition?
• If the proposition will require changes to the organisation, how will they be managed, communicated and rolled out?
• How will the proposition be supported once it is let loose in the market?
• How will it be communicated to customers?
• How will you create new sales – sales force.

Technology

• What is the technology that will underpin the proposition?
• Is it possible to test the ideas using rapid languages such as Ruby on Rails before committing it to the enterprise Java stack?
• What integration is really necessary and what can be worked around?
• How can you deliver a beta version in the shortest period of time?
• How will you avoid heavyweight frameworks and develop incrementally to deliver value early and often?
• How performant and scalable must the innovation be?

Value Engine

At its most simplistic, how much will the proposition cost and how much revenue will it generate?  Does it offer cost saving opportunities?  Are there intangible benefits that will be accrued?  Ultimately is it a viable proposition that is worth pursuing, or will the cost to develop and run outweigh the value it will add?  Building out a financial model can take time, in the first instance this should be a napkin analysis, a wake-up call to make sure there is value in the proposition before too much time is invested in it.

Cost

Every day someone is working on the proposition it is costing you money.  The quicker you can get something to market the faster you will start seeing a return on your investment, similarly the sooner you can “get something out there”, “test and learn” the sooner you can kill a proposition that does not fulfill its promise.

• How quickly can you get a beta to market?
• How many people, how many days?
• What will the cost be to develop the infrastructure?
• Do you have the skills in house or will you need to go external?

Benefit / Revenue

At its most crude, how will the proposition make money, but there may be more to what we wish to achieve.  Is the proposition actually going to cut costs, a result of regulatory pressures or a CSR initiative?

What are the benefits that will be accrued – both tangible (e.g. financial) and intangible (e.g. social, environmental etc)

• If you are selling units are you going for high volume low margin or low volume high margin?
• If it an on-line proposition “advertising” is often seen as the source of revenue.

There are two additional components to the model…

Implementation

Having a compelling proposition is one thing, it is another to successfully communicate it and roll it out to target customers.

• In a crowded market place, how will the proposition stand out?
• What are the brand values it will communicate?
• What is the story that customers will hear and how will they hear that story?
• How will customers interact with the proposition, what channels will you use to take it to market?
• What is the roll out strategy?

Retain and grow

Winning customers is only the first step.  A successful proposition will maintain a long-term relationship with its profitable customers, maintaining the warmth they have to the original proposition and cross-selling and up-selling new ones.

• How will you retain them and turn them into repeat customers and passionate advocates of the proposition?
• How will the proposition grow lifetime customer value?
• What can be cross-sold or up-sold?
• What can you bundle?
• How will the proposition deal with churn?

OK, so it’s not a perfect model and by no means complete.  There’s some duplication in the thinking and many questions missing, but as any model it can be used to guide and prompt thinking and ensure there are no elephants left in the room when the first line of code gets cut.  I’d welcome any comments on its usefulness, utility and direction.

OK, OK, I admit it. Maybe significant whitespace isn't all bad. (But don't let me ever catch you quoting me say that.)

The reason for my (maybe) shift in thinking? Manning Publications sent me a copy of Iron Python in Action, and I have to say, I like the book and its approach. Getting me to like Python as a primary language for development will probably take more than just one book can give, but... *shrug* Who knows?

Bear in mind, I have plenty of reasons to like IronPython (Microsoft's Python implementation for the .NET environment):

• A good friend of mine, Harry Pierson (aka @DevHawk), is the PM on the IPy project, and I'm generally prejudiced in favor of those things that people I know and respect.
• I'm generally a fan of dynamic languages, particularly those that let you do strange and twisted things to the type system and its instances at runtime. (Yes, I'm looking at you, ECMAScript...)
• I spent some quality time with IronPython Studio last year while researching a Visual Studio Extensibility "Deep Dive" paper.
• I've known Jim Hugunin (the creator of IronPython, and Jython before that) for some years, ever since his days working on AspectJ, and he's one of those scary-smart guys that, despite knowing they're scary-smart, still render me stunned when I listen to them.
• I'm a huge fan of the DLR. It's like having Parrot, but without having to wait a decade (give or take).

But, just to counterbalance the scales, I have plenty of good reasons to dislike IronPython, too:

• Significant whitespace.
• The "There's only one way to do it" oath that Pythonistas seem to hold as religion. (Somebody told me that building C-Python—the original implementation—only works for you if you swear a holy oath to The One True Way on the One True Way Bible. Needless to say, I believe them, and have never tried to build C-Python from sources as a result.)
• Significant whitespace.
• Uh.... did I mention significant whitespace yet?

I admit, it was with some hesitation that I cracked open the book. Actually, to be honest, I was really ready to just take out all my dislike of significant whitespace and pour it into a heated, vitriolic diatribe on everything that was just wrong with Python.

And...?

Well, OK, I admit it. Maybe significant whitespace isn't all bad.

But this is a review of the book, not the technology. So, on we go.

What I liked about the book

• The focus is on both .NET and Python, and doesn't try to short-change either the "Python"-ness or the ".NET'-ness by trying to be a "Python book (that happens to run on .NET)" or a ".NET book (that happens to use Python for code samples)". The authors, I think, did a very good job of balancing the two, making this the book to get if you're in that area on the Venn diagram where "Python" overlaps with ".NET".
• Part 2, "Core development techniques", starts down the "feed you the Python Kool-Ade" pretty quickly, heading straight into Chapter 4 ("Writing an application and design patterns with IronPython") without much of a pause for breath. The authors get into duck typing, protocols, and Model-View-Controller within the first four pages, and begin working on a running example to highlight some of the ideas. (Interestingly enough, they also take a few moments to point out that IronPython on Mono works, and include a couple of screen shots to that effect as we go, though I personally wonder just how many people are really going down this path.) I like the no-holds-barred, show-you-the-code style, but only because they also take time throughout the prose to talk about some of the concepts at work underneath and laced throughout the code. "Show me then tell me" is a time-honored tradition, but too many authors forget the "tell me" part and stop with code. These guys do a good job of following through.
• The chapters in Part 3, "IronPython and advanced .NET", form an interesting collection of how IronPython can fit into the rest of the .NET stack, demonstrating how to use IronPython with WPF, ASP.NET, and IronPython's crowning glory, Silverlight. If you're into front-end stuff, this is the section where I think you're going to have the most fun.
• The chapters in Part 4, "Reaching out with IronPython", is I think the most important part of the book, showing how to extend IronPython (chapter 14) with C#/VB extensions (similar to how a C-Python developer would extend Python by writing C code, but much much simpler) and the opposite—how to embed IronPython inside of existing C#/VB applications (chapter 15), which is really an exercise in using the DLR Hosting APIs. While the discussion in chapter 15 is good, I wish it'd had a bit more thorough discussion of how the DLR could be hosted regardless of the scripting language, though I admit that's pretty beyond the scope of this book (which is focused, after all, entirely on IronPython, and as a result should stay focused on how to host IPy).

What I found "Meh" about the book

• Part 1 ("A new language for .NET", "Introduction to Python", and ".NET objects and IronPythong") does a good job of bringing the rank beginner up to speed, getting some basic Python ideas across in the same breath that they bring .NET home. The only problem is, it only works well if you're neither a Python programmer nor a .NET programmer. Chapter 1, for example, does a sort of Cannonball-into-the-pool kind of dive into Python, but dives equally into the "Iron" parts as it does the "Python" parts. If you're either a Pythonista or a .NETter, I suspect you're going to be tempted to flip pages pretty quickly, and (I suspect) miss a few things. Chapter 2 is all about Python (meaning .NETters will probably spend some time here), but it certainly doesn't feel like an exhaustive reference, nor does Chapter 3 stand as an exhaustive discussion about all things .NET, either. I almost wish all three chapters had been collapsed into one—suffice it to say, I don't feel like I know the Python language, and don't feel like this book could be my Python reference next to me as I learn it, and I know that it's not a great .NET reference, either. Fortunately, the goal of these three chapters feels pretty clearly to be "Teach you just enough to make you dangerous (and able to understand the rest of the book)", and once we hit Part 2, rubber meets road pretty quickly.
• By the time you hit Chapter 7, less than halfway through the book, the authors have created a fairly nice, if simplistic, application for later dissection, but it's not until you hit Chapter 7 that they begin to start unit-testing, even though they insist (on page 17) that "Dynamic language programmers are often proponents of strong testing rather than strong typing" (a quote they attribute to Bruce Eckel, though I'm relatively certain I heard Dave Thomas and Neal Ford say it with respect to Ruby, long before Eckel started "Thinking in Python... or Flex... or whatever"). If unit-testing is that important, why wait three chapters into the application's development before writing a single unit-test? This doesn't jibe with me, somehow.
• If you're into back-end stuff, chapter 12 on "Databases and web services" is pretty bland. The fact that the two are combined into a single chapter is indicative, all by itself, of how deep or intensive the coverage goes, and there's zero mention of anything beyond basic ADO.NET. The coverage on web services covers REST relatively well, but there's zero coverage of WCF, and the whole of SOAP-based services is all of four or five pages. And Workflow? Doesn't exist, isn't even mentioned (except for an appearance in a table, "The major new APIs of .NET 3.0"). Yikes.

What I actively disliked about the book

Actually, not much. Manning did their usual superb job of arrowed callouts to point out particular concepts in the code listings, the copyediting is professional (meaning there's no obvious typos or misspellings that just break up the flow of prose, something that not all publishers seem to take seriously), and the graphics flow nicely alongside the prose, not dominating the page but accentuating it.

In fact, about the only thing I'd care to criticize is the huge number of footnotes, particularly in the first chapter. (By page 20 in the book, there have already been 30 footnotes.) When you have three footnotes per page, on average (and sometimes more), it does tend to distract, at least to me it does. It feels like there were ways, for most of them, to inject the idea or concept into the main prose, or leave it out entirely, but that could just be a difference of writing style, too.

Summation

If you're a .NET developer interested in learning/using IronPython on your next project, this is a definite winner. If you're a Python developer looking to see how to break into .NET, I'm not so sure this is your book, but I say that mostly because I'm not a Pythonista and can't really speak to how that mindset will find this as an introduction to the .NET space. My intuition tells me that this would be a good springboard into another book on .NET for the Python programmer, but I'll have to leave that to Pythonistas who've read this book to comment one way or another.

At some point last year I was asked to review the architecture of the software behind a large and popular website. The resident architect explained how he had followed a modern approach, decoupling the web front-end from back-end services that provide content. To gain further flexibility he had put the front-end and the services on an ESB, mostly to cater for the possibility to make the content available to other consumers. In short, the architecture diagram looked a lot like many others: nothing to see here, move on.

The diagram above only shows one of the content services, which for the sake of this article is a service that provides contact details for a person.

Based on conversations with the project sponsors I began to suspect that at least the introduction of the ESB was a case of RDD, ie. Resume-Driven Development, development in which key choices are made with only one question in mind: how good does it look on my CV? Talking to the developers I learned that the ESB had introduced “nothing but pain.” But how could something as simple as the architecture in the above diagram cause such pain to the developers? Was this really another case of architect’s dream, developer’s nightmare?

Often a picture says more than a thousand words, but at the same time traditional architecture diagrams simply do not provide enough information. (I’ve argued this before, here and here for example.) The answer to my questions, and the anxious project sponsors’ question whether the project would deliver on time, lay in a different picture.

The picture should not only show the layering of the system in more detail than the architecture diagram but it should also visualise the complexity introduced by the ESB. It’s often possible to produce more detailed diagrams automatically from source code, configuration, or runtime inspection but in this case I couldn’t find a way to do that, which is which I drew the following:

In the centre column this diagram shows the layers involved, from the Struts action that needs to look up details by an ID down to the back-end that provides a “RESTful” service to retrieve the details. The top box contains the front-end layers while the box in the middle represents the ESB. The layers labelled in italics are frameworks or infrastructure software, the rest is project-specific code.

Key to this diagram are obviously the coloured representations of data travelling up and down the stack. Primitive types are grey, Java objects are purple circles, XML documents are shown as turquoise hexagons, and HTTP messages as green blobs. Mule messages, which are Java objects, are shown in dark purple. If one representation is wrapped in another it’s displayed inside the wrapper.

The diagram highlights that it’s a simple piece of data that travels all the way down the stack (actually further until it hits a relational database at some point) and, equally, that the data needed by the action is available in the right form from the service. All that the intermediate layers are doing is wrapping and unwrapping the data in various ways. In fact, out of the four layers of project-specific code three layers do nothing but wrapping. No wonder the developers felt some pain.

As a side note: the diagram also makes it obvious that the wrapping is asymmetric between the WsToRestHandler and the RestServiceWrapperImpl, which makes the point of this layer boundary even more questionable.

This diagram made people more interested in the cost of what had just been a simple, innocuous line in the architecture diagram; which was the point of the exercise. So, what would this diagram look like without the ESB? Easy:

Apart for the obvious simplicity the diagram highlights that every layer has a purpose and there is only one wrapping, which is not undone either. Would it surprise you if I told you that the project went live without the ESB?

Footnotes

1) This article is by no means implying that any of the mentioned technologies are useless or harmful. Their use was harmful in this case.

2) The pattern of wrapping and unwrapping the same data through multiple layers is not new, it was also quite common in early 3-tier architectures. (In fact, some people called their architectures n-tier; I guess to show that the number of tiers was somewhat arbitrary but large.) We had a term to describe this pattern: the wormhole pattern.

3) This project is not an isolated incident of a Wormhole ESB. I have personally seen several in the past years.

What's the most common programming language in the world?

I'm not sure how you could go about measuring this, but one thing you'd need to do is consider what we mean by programming. My candidate answer considers that the most popular programming language is one used widely by people who do not consider themselves as programmers. This language is Excel, or more generally spreadsheets.

Spreadsheets are easily used for small tasks, but are also used for surprisingly complex and important things. Often I've seen professional programmers gulp when they realize that some vital business function is being run off some spreadsheet that they'd find too complicated to muck with.

In general, we've not had much success with programming languages for these kind of LayProgrammers. Whenever someone talks about some new environment that's going to allow people to specify complex behavior "without programming" I mention COBOL, which was originally designed to get rid of programmers. So it's important to consider what Excel can teach us about programming environments.

One property of spreadsheets, that I think is important, is its ability to fuse the execution of the program together with its definition. When you look at a spreadsheet, the formulae of the spreadsheet are not immediately apparent, instead what you see is the calculated numbers - an illustration of what the program does.

Using examples as a first class element of a programming environment crops up in other places - UI designers also have this. Providing a concrete illustration of the program output helps people understand what the program definition does, so they can more easily reason about behavior.

So why do I feel we need this particular Neologism? Essentially because I think it deserves more thought. We pass by illustrative programming examples without really thinking about them or what makes them special - or even that they are special in some way. We've used illustrative programming for years, but we've not paid enough attention to it. We've not thought enough about what are its essential qualities and what its strengths and weaknesses are.

I've chosen the term "Illustrative Programming" to describe this, partly because "example" is so heavily used (and illustration isn't) but also because the term "illustration" reinforces the explanatory nature of the example execution. Illustrations are meant to help explain a concept by giving you a different way of looking at it - similarly an illustrative execution is there to help you see what your program does as you change it.

When trying to make a concept explicit like this, it's useful to think about the boundary cases. One boundary is the notion of using projections of program information during editing, such as an IDE that shows you the class hierarchy while you are working on the code. In some ways this is similar, as the hierarchy display is continuously updated as you modify the program, but the crucial difference is that the hierarchy can be derived from static information about the program. Illustrative programming requires information from the actual running of the program.

I also see illustrative programming as a concept beyond the classic REPL loop of dynamic languages. REPL loops allow you to explore execution, but they don't make the examples front and center in the way that a spreadsheet does its values. Illustrative programming techniques put the illustration in the foreground of your editing experience. The program retreats to the background, peeping out only when we want to explore a part of the illustration.

I don't think that illustrative programming is all goodness. One problem I've seen with spreadsheets and with GUI designers is that they do a good job of revealing what a program does, but de-emphasizes program structure. As a result complicated spreadsheets and UI panels are often difficult to understand and modify. They are often riven with uncontrolled copy-and-paste programming.

This strikes me as a consequence of the fact that the program is de-emphasized in favor of the illustrations. As a result the programmers don't think to take care of it. We suffer enough from a lack of care of programs even in regular programming, so it's hardly shocking that this occurs with illustrative programs written by lay programmers. But this problem leads us to create programs that quickly become unmaintainable as they grow. The challenge for future illustrative programming environments is to help develop a well structured program behind the illustrations - although the illustrations may also make us rethink what a well structured program is.

The hard part of this may well be the ability to easily create new abstractions. One of my observations of rich client UI software is that they get tangled because the UI builders think only in terms of screens and controls. My experiments here suggest to me that you need to find the right abstractions for you program, which will take a different form. But these abstractions won't be supported by the screen builder as it can only illustrate the abstractions it knows about.

My colleagues Rebecca Parsons and Neal Ford have been spending a lot of time involved in thinking along these lines too. So here's some thoughts that Neal had in an email exchange

One of the few people to take illustrative programming seriously is Jonathan Edwards. He's come up with many very imaginative ideas as to what such an environment should look like. His vision of illustrative programming is also closely bound to the notions of projectional editing and controlled copy-and-paste.

The trigger for me in wanting to coin a term here, is the use of illustrative programming by Language Workbenches by people like IntentionalSoftware. These Language Workbenches encourage you to build illustrative DSLs. Using illustration is important in this case since this should help engage lay-programmers, which is one of the aims of using DSLs. The challenge is to do this without falling into the trap of poor program structure.

不知道从什么时侯开始，周边的人把我当作一个老手，尽管有些自己不那么情愿，但现实是和我在一起工作的人大多比我工作经验少。



新手之所以为新手，是因为他们的经验较少，所以，不可避免的会犯一些错误。记得别人还在用新手标准要求我时，有一次出差，到了现场，项目负责人要我去安装我们的程序，可我根本就没把我们的程序带来，结果可想而知，项目负责人劈头盖脸的把我骂了一顿。



换我扮演老手的角色，我会尽我所能把项目中的一些不确定问题解决掉，然后，让新手们去解决那些确定性问题，通常，他们的能力应对这样问题会游刃有余。最近一年多的项目里面，涉及到了很多探索新技术的工作，我会从项目中挑出一个最简单的情形用新技术实现出来，这个例子可以帮助我弄清楚技术背后的来龙去脉。之后，我就可以把他解释给其他人。我的同事们都很聪明，理解了基础结构，加上一个现成的例子，他们就完全可以继续进行下面的工作了。



或许单独看起来，我的做法并没有什么特别的地方。刚好最近项目里面有两件点要去探索，我自己做了一个，把另外一个分给另外一个同事。他很快就做完了，而就是这个最简单的例子，我也做了好几天。探索的工作结束，进入到正式的开发工作，其他同事很快就可以接着我的工作继续开展。就是这个最简单的例子，除了了解技术的目的之外，我还写所有相关的脚本，万事俱备，只欠东风。而当我了解另外一个同事的工作时，我发现，他真的只做了一个最简单的例子，了解了一些基本概念。真正把这些内容运用到项目里时，左一个困难，右一个麻烦相继出现，我陪着他一个个克服了这些点，又把脚本预备好，几天之后，才真正进入到可以开发的状态。



有人曾经问过我，你把有趣的工作都做了，把无聊的工作留给别人是不是合适呢？毕竟每个人都希望在项目中成长。前面说过，我的工作是削除项目中的不确定因素。是的，很多人喜欢的就是解决不确定问题带来的快感。但不确定对每个人来说是不一样的。只要最基础的例子跑通了，这项技术的不确定性对我而言就解决了，但实际上，在具体使用这项技术中还会有一些不确定的问题等待解决，解决这样的问题，同样可以提高。



谈到成长，每个人都希望自己能够不断的成长，但是，即便是做同样的项目，每个人得到的机会也是不一样的，这样的差异实际上对应着各人不同的表现。和其他人一起工作的过程中，通过观察，我会把不同的工作交给不同的人来做。像前面说的那个同事，我之所以肯把一个探索的机会交给他，是因为他在之前的工作中表现出的态度和能力，虽然他的探索并不完全令人满意，但是，有了之前的信任，我会告诉他，怎么做才会做得更好。



之所以会想起这个话题，是因为一个同事与我聊起他项目焦头烂额的状态。他在那个项目组中，扮演着和我类似的角色，他感觉自己做得非常累，项目进展也有些问题，于是，我们俩聊起了如何发挥其他人的作用，既让新人感觉自己得到了锻炼，也让自己轻松一些。

I did a presentation of some of the stuff that I've learnt from playing around with F# over the last six months or so at the most recent Alt.NET Sydney meeting.

I've included the slides below but there was also some interesting discussion as well.

• One of the questions asked was around how you would deal with code on a real project with regards to structuring it and ensuring that it was maintainable. I'm not actually sure what the answer is to this question as I haven't written any code in F# that's in production but there are certainly applications written n F# that are in production - the main one that I know a bit about is one which Amanda Laucher worked on which she spoke about at the Alt.NET conference in Seattle.
• There was some discussion about dynamic v static languages - Phil spoke of not caring about what type something is rather caring about what it does. I pretty much agree with this and I think when using languages which have quite strong type inference such as F# (and more-so Haskell from what I hear) then I think we do move more towards that situation.
• Erik raised the point that functional languages aren't the solution for everything and I certainly feel it's niche is probably around operations with heavy data parsing/mining involved. I'm not sure I'd fancy doing an ASP.NET MVC application only in F# although I've seen some WPF code written using F# (unfortunately can't remember where) which looked reasonable so I'm not sure we should write it off just yet.

I've put the code that I walked through in the presentation on bitbucket.

I just made Jurgen Appelo’s list of ‘Top 200 Blogs for Developers’. I’m sneaking in there at number 198, apparently down from 150 when Jurgen last published the list. It’s my own fault for not blogging for almost three months!

There are some great blogs in the list that I’m already aware of and a heap more that I’m going to trawl my way through. Let me know if you come across any that are worth a look…

Oh, and the irony of being a non-developer appearing in the list isn’t lost on me either!

You’ve logged into your on-line banking checked your balance, paid your bills.  What do you do now?  Click on the logout button?

What do you expect will happen now?  Well given that you have actively chosen to log-out (it’s not something you are likely to click on my mistake), you’d expect to do exactly that.  Logout.  The next screen you get will probably be something that thanks you for on-line banking, with a cross sell for a product or two.

That’s what I assume most customers would expect.  So what are Alliance and Leicester thinking about with this screen?

The customer has clicked log-out but they are still logged in?

Why?

“You are still logged in to Internet Banking - before you go have a look at Your offers.”

Excuse me, I logged out, I don’t need to be logged in for you to show me offers.

Worse: “Are you sure you want to log out?”

OF COURSE I WANT TO LOG OUT!!! Why else would I have clicked the link.

Alliance and Leicester fail here in a fundamental usability rule, that of managing the customer’s expectation. In an application where security isn’t paramount this would be an error, in an application where customers expect their action of leaving their secure accounts will do exactly that… but doesn’t, is inexcusable.

好久没有过了也～～Cruise全绿，连IE6的测试也通过了也～～

Cruise是个让人快乐的软件，Mingle是个让人不快的软件，因为Cruise不要你去管它，Mingle总要你拖来拖去 ^_

Business is tough right now, and it’s going to be so for a while. In tough times, you want to be very good at what you do. The more “fighting fit” you are, the more likely you are to survive a challenge.

Unfortunately, IT isn’t all that good at what it does. In fact, on the whole, it’s pretty bad. That means that IT isn’t very well prepared for this downturn.

How bad is it? The research organizations have historically reported a pretty high failure rate of IT projects: about 30% of all IT projects fail outright, while another 30% disappoint their business sponsor (e.g., excessive cost, wrong functionality).¹ On the whole, an IT investment has, at best, a 4 in 10 chance of success.

Companies are already reticent to invest in this climate. IT doesn't offer scared capital a safe haven.

It also suggests that IT is on a trajectory of self-destruction. If we want to look ahead to where IT is headed, we need look no further than present day Detroit.

Photo credit: Ben Wojdyla, The Ruins of Detroit Industry

How did this happen? Consider some of the forces that have shaped the current IT landscape in the past 20 years. The steady growth of IT that was accelerating slightly with the advent of client/server gave rise to explosive growth driven by the combination of internet and Y2K. By the mid-1990's, demand for IT was dramatically outstripping supply. To satiate this demand, IT went in pursuit of scale. To get scale, IT took professional jobs and codified them into industrial tasks, because it’s easier to staff vast numbers of people in highly specialized roles than it is to develop professional capability to solve business problems using technology.

Today, businesses buy, recruit, staff, govern, gatekeep, develop, analyze and test following a model that puts a priority on “big.”

Unfortunately, all the time we’ve been in pursuit of scale, we’ve not been in pursuit of results. Results are assumed. We assume armies of specialists will follow an explicitly defined project plan to produce a solution that is technically sound, functionally complete, and financially satisfactory, all with minimal risk of impairment.

With a 4 in 10 batting average, results cannot be assumed.

By placing a priority to scale, IT mistakes effort for results. We often see success expressed as a function of hours to be invested. It isn’t that simple. Successfully delivering an IT solution is a function of a lot of factors, such as clear communication, effective collaboration and capability; well-informed decision making about technology, functionality and commercial viability throughout the life of a project; flexibility and responsiveness; and ultimately, producing meaningful things for our business partners. These can’t be captured in task orders and forecasts of work effort. They’re lifestyle decisions of how IT goes about its business.

It is time to restructure IT, to move away from an effort-centric industrial model, towards result-centric professional one.

¹ As an example, the 2009 CHAOS report from The Standish Group shows that things haven't changed all that much, reporting 44% of IT projects were challenged (late, over budget, and / or with less than required features and functions) while another 24% failed.

find . -name "*.sql" | xargs cat | wc -l

I recently had the pleasure of supporting a new system throughout its first month of production. This was a good opportunity to refresh my command line skills. As it happened I spent a lot of time looking at log files trying to figure out what happened to the productions system. I figured, that a graphical representation of the events would be nice and started using gnuplot.

First I started out with a bunch of bash scripts, using what your usual unix installation provides, but then I actually came up with some groovy scripts to provide better abstractions. A log file generally looks somewhat like this:

04/01/1970 07:55:13 garbage
04/01/1970 09:27:48 Event 2
04/01/1970 10:01:28 garbage
04/01/1970 10:38:30 garbage
04/01/1970 10:48:36 garbage
04/01/1970 10:51:58 Event 2
04/01/1970 11:03:45 garbage
04/01/1970 11:34:03 Event 1
04/01/1970 12:24:33 garbage
05/01/1970 04:35:50 ERROR

There is a lot of garbage plus some events we might be interested in. It allows to specify events, e.g. by providing a regexp:

 
Event EVENT1 = new RegExEvent("Event 1", ~/Event 1/)
Event EVENT2 = new RegExEvent("Event 2", ~/Event 2/)
Event ERROR = new RegExEvent("Error", ~/ERROR/)

The next step is newing up a TimeLineVisualizer on these events and passing in a stream with the actual log:

 
def logFile = new File("test.log");
 
logFile.withInputStream {InputStream stream ->
  def visualizer = new TimeLineVisualizer([
          EVENT1,
          EVENT2,
          ERROR
  ]);
  visualizer.visualize(stream)
}

If you have the gnuplot binary on your path this will yield something like this:

Also in some cases you would like to know which time of day events are most likely to happen. For producing histograms I created another visualizer (which currently takes only one event).

 
logFile.withInputStream {InputStream stream ->
  def visualizer = new HistogramVisualizer(EVENT2, HistogramVisualizer.HOUR_OF_DAY_BINS)
  visualizer.visualize(stream)
}

For the example log file, which unfortunately has an even distribution of events, we get this:

The cool thing about gnuplot is, that you can actually run these things in a cron job to produce daily reports (and mail them to the appropriate people) or on a continuous integration server to visualise how the system is being exercised by the test suite.

Service Orientated Architecture (SOA) is something that is easy for the lay person to understand.  (Try getting a techie to explain REST and you will see the attraction of SOA to the business person - it’s understandable!)  Understandable, in my non-techie hands, is dangerous.  I am entirely unqualified to pass judgment on it, but there are a couple of  things I’ve observed and have been on my mind when I’ve seen SOA nastiness going on.  So excuse me whilst I wax lyrical.

Problem 1. IT haven’t got a clue.

My (lay) understanding of SOA: IT build ’services’ that can be consumed by different applications.  SOA enables us to remove duplication and build the foundations for a scalable architecture that will accommodate changing requirements as the business evolves and grows.  Herein lies the problem; IT build ’services’, second guessing what the business actually needs from said service.

Exhibit One:  Customer Details Service. It exposes details of customers to any application that will use information about customers.   It was designed by the architect in isolation based upon what IT believe a Customer Details Service will be required to do (e.g. the nature of the fields, the domain etc).  This is even though no application has been built, yet alone specified for (”we just know we are going to need customer details”).  It’s putting the cart before the horse.  But IT go ahead and build the service anyway, because they own SOA.   At a later date the business articulate requirements for a new downstream application that requires Customer Details.  It’s the Corporate Business whose domain is Corporate Customers.  But what happens?  The service doesn’t quite meet their requirements.  The fields are wrong.  The Customer Details Service fits the domestic consumer model but not the corporate customers model.  What gives? More likely than not the downstream application.  The corporate customer has to be shoe-horned into the domestic customer service. I’ve seen this done.

Lesson 1. Don’t build SOA in a void.  Get out of that architectural ivory tower and engage with the business (if you can get them to listen - see next point). Better still engage in Guerrilla SOA.

Problem 2. The business haven’t got a clue.

One of the sad realities of the corporate world is that walls that have sprung up and created internal silos that are difficult to bridge.  As the business, the consumer of technology, I want IT to deliver to my requirements, no more, no less.  If I am in the domestic consumer part of the business, frankly I don’t care about Corporate customers.  I’m fighting for my budget, and hell, if this SOA thing is going to cost more than doing a closed application that fits only domestic customers, that only I can use I don’t care.  I’m not going to pay for a “Customer Details” service that does anything except give me what I need to know about my customer.

Lesson 2. The architects should facilitate the discussion.  SOA is as much about your business vision as it is technical architecture.  Unless the business grasps what you are trying to do, drives the solution and requirements are both local and global, before long you’ll see some grand services that few use in the core and chaos is the periphery where the real business is done.

Bottom line?  All too often architects fail because they tend to focus upon the architecture part of SOA rather than the services.    Unfortunately, because of the siloed nature of so may organisaitons, unless it is driven by the architects it is unlikely to gain traction.  If there is a maxim that should be followed when considering SOA in an organisation, it is probably instilling the notion of ‘think local, act global’.

It’s about the subtleties. You know – it’s about the little things. It’s about the stuff that’s not so obvious – the subtleties… the things that others hear in what you say whether you were aware of it or not…

Repetitors are usually articulate. They are able to express themselves. In the positive way, without the unneeded repetitions, a Repetitor would be an Articulate. By repeating themselves, without checking to see whether the listener is understanding, the Repetitor turns a Pattern into an Antipattern.

Dealing with a Repetitor is as simple as a variant on the Facilitation Four-Step: Interrupt, Ask, Redirect, Commit.

Interrupt

“Excuse me, Frank.”

Ask

“Do you mind…”

Redirect

“…if I check in with the others for a moment?”

Commit

“We’ll get right back to what you were saying.”

Action (yes, a 5th step )

“Sue, just so we’re clear, can you tell us what Frank’s point was?”

In this way, I validate that others have heard Frank, check to make sure that they’ve understood Frank, and break the pattern of repetition.

Tweet This Post 

Related posts

• Facilitation Antipattern: Dominator (0)
• The Facilitation Four-Step(tm) (3)
• Facilitation Pattern: Co-Worker (0)
• Facilitation Antipattern: The Qualifier (1)
• Facilitation Antipattern: Superhero (0)

Prioritization is crucial for a project to succeed. We need to make sure that we deliver what’s most important, what adds more value. The interesting and most challenging part of this is when we have to ask the business (client, product owner) to prioritize items. By calling them items I mean they are not necessarily stories, features or a product backlog. Sometimes we want to prioritize needs, purpose, goals and outcomes as well. The way we ask them is the trick…

I’ve seen many different ways of defining the priorities and usually there are ranges involved, sometimes with labels or numbers, like:

• Very High / High / Medium / Low
• Must Have / Should Have / Could Have
• Essential / Important / Not very important
• 1 - 5

There is nothing wrong with creating the labels, but when we ask the priority of one item only, that’s what happens sometimes:

How do we understand business priorities

The most effective prioritization exercise I’ve seen was to give the business a randomized list of items we want to be prioritized, then we ask them to give us the X most important items. And these will be the ones considered during the next timeframe. We should always let the list visible to the stakeholder who is prioritizing… It has to be clear for him that he is giving up on items in favor or others… at least for the next timeframe, which could be the next iteration of release.

It’s like a son’s birthday gift…

Good parents apply this rule with their children. Let’s say it’s close to my son’s birthday, I don’t have a son, this is totally hypothetical :), but my mother used this rule with me… So, I want to buy him a gift, ONLY ONE, of course, because I’m not the kind of dad that gives everything that my son wants. So I have a list of gifts that I know my son wants. Bike and video game are top of the list, but I can give him only one now, because it’s his birthday. We all know It’s close to Christmas (next iteration/release) and I will give him the other one on Christmas, but now it’s his birthday time, only one gift… I, then, make sure that my son understands the one gift per event rule and ask him which one does he want as his birthday gift… As a clever boy, he will think carefully which one really matters more to him because it’s clear that he has to give up on one in favor of the other. He will not give up on it forever though, it’s just until Christmas (next iteration/release), however, he knows that one of them will be bought and received first, and the other will come later…

As I said, there is no problem in defining labels and priority buckets, the only thing we have to bear in mind is that whenever we ask the business to prioritize items, we do not ask them in which bucket he wants to put one single item. And that’s when the principle comes.

Principle of Relative Priority:

One item by itself is usually important, otherwise it would not even be in the wish list… If I ask my son:

• Dad: “Do you want a bike?”
• Son: YES!
• Dad: “Do you want a video game?”
• Son: YES!

But when they have to CHOOSE… That’s when the most important items are picked… And during times of crisis, like now, with budgets constraints, we don’t even know if we will be able to afford a Christmas gift, so let’s make our kids happy during their birthday…

• PostgreSQL Database Server Configuration in Debian
• Query on decimal fields with long string representation of float causes exception

In Coding Tip #27 I explained how I rarely like to use booleans to represent states, and prefer to use an enum. Now that I have all nice little enums everywhere, another pattern that I see emerge is that I want more from my enum than just a value: perhaps other values associated with it, or actual behaviour. These enums then get converted into fully-fledged objects (ie classes) in the system with a very simple refactor.

So, the code started with:

private bool hasVehicle;

then, by following Coding Tip #27, went to:

private Vehicle theVehicle;
public enum Vehicle
{
    Car,
    MotorBike,
    PeopleMover
}

and now can turn into:

private Vehicle theVehicle;
public class Vehicle
{
    public static Vehicle Car = new Vehicle("Car");
    public static Vehicle MotorBike = new Vehicle("MotorBike");
    public static Vehicle PeopleMover = new Vehicle("PeopleMover");

    private Vehicle(string name)
    {
        this.name = name
    }
    private string name;
}

so my objects are all setup to add additional functionality:

private Vehicle theVehicle;
...
theVehicle.WillSeatAPartyOf(4);
...
public class Vehicle
{
    public static Vehicle Car = new Vehicle("Car", 5);
    public static Vehicle MotorBike = new Vehicle("MotorBike", 2);
    public static Vehicle PeopleMover = new Vehicle("PeopleMover", 8);

    private Vehicle(string name, int passengerLimit)
    {
        this.name = name
        this.passengerLimit = passengerLimit;
    }
    private string name;
    private int passengerLimit;

    public bool WillSeatAPartyOf(int numberOfPassengers)
    {
        return numberOfPassengers =< passengerLimit;
    }
}

I debate whether the class should contain the name or the index, usually I use the name; you can use both. Obviously if someone was now relying on the enum order, you would need to give it the index. Another advantage of this is because the constructor is private, you know that there are a definitive list of representations for it.

I don’t usually use class straight away. Indeed, I usually take bets to see how long it will stay an enum – sometimes the refactor is immediate, other times it remains an enum.

What's an iteration again?

So where do we start?

Weren't we talking about what goes into an iteration?

Sometimes the answer is - YAGNI

Related Posts

© Sumeet Moghe, 2009

We seem to have come full circle in our debate around the impact of culture on Enterprise 2.0. Couple of years back, Tom Davenport stirred up a hornet's nest with his "Why Enterprise 2.0 won't transform organizations" post. In that he said:

"The absence of participative technologies in the past is not the only reason that organizations and expertise are hierarchical. Enterprise 2.0 software and the Internet won't make organizational hierarchy and politics go away. They won't make the ideas of the front-line worker in corporations as influential as those of the CEO. Most of the barriers that prevent knowledge from flowing freely in organizations - power differentials, lack of trust, missing incentives, unsupportive cultures, and the general busyness of employees today - won't be addressed or substantially changed by technology alone. For a set of technologies to bring about such changes, they would have to be truly magical, and Enterprise 2.0 tools fall short of magic." [ Emphasis mine ]

More recently, Steve Radick wrote a meticulous post on how Enterprise 2.0 reflects the culture in an organization. He had a bunch of snippets from conversations in their Yammer network. He goes on to make a few recommendations : " Consider incentivizing employees to share information and collaborate with each other.  Make information sharing part of their annual review (my team reviews the employee’s contributions to our internal network during their annual assessment debrief).  Reward staff for taking risks."

And couple of days back, Gil Yehuda in his post on his experiences in the E2.0 conference in Boston says he is frustrated the "motherhood and apple-pie" lessons about E2.0 . He agrees that culture does play a key role and that the Enterprise 2.0 community needs to start working on this.

My triggering point for this post was a post by Peter Bergman in the Harvard Business blogs on the best way to change corporate culture. It is in many ways a recapitulation of fundamental issues organizations face on the cultural side. He says:  "Performance reviews and training programs define the firm's expectations. Financial reward systems reinforce them. Memos and communications highlight what's important. And senior leadership actions — promotions for people who toe the line and a dead end career for those who don't — emphasize the firm's priorities. In most organizations these elements develop unconsciously and organically to create a system that, while not always ideal, works."

What all of this really boils down is two things - human and social capital. Toyota in my view could be one such company - the robust and high performance knowledge sharing network they have built across their supply chain is a case in point. See research paper here .

Quoting from the paper:

"Toyota’s network has solved three fundamental dilemmas with regard to knowledge sharing by devising methods to (1) motivate members to participate and openly share valuable knowledge (while preventing undesirable spillovers to competitors), (2) prevent free riders, and (3) reduce the costs associated with finding and accessing different types of valuable knowledge. Toyota has done this by creating a strong network identity with rules for participation and entry into the network. Most importantly, production knowledge is viewed as the property of the network. Toyota’s highly interconnected, strong tie network has established a variety of institutionalized routines that facilitate multidirectional knowledge flows among suppliers."

Remember this was years before we even started speaking about Web 2.0. Inevitably, organizations that have invested in building human/social capital and a collaborative culture will stand to benefit the most from E2.0. Anecdote's whitepaper on "Building a Collaborative Workplace" neatly summarizes the essence of this :

"Of course technology plays an important role in effective collaboration. We are not anti-technology. Rather we want to help redress the balance and shift the emphasis from merely thinking about collaboration technology to thinking about collaboration skills, practices, technology and supporting culture. Technology makes things possible; people collaborating makes it happen."

And remember, this is hard work!! Building this work culture and ensuring that there is value alignment [ everyone believing in the lean philosophy in Toyota is an example] across multiple stakeholders are not easy. In Toyota's case, one of the challenges was to optimize the entire supply chain using lean principles and the supplier network in particular and the larger Toyota Production System in general played a key role in that. It seems to have taken a lot of hard work to build out these networks, evolve norms, align values and ensure that all of these rolled up to specific business objectives like the rapid diffusion of lean production techniques across the supply chain. Could Toyota have done this better with E2.0 technologies? Looks like it would have helped them accelerate this journey but then it would have been possible only because they had a strong cultural and business foundation.

The interesting inference I can make is this-At a more fundamental level organizations need to realize that Enterprise 2.0 is possibly a new lever to set the flywheel in motion and make it go faster - The principle of the flywheel is one of the analogies Jim Collins uses in his book "Good To Great". He speaks about a host of things that add up to set the flywheel in motion - Level 5 leadership, getting the right people on board, confronting brutal facts, the "Hedgehog" concept, a culture of discipline & technology as an accelerator.  Jim did not think of technology as a change agent but this could be changing - E2.0 and a Gen Y/X workforce could fundamentally challenge this notion IF organizations focus on the other levers as well.

Here is the bottom line as I see it : If yours is a great organization, E2.0 can help your flywheel go faster in the direction you want. If yours is a good organization, E2.0 can be one of the levers to help you set the flywheel in motion and eventually make it go faster. And this time around, technology holds a real good chance of being a change agent by enabling socio-technical ecosystems.

One of the most frustrating things for me lately about interacting with C# libraries from F# has been setting up objects through the use of properties.

While I am against the use of setters to construct objects in the first place, that's the way that a lot of libraries work so it's a bit of a necessary evil!

In C# we would typically make use of the object initializer syntax to do this, but in F# I've been writing code like this to do the same thing:

type MessageBuilder(?message:string, ?user:string) =              
	let buildMessage message user =
	   let twitterStatusBuilder = new TwitterStatus()
	      twitterStatusBuilder.Text <- message
	      twitterStatusBuilder.User <-
	      	let userBuilder = new TwitterUser()              
	         userBuilder.ScreenName <- user
	         userBuilder
	      twitterStatusBuilder
 
	member self.Build() = 
		buildMessage (if message.IsSome then message.Value else "") (if user.IsSome then user.Value else "")

This is more verbose than strictly necessary but I wanted to try and ensure all mutations to objects were being done within the function creating it rather than creating an object and then mutating it which feels strange to me in F# land.

I recently realised that it's actually possible to call properties in the same way that we can create objects using named parameters.

We therefore end up with the following code:

type MessageBuilder(?message:string, ?user:string) =              
    let buildMessage message user = new TwitterStatus(Text = message, User = new TwitterUser(ScreenName = user))
 
    member self.Build() = 
        buildMessage (if message.IsSome then message.Value else "") (if user.IsSome then user.Value else "")

Which is much more concise and does the same thing.

How many times have you found yourself feeling angry or hurt or amused, and yet been unable to put your finger on just what it was?

“He insulted me!”

“She hurt my feelings.”

“That was ridiculous.”

And the other person is confused, surprised, or hurt by your reaction. Why is that?

As attuned as I try to be to subtleties, I still find myself surprised at times.

One of the more common subtleties that I try to remain aware of is using comparative and judgmental words unintentionally, especially when facilitating meetings/discussions.

For instance, one person offers a comment, to which I say “thank you.” The next offers a comment, to which I respond “That was good.” Another offers a comment to which I say “Excellent!” What’s the impact on the first person? After all, I didn’t say their comment was good or excellent, so was their comment not as good as the other two? Did I somehow just slight that person? And the second person – did I imply that the third person’s comment was even better than theirs?

I can hear folks now saying “Are you telling me, Doc, that I have to think about every word I say before I say it? I mean, won’t that be a lot of work?”

Yes, and yes. Especially if you are in a position or role where your words have power and influence.

I haven’t forgotten my own philosophy – that It’s All About Me – that I’m not responsible for my listeners’ behavior or feelings.  But I also remember that part of my thinking is that I can choose to be aware of the impact that my behavior may have on others, and choose to modify my behavior.

When I facilitate, I struggle to be aware of how what I say may impact those present, and to choose my words with care. I try to avoid comparatives (”better”), and judgmental terms (”good”, “thoughtful”). As a facilitator, it’s appropriate for me to recognize someone for speaking (”thank you”) and acknowledge them, but not to judge them or compare them.

Tweet This Post 

Related posts

• The Facilitation Four-Step(tm) (3)
• Facilitation Pattern: Convergence (0)
• Facilitation Pattern: Co-Worker (0)
• Facilitation Antipattern: The Gladiator (1)
• Facilitation Antipattern: Dominator (0)

I’ve felt the need for this since I put out the CLR version of Ioke, and now I’ve finally managed to make it happen. Even though I’m the only person with commit rights to Ioke so far, it is still good to have continuous integration running, especially since there are at least seven different builds I want to test, 3 on linux and 4 on windows.

I now have two servers running this. They are not public right now - I will post something when the dashboard is up - but the CI server will send notification emails to the ioke-language Google Group with status.

The current setup tests Java 1.5, Java 1.6 on Linux and Windows. It tests Mono on Linux and Windows, and .NET on Windows.

As a CI server I’m using Cruise, ThoughtWorks own Continuous Integration server. Cruise is a commercial product, but open source projects can use it for free. I’m very happy about it from earlier projects, which is why I decided to use it for Ioke.

ThoughtWorks also gave me two virtual machines to run this CI server - which I’m very grateful for.

(Disclaimer: In the spirit of full disclosure, Stu is a friend, fellow NFJS speaker, and former co-worker of mine from DevelopMentor.)

I present this review to you in two parts.

Short version: If you want to learn Clojure, and you're familiar with at least one programming language, you'll find this a great resource. If you don't already know a programming language, or if you already know Clojure, or if you're looking for "best practices" to cut-and-paste, you're going to be disappointed.

Long version: Recently, fellow NFJS speaker Stu Halloway decided to take up a new language, and came to Clojure. He found the language interesting enough to write a book on it, something he hasn't done since his Java days, and the result is a nice walk through the language and its environment for experienced Java developers who want to understand Clojure's language, concurrency concepts, and programming model.

Now, let's be 100% honest about this: if you're coming at this book expecting it to be a language reference, you will probably be disappointed (as this guy obviously is). Stu's not like that—he's not going to re-create material that's available elsewhere, or that can be found with an easy Google search. Stu will not waste your time that way—he wants to tell you a story, one that takes you from "I'm a Java guy, but clueless about Lisp, dynamic languages, functional programming, concurrency, or macros" to "Wow. I know kung-fu." in the shortest path possible, but without trying to lobotomize you. He wants—no, expects—the readers of his book to be propping the text open with a cell phone on one side and the dinner plate on the other, craning your neck over to scan the pages and type in the examples into the REPL shell to try them out, see them work, then spend a few minutes experimenting with them before moving on to the next paragraph or page.

(Oh, I suppose you could just cut and paste them from the PDF version of the book, but where's the fun in that?)

The fact is, the concepts behind Clojure make up what's important to learn here, and readers of this book will come away like the panda from the movie, realizing that "There is no Secret Ingredient", that the power of Clojure comes not from its super-secret language sauce or special libraries, but in the way Clojure programmers approach problems and think about programming. And for that reason, if you're a programmer—even if you don't program on the JVM—you really want to take a look at what Stu's talking about (and Rich Hickey is creating).

Just remember, cellphone and dinner plate. Otherwise you'll be missing out on so much.

user> (defn a [{b :b c :c}] (- b c))



#'user/a



user> (a {:c 5 :b 11})



6

This post is late, but the slides from our RailsConf presentation are online:

Rails in the Large:How We’re Developing the Largest Rails Project in the World

I've been spending a bit of time reading through the Fake source code to try and understand how it works and one of the things which I quite like about it is the way the authors have made use of different F# operators to make expressions easier to read by reducing the number of brackets that need to be written and reordering the functions/values depending on the particular context.

One which I hadn't seen before is the application operator which is the opposite of the forward operator which I have previously written about.

The application operator (<|) applies a value to a function, the function being on the left and the value on the right.

It is used in FileHelper.fs as part of the DeleteFile function:

let DeleteFile x =   
  let file = new FileInfo(x)    
  if file.Exists then 
    log <| sprintf "Deleting %s" file.FullName
    file.Delete()
  else
    log <| sprintf "%s does not exist." file.FullName

The log function is of type 'string -> unit' and the sprintf call helps create that string. Without the application operator we would have to put in extra parentheses:

log (sprintf "Deleting %s" file.FullName)

The code also makes use of the forward operator which I think panders more to the object oriented style of programming whereby you have have some data/object and then apply a method/function to that. I find that code written in this way reads more intuitively to me at the moment.

One example of this is the SetDirReadOnly function in FileHelper.fs

  let rec SetDirReadOnly readOnly (dir:DirectoryInfo) =
    dir.GetDirectories() |> Seq.iter (fun dir ->
      SetDirReadOnly readOnly dir
      setDirectoryReadOnly readOnly dir)
    dir.GetFiles() |> Seq.iter (fun file -> file.IsReadOnly <- readOnly)

In this case if we didn't have the forward operator then in theory we should be able to just put the 'dir.GetFiles()' can be passed as the second argument to 'Seq.iter':

  let rec SetDirReadOnly readOnly (dir:DirectoryInfo) =
    dir.GetDirectories() |> Seq.iter (fun dir ->
      SetDirReadOnly readOnly dir
      setDirectoryReadOnly readOnly dir)
    Seq.iter (fun file -> file.IsReadOnly <- readOnly)  dir.GetFiles()

In fact what we get is a compilation error:

Successive arguments should be separated by spaces or tupled, and arguments involving function or method applications should be parenthesized.

In this case we need to paranthesise the 'dir.GetFiles()' method call:

  let rec SetDirReadOnly readOnly (dir:DirectoryInfo) =
    dir.GetDirectories() |> Seq.iter (fun dir ->
      SetDirReadOnly readOnly dir
      setDirectoryReadOnly readOnly dir)
    Seq.iter (fun file -> file.IsReadOnly <- readOnly)  (dir.GetFiles())

Which leads to another compilation error:

Lookup on object of indeterminate type based on information prior to this program point. A type annotation may be needed prior to this program point to constrain the type of the object. This may allow the lookup to be resolved

In this case what we're being told is that the compiler is unable to work out the type of 'file' in the function being passed to 'Seq.iter'. We can fix this by specifically stating its type:

  let rec SetDirReadOnly readOnly (dir:DirectoryInfo) =
    dir.GetDirectories() |> Seq.iter (fun dir ->
      SetDirReadOnly readOnly dir
      setDirectoryReadOnly readOnly dir)
    Seq.iter (fun (file:FileInfo) -> file.IsReadOnly <- readOnly)  (dir.GetFiles())

It works but it seems to miss the point of getting the F# compiler to infer which types you're talking about - the forward operator simplifies the code a lot. I also think the code is more readable having 'files' at the beginning as it seems more obvious that the function is being applied to the sequence of files when written this way.

These operators are pretty cool and I've found it quite useful to look at the full list of the F# operators available on the Microsoft Research website as there may well be even more built in functions that can help simplify our code further.

I got the Douglas WF-165 TA for fathers day. Looks pretty good. The neck feels really great. Bending seems significantly easier than on my Fender Nashville Tele (I’ve been lazy and really need to spend some loving time with that one). I wasn’t expecting the pitched back headstock (must not have look close enough to the webpage). The setup was a little out of whack, but thats fixed now. Just switched out the strap buttons with Dunlop strap locks. Here it is on my workbench:

The pickups are Ok. I’ll hold off on changing them out until after the the rest of the work is done. I’ll see if I will ever use them, or just play exclusively through the VG-99.

This week's dojo involved coding a familiar problem - the bowling game - in a different language, Groovy.

The code we wrote is available on bitbucket.

The Format

Cam, Dean and I took turns pairing with each other with the code projected onto a TV. As there were only a few of us the discussion on where we were taking the code tended to included everyone rather than just the two at the keyboard.

What We Learnt

• I've sometimes wondered about the wisdom of running dojos in newer languages but this one worked quite well because Cam has been learning Groovy and he was able to point us in the right direction when we started writing Java-esque Groovy code. The particular syntax that I didn't know about was that you can define and put items into a list in a much simpler way than in Java:

  I was starting to write code like this:

  def frames = new List<Frame>()
  frames.Add(frame)

  Which Cam simplified down to:

  def frames = []
  frames << frame

  I didn't feel that I missed the static typing you get in Java although IntelliJ wasn't quite as useful when it came to suggesting which methods you could call on a particular object.

• I'm even more convinced that using various languages functional equivalent of the 'for each' loop, in this case 'eachWith' and 'eachWithIndex' is not the way to go and we could see our code becoming very complicated when trying to work out how to score strikes thanks to our use of it!
• I think we actually got further this time in terms of the implementation although we did slow down when it came to scoring strikes to try and work out exactly how we wanted to do it.

  Prior to this we had been following the idea of just getting the tests to pass and driving the design of the code that way but we at this stage it seemed foolish to keep doing that as the code would increased dramatically in complexity by doing so.

  The two approaches we were thinking of involved using the state pattern to determine what the current frame outcome was and then work out the cumulative score based on that by looking forward to future frames or an approach that would make use of functional collection parameters (not sure exactly which ones!) to calculate the score in a more function rather than OO way.

For next time

• We'll probably keep going with some more Groovy as it's actually more interesting than I thought it would be. I'm also keen to do a coding dojo where we never make use of the if statement.