Reading time: 3 – 4 minutes

Web apps have numerous choices for storing stateful data in between requests. For example, when a user goes through a multi-step wizard, he or she might make choices on page one that need to be propagated to page three. That data could be stored in http session. (It also could get stored into some backend persistence and skip session altogether).

So where does session get stored? There are basically four choices here.

1. Store state on one app server, i.e. for java in HttpSession. Subsequent requests need to be pinned to that particular app server via your load balancer. If you hit another app server, you will not have that data in session.
2. Store state in session, and then replicate that session to all or many other app servers. This means you don’t need a load balancer to anchor a user to one app server; multiple requests can either hit any app server (full replication). Or use the load balancer to pin to particular cluster (themselves replicating sessions, and giving higher availability). Replication can be smart so only the deltas of binary data are multicast to the other servers.
3. Store the state on the client side, via cookies, hidden form fields, query strings, or client side storage in flash or html 5. Rails has an option to store it in cookies automatically. Consider encrypting the session data. However, some of these options can involve a lot of data going back and forth on every request, especially if it’s in a cookie and images/scripts are served from the same domain.
4. Store no state on app servers, instead write everything in between requests down to backend persistence. Do not necessarily use the concept of http session. Use id’s to look up those entities. Persistence could be a relational database, distributed/replicated key/value storage, etc. Your session data is serialized in one big object graph, or as multiple specific entries.

What you choose is up to many factors, however a few guidelines help:

1. Try to keep what is in session small.
2. If possible, keep session state on the client side.
3. Prefer key/value storage replicated among a small cluster over replicating all session state among all app servers.
4. Genuinely consider sticky sessions, which home users to a particular app server. Many benefits abound, including what Paul Hammant talks about w.r.t Servlet spec 7.7.2 Appengine’s Blind Spot.
5. If you serialize an object graph, recognize that when you do a deployment it will probably mean existing sessions are now unable to be deserialized by the newly deployed app. Avoid this by using your load balancer to swing new traffic into the new deployments, monitor errors, and then let the old sessions expire before switching all users over to the new deployment. Bleed them over.
6. Session is not for caching. It may be tempting to store data in session for caching purposes, but soon you will need a cache.
7. Store in session what is absolutely necessary for that user, but not more. See caches, above.

In the quest to get agile and UX to get along better, and following successful retreats in the US, last weekend Johanna Kollmann brought together a bunch of agile and UX folk for an Agile UX retreat in London sponsored by.  Giving up a weekend was hard, but was worth it, meeting a great bunch of people and sharing thoughts and experiences from the agile and UX camps.  So what did I learn.

Rethink what we do

Coming out of the retreat it is clear that the way we do UX today needs a fundamental rethink.  As UX professionals we have fought long and hard to gain credibility and traction in organisations for what we do, but we need to be ready to evolve and embrace the changing world around us.  A world where IT no longer needs to have detailed specifications signed off before development start.  We no longer have the need (or the luxury) to do the up-front research that we are used to doing.  We no longer need to sign-off detailed wireframes before handing them over the fence to the developers to implement.  Software today really is soft.  It is more about creativity than engineering (see below).  The serialisation of activities is inefficient and wasteful.  It is time to ask how do we focus upon doing what is needed and when, working in parallel and infecting the whole system with user-centric thinking rather than siloing it into the upfront design.  This after all is what systems ergonomics is about; a forerunner to UCD that we know today, thinking about the macro (a broad system view of design, examining organizational environments, culture, history, and work goals) as well as the micro (fitting the task to the human).

But I am digressing from what I wanted to blog about, the Agile UX retreat.  Some key takeaways for me included Anders Ramsey’s analogies to the restaurant and the theatre.

Thinking analogies

Think of a restaurant.  We have the kitchen, the back room world that is focussed upon delivering consistency of servings.  Everything in the kitchen is utilitarian, serving the purpose to meet this goal.  At the front of the restaurant we have the dining room where the dishes (of consistent(ly good) quality) made in the kitchen are served.  The dining room is all about the ambiance.  Quality here is far more subjective, but a successful restauranteur will be as passionate about the dining room as she is about the food that is cooked in the kitchen.  This is the way that software is all too often built, with the kitchen and dining room being separate entities, however the way they are organised, paid for and owned, there’s little communication between the two.  To quote another Ramsey, Gordon, it is a Kitchen Nightmare.

Anders’ second analogy to consider was the Theatre.  An overly simplistic representation is that the director starts with a script.  From the script he iterates the production.  The producer’s role is to provide the director with what he needs to make the production successful.  Just be ready for the premiere which is on a fixed date.  In the lead up to the premiere the director assembles the cast, the crew and they rehearse.  They’ve got a strawman plan to work from – MacBeth, but how they implement it will evolve according to the stage, actors and artistic direction the director wants to take.  The producer does not care how or when they rehearse, she is only concerned with the success of the end goal.  As they rehearse they increase the fidelity of their performance until they premiere (go live).  but even then they are not done.  They are happy to accommodate changes to the performance, and if something different happens that clearly delights the audience they will happliy incorporate that into future performances (releases).  Sure, the audience is seeing a performance of Shakespere’s MacBeth, but it is a unique performance that has taken the initial plan and evolved as it has been created.

And so should we approach software development.  Not as an exercise in engineering, where our raw materials are fixed and highly stable, but as a creative artform, where our iterations are rehearsals for the premier and ongoing performances.

Thinking tensions

I’m sure there are more, but some examples of tensions that emerge when we try to work together:

AUX promotes rapid open communication and sharing but designers fear sharing.  (They worry early designs will be seized upon before they are ready)

AUX promotes visualisation and use of walls but corporate policies prevent this

AUX promptes doing just enough, just in time but a legacy of deliverable expectations gets in the way (research is rarely bought by the developers who will ultimately consume it).

Thinking people

At the end of the day, success comes down to people.  Agile zealots have done Agile no favours when they bang on about business value and see anything other than code as waste.  Good product design needs vision, it needs research to ensure you are building the right thing for the right people.  No one has the right to tell a UXer that testing ideas or building a prototype or undertaking research is waste if it is right for their context.  But it doesn’t need to take the time it does today.  The UX community needs to get out and spend time with the development community and understand how software is built today.  UXers need to start seeing developers as partners rather than consumers of what they do.  What if we aligned our teams around the products we build rather than the functional silos that the roles describe?  Bringing agile and UX together is more fundamental than arguing about the process (one iteration in front, washing machine cycles etc), it is about fundamentally changing the way we build software; see it as a team activity that works collaboratively rather than a factory production line with process gates and separation of responsibility.

More on the #auxretreat twitter feed

• Stevey's Blog Rants: Wikileaks To Leak 5000 Open Source Java Projects With All That Private/Final Bullshit Removed
• Other Android Languages
  
  android python

As I’ve covered in several blog posts, I visited the Emerging Languages camp last week. It was an interesting experience for many reasons, and some of my conclusions are still half formed. I wanted to talk a little bit about some common themes in several languages presented at this camp, and also what the trends are leaning towards. Now, I’m not entirely sure what my insights will be yet, but hopefully I’ll know as I approach the end of this blog post.

There are several different axises you can divide the presented - about 26 - languages. The first one is in terms of age and maturity. The oldest language presented was probably Parrot, Frink, Io, Factor and D - who have all been around for eight to ten years. All of these languages are very mature and you wouldn’t hesitate to use them for real life work. The second category are the languages that range from a few years in age to quite new ones. Most of these languages are still evolving, still not stable, but definitely on the path to getting there. The final set of languages are the most interesting in my view - the new ideas that have just started germinating, or even concepts that aren’t actually there yet. From my list of interesting things, Wheeler is definitely a language in that category.

But you can also look at the types and features you find in the new languages (and I will exclude the oldest group of languages when talking about these features and ideas). There is a large group of languages that are evolutionary rather than revolutionary. This is fine, of course. Many of these languages take much inspiration from Ruby and Smalltalk. Object orientation is extremely common among these languages, several of them prototype based. There was also several languages with indentation based syntax. I was surprised about the amount of languages that target native instead of a virtual machine. AmbientTalk, Clojure, Ioke/Seph, Frink and Mirah targets the JVM, Stratified JavaScript, E/Caja and CoffeeScript targets JavaScript and F# targets the CLR. All the other languages can be considered native. I was quite surprised about this - anyone got a good explanation?

There are also several graphical languages presented. These are harder to categorize from a traditional paradigm perspective. Thyrd is more of a proof of concept, very graphical, but backed by a stack language in the style of Forth. Kodu seems to have a quite traditional backend, but the graphical interface hides that in most cases. Both of these languages are optimized to run in situations where you don’t always have a keyboard - Thyrd for tablet PCs and Kodu for the XBox. Subtext/Coherence is based around non-syntactic thinking, but didn’t seem to have a graphical interface either at this point.

As mentioned above, the trend of using JavaScript as a target language also seems to be on the way up - both CoffeeScript, Caja, and Stratified JavaScript follows that approach. Parts of Ur/Web are also compiled to JavaScript to allow the based language to describe behavior in both the server and the client. Gilad reported that he was very interested in getting Newspeak to run on JavaScript too, and there has been a lot of talk about getting Io to work on that platform too. This seems to be an interesting idea for many languages, and the benefits of compiling to a language that can run in any browser is definitely compelling. However, there seems to be a lot of problems with that approach too. Some people create a bytecode machine in JavaScript that then executes generated bytecodes. Some languages have to do lifting of functions because of bugs in several JavaScript implementations. And of course, the JavaScript language doesn’t give you good ways of generating code that is easy to debug.

The low level languages all seem to give interesting capabilities. D is what C++ should have been. BitC allow you to write programs with strong guarantees. ooc gives many of the high level benefits to a low level language. Go makes it possible to handle concurrency in an easy and powerful way.

I’m at the end of my thoughts right now, and there is no grand conclusion to be found. Just some interesting observations. Maybe they are indicative of the future in language development, and maybe not.

When I first studied how Toyota engineers implemented TPS at a supplier, they always knew what the next step was, and so I assumed (as did the supplier’s engineers) that had a roadmap of how they wanted to make the cell progress. I kept badgering them to show me this map, thinking that they didn’t want to share this for proprietary reasons. The Toyota guys always refused, explaining that they didn’t have a roadmap. All they were doing was focusing on problems as they appeared and then working hard at solving them. One day, exasperated with my annoying questions, the lead engineer said to me” “WE DON’T HAVE A ROADMAP. But—we do have kind of a golden rule: making people before making parts.” I’ve tried to puzzle this out ever since.

“I don’t want people who will do the job right. I can find those any time. I want people determined that the next time they’ll do the same job, they’ll do it better. Those are the people I need."

I’ve just come back from three days in Santa Clara, spending time with some of the brightest people in the Java world - the JVM language summit is truly a fantastic collection of great people. And I was there too…

THe goal of the JVM language summit is to collect the people who work with languages on the JVM and have them share their projects, their experiences and their networks - and let them network with the people in charge of implementing the JVM’s for different companies. This year, a lot of discussion about JSR 292 and project lambda was on the plate. The presence of hardware and VM people was also more pronounced. I counted principals for at least six different virtual machines in the audience or presenting (Hotspot, JRockit, J9, Azul, Maxine, and Monty).

Among the experienced platform and language people there, some of the notables included Kresten Krab Thorup, Joshua Bloch, Bob Lee, Neal Gafter, John Rose, Brian Goetz, Alex Buckley, Rich Hickey, Charles Nutter, Cliff Click, Doug Lea, Per Bothner and many more. A great collection of people.

As an example of the funny happenstance that can happen in this collection of people, I was sitting rebinding my Java implementations for Mac OS X - and I had remove lots of links in /usr/bin. A few minutes later the person next to me started asking some questions about my experience with Java on the Mac - and it turns out he’s the manager for the Apple JVM team. Or at one point Rich Hickey reported on a quite puzzling problem that causes bad semantics when iterating over data that doesn’t fit in memory - and Cliff Click immediately opens up his laptop, says “give me an half hour and I’ll see what I can do”.

Another funny anecdote was when Doug Lea pointed out that if you use fibonacci to test performance against yourself or others, it’s important that the implementations actually agrees about the first values of fib. Funnily enough, I saw three different implementations of the ground rule in fib during the summit - all of them different. (if n < 2 return 1, if n<=2 return n, if n < 2 return n).

There were way too many interesting presentations and discussions for me to be able to talk about all of them - instead I just wanted to give some highlights.

Charles Nutter

Charles gave a quick introduction to JRuby and Mirah, and what kind of optimizations JRuby is currently doing. He also talked about how far he’s gotten in inlining invoke dynamic calls inside of JRuby (and he’s gotten very far - it’s really cool).

Fredrik Öhrström

Fredrik is the JRockit representative on JSR 292, and way too smart. He presented a solution to how you can use method handles integrated with function types to solve many of the current problems in project lambda. A very powerful and interesting presentation.

Doug Lea

Doug spent his keynote trying (quite successfully) to concinve the room of the hegemony of fork-join as a good solution to concurrency problems. A very good and thought provoking keynote.

Josh Bloch

Last year at the JVM language summit, Josh talked about what he called “the Semantic Gap”. This year, after being beat up by some linguists, he changed the name of this concept to “Performance Anxiety”. The basic idea is that in our current infrastructure we have traded performance for predictability. Two examples from his talk about when this happens in Java was pretty interesting. He had one benchmark that consistently showed about the same numbers for the same JVM run, but differed between JVM runs. There was no undeterminism in the benchmark itself, but they benchmark times continued to oscillate between 0.7 and 0.85 depending on JVM run. Cliff Clicks explanation for this is that it is probably the compilation planner, which is a separate thread. Depending on when that thread runs the compilation strategy will be different, and makes a difference in times. And it’s really hard for the programmer to take this difference into account.

The other example is simpler (and don’y change your code because of this). In some circumstances it turns out that & is faster than && in Java, because a && will short curcuit, which means it will branch. The single ampersand will always execute both sides, which means the CPU can pipeline both of them to execute at the same time.

All the examples he shown comes down to the same thing - we can’t really reason intuitively about the performance of our language constructs anymore. Our systems have become to complex in order to support better performance, and we give up predictability to get that performance. And at the end of the day it doesn’t even matter if you go down to C or assembler - you still cannot control exactly what the CPU is doing anymore.

Kresten Krab Thorup

Kresten is the CTO of Trifork, and one of the main organizers of many of my favorite conferences (like JAOO and QCon). The last nine months he has worked on an Erlang implementation for Java, which he talked about. It seems to be a very good implementation, and he’s getting surprisingly good performance and context switching numbers. In fact, several of the ideas in Seph will be stolen from Erjang.

Rémi Forax

Rémi showed off his PHP.reboot project, implemented using JSR 292 and getting quite good performance. His JSR 292 backport seems to be really useful and I think I’ll use that to make sure Seph can run on pre Java 7 machines. Good stuff.

Rich Hickey

Rich spent some time collecting comments from people in the room of what was problematic with the JVM in its current incarnation. To start us off, he showed one piece of hilarious/horrible Clojure code. Any one wants to guess what it does?

static public Object ret1(Object ret, Object nil) {
    return ret;
}

public static int count(Object o){
    if(o instanceof Counted)
        return ((Counted) o).count();
    return countFrom(Util.ret1(o, o = null));
}

We then went on to a few other things (which you can find on the JVM Language Summit wiki). The consensus seemed to be that tail calls is really very important. Last year, it wasn’t as crucial but now that we see how powerful method handles and lambda will be, tail calls turn out to be very nice to have. Hopefully we can make that happen.

JSR 292

The JSR 292 expert group got lots of chances to work on ideas and designs for the future. Lots of interesting results came out of these discussions. Some of the more notable ones are skisses on how method handles and function types can work together, how invoke dynamic and bootstrap method can be used to implement defender methods and several other interesting ideas.

All in all it has been a fun few days, going far out in language and implementation geekiness. I hope to come back to this next year.

Software Product is a bunch of functionality assembled together and tied within the boundary of business rules. Each line of code which is written to implement a functionality has a potential bug in them. And each valid bug raised requires a change in the code written.

Are these forming a vicious circle ? Let me explain.

In a typical agile methodology, we write the Unit Tests first and then the actual functionality. So that means we actually know how the system would be tested (to some extent). But after writing the functionality, the developer calls a fellow tester to do a Dev Box Testing. Dev Box Testing is generally done on a developer’s machine with a very minimal set of test data (Thanks to developers laziness ! :)). It’s done to ensure that the intended functionality is properly developed and there are no major bugs. It also gives developer the chance to debug the application in case of any exception (if encountered). So till now we have done Unit Testing and Dev Box Testing. What next? Post the developer check in, the application is generally deployed in a Test Environment for a further round of testing. This testing is intensive and is iterated over a large set of sampled data. This kind of full fledged testing often results in uncovering of many bugs.

So we are still left with bugs even after Unit Testing –> Dev Box Testing. So if we see the whole process process,

We write code –> We induce bug –> We find bug -> We write code to fix bug –> We induce bug –> …

I have something to confess:  I’m an information addicted. I’m always reading books, checking out news, following a few hundred blogs, following another few hundred people on twitter and the list just goes on and on. I bet you do the same.

First of all, this is not about how to just pass over your next university exams but how to keep yourself updated or learn something with all the information we are throw everyday. It’s not possible (for me at least) to keep track from everything I read. People like me reads hundreds of information sources, mostly online. Besides that we also read books, attend conferences, meet new people, etc. We are literally bloated by information from everywhere.

how can we manage to digest this huge amount of information we face everyday ?

We all know there’s no unique answer for that question. Everybody has a different approach when it comes to learn something. Some people prefer to take notes while reading, some people prefer to record audio snippets, some people like to create songs that resemble the information, (your method here). As you might think, I happen to have my own technique (probably not only mine) and it’s being very useful to me.

I read somewhere else a good analogy: we have two “databases” in our brain. One is the short-term memory database and the other is the long-term one.  When we read something, every piece of information goes to our short-term memory which is something like a “heap” of unclassified information . As this “database” isn’t classified, our brain doesn’t know if the information is useful or not.  Later on, it can be even 10 minutes, when we try to recall some information from it, we may not be able to easily do it because isn’t classified yet. But no worries : how many times happened to you that you remembered something just a few hours later while doing something else? That happens because our searching process is asynchronous.

Ok, our information searching process is asynchronous, so what ? It literally means, your brain is still learning some subject even after you stopped thinking about it. It’s like dispatching a background process. The problem is you need to constantly remember that subject in order to transfer it to your long-term memory.

You can achieve that by keeping a sort of learning log. I’ve tried many tools to help me keeping track of what I want to learn. For instance, a simple text editor, an iphone app. You just need to ensure you have quick and easy access to it….As I’m a very moody person I keep changing, right now, for instance, I’m using a folder of my browser’s favorites bar which I sync across my computers. Sometimes I also use the favorites features from twitter. As I’m always checking them, I keep remembering the subjects I have demonstrated interest. Most of this links are shortcuts to blog posts, interviews, podcasts, which I may have read/listened only once. But just the fact I’m recalling the title, it helps me to internally recall the content I read before from that source. In case I don’t remember what’s the content is about I just go there and skim over the text so I get the general idea. If even doing so, I don’t understand the content, then I mark it to read it later using a javascript snippet.

What happens when my learning log stack is getting excessively big ? Just use your common sense! If you get to the point where you always need to recall every item in your list, something is wrong! So, how many items should you have in your list? It may vary from person to person. In my case I found 10 to be a good size. It doesn’t mean I need to first learn all 10 items and them move to the next 10 items. Some items can be in your list for a relatively long time, it’s strongly dependent on how complex is the subject and how fast you can learn.

Defining when you “learned” something is very subjective. In my opinion, you can’t know everything. My “learned” definition is when I get to the point where I can explain the subject to someone else and they properly understand it. Having a time constraint can help too, like imposing yourself a 15 days limit for the items in your list.

Resources used:

• Learning log (official way) http://www.csudh.edu/titlev/learninglog.htm
• Read it later http://readitlaterlist.com/

Probably Related posts:

1. RELEASED – “Rails 2.1 – What`s new ?” Book
2. Why I have been away
3. Google Summer of Code

One of the steady themes I've seen throughout my career is that of the nature and importance of software development. Recently a prospect told one of our salespeople that "software is like sewage pipes, I want it to work reliably and I don't want to know about the details". This is the kind of approach that Nicholas Carr talked about in IT Doesn't Matter. On a contrasting note we've done work for many businesses where IT has been a clearer strategic enabler to their business, allowing them to enter new markets or significantly increase their market share. So is IT a utility, like sewage pipes, or a strategic asset?

I take that the view is that it can be either, depending on the system. A classic example of a utility IT project is payroll, everyone needs it, but it's something that most people want to 'just work'.

So what is the distinguishing factor between utility and strategic projects? To my mind it's all about whether the underlying business function is a differentiator or not. If how you do this function is a crucial part of what makes you better than the competition, then the software that supports this function needs to be as good as you can make it. Note that this distinction isn't about the software, but the business function. As Ross Pettit puts it "This is not a separation of IT by the nature of the technology, but into what technology does for the host business".

The most important point about this dichotomy is to realize that there are two kinds of software projects and they need to be treated entirely differently. The way you staff, run, and budget a strategic project is entirely different to how you do a utility project. Too often people assume that what is good for one is good for the other - and trouble inevitably follows.

Another consequence is that only a few projects are strategic. The 80/20 rule applies, except it may be more like 95/5. While certainly it's most common for people to not recognize the dichotomy at all, it's also common for people to think that too many projects are strategic.

One of the most important ways in which these efforts differ is where the risks lie. For utility projects the biggest risk is some kind of catastrophic error - you don't want the sewage pipe to break, or to miss payroll. So you need enough attention to make sure that doesn't happen, but other than that you want costs to be as low as possible. However with strategic projects, the biggest risk is not doing something before your competitors do. So you need to be able to react quickly. Cost is much less of an issue because the opportunity cost of not doing something is far greater than costs of software development itself.

This is not a static dichotomy. Business activities that are strategic can become a utility as time passes. Less often, a utility can become strategic if a company figures out how to make that activity a differentiator. (Apple did something like this with the design of personal computers.)

One way this dichotomy helps is in deciding between building custom software and installing a package. Since the definition of utility is that there's no differentiator, the obvious thing is to go with the package. For a strategic function you don't want the same software as your competitors because that would cripple your ability to differentiate.

Often people realize this and buy a package for a utility project, but then spend huge amounts of money customizing this - which is just as wasteful. My view is that for a utility function you buy the package and adjust your business process to match the software. Usually this is politically infeasible, so the workaround is to put a low grade software team to work on it. Provide enough care to avoid catastrophe, but otherwise you don't need a high-grade team.

Another way the dichotomy makes its influence felt is the role of agile methods. Most agilists tend to come from a strategic mindset, and the flexibility and rapid time-to-market that characterizes agile is crucial for strategic projects. For utility projects, however, the advantages of agile don't matter that much. I'm not sure whether using an agile approach for a utility project would be the wrong choice, but I am sure that it doesn't matter that much.

Like many classifications, there's a lot of grey in between. Yet this is one of those rare cases where I think there's a strong argument to turn up the contrast and force more binary thinking. As Ross commented in a discussion of a draft of this post: "'shades of grey' give license to pile things into the wrong category; things that are really utility will be given an inflated importance, rather than dispositioned as the utilities they really are." Forcing a binary decision, tilted to minimize what's in the strategic bucket, would help provide the focus that's often lacking in IT initiatives.

Ross goes so far as to argue that there shouldn't be a single IT department that's responsible for both utility and strategic work. The mindset and management attitudes that are needed for the two are just too different. It's like expecting the same people who design warehouses to design an arts museum.

To explore more...

• Ross's article calls for a Glass-Stegall of IT departments.
• Marc McNeil talks of projects as Tractors, nuclear power plants and the bleeding edge
• Neal Ford points out that integration efforts, such as SOA, can't be strategic.
• There is video of the talk I gave on this theme for AMP last year.

I’m currently in the process of implementing Seph, and I’ve reached an inflection point. This point is the last responsible moment to choose what I will target with my language. Seph will definitely be a JVM language, but after that there is a range of options - some quite unlikely, some more likely. The valid choices are:

• Target Java 1.4
• Target Java 5/6
• Target Java 7
• Target Java 7 with extensions

Of these, the first options isn’t really interesting for Seph, so I’ll strike it out right now. The other three choices are however still definitely possible - and good choices. I thought I might talk a little bit about why I would choose each one of them. I haven’t made a final decision yet, so that will have to be the caveat for this post.

Before talking about the different choices, I wanted to mention a few things about Seph that matters to this decision. The first one is that I want Seph to be useful in the real world. That means it should be reasonably fast, and runnable for people without too much friction. I want the implementation to be small and clean, and hopefully as DRY as possible - if I end up with both and interpreter and just-in-time compiler, I want to be able to share as much of these implementations as possible.

Java 5/6

The easiest way to go forward would be to only use Java 5 or 6. This would mean no extranice features, but it would also mean the barrier to entry would be very low. It would mean development on Seph would be much easier and wouldd in general make everything simpler for everyone. The problem with it would mainly be implementation complexity and speed, which would both suffer compared to any of the Java 7 variants.

Java 7

There are many good reasons to go with Java 7, but there are also some horrible consequences of doing this. For Seph, the things that would make things from Java 7 is method handles, invoke dynamic and defender methods. Other things would be nice, but the three previous ones are the killer features for Seph. Method handles make it possible to write much more succinct code, not generate lots of extra classes for each built in method, and many other things. It also becomes possible to refer to compiled code using method handles, so the connection between the JIT and the interpreter would be much nicer to represent.

Invoke dynamic is quite obvious - it would allow me to do much nicer compilation to bytecode, and much faster. However, I could still build the same thing myself, to much greater cost and it would also mean inlining wouldn’t be as easy to get.

Finally, defender methods is a feature of the new lambda proposal that allow you to add new methods to interfaces without breaking backwards compatibility. The way this works is that when you add a new method to an interface, you can specify a static method that should be called when that interface method is invoked and there are no other implementations on the concrete classes for a specific object. But the interesting side effect of this feature is that you can also use it to specify default implementations for the core language methods without depending on a shared base class. This will make the implementation much smaller and more flexible, and might also be useful to specify required and optional methods in an API.

The main problem with Java 7 is that it doesn’t exist yet, and the time schedule is uncertain. It is not entirely certain exactly what the design of the things will look like either - so it’s definitely a moving target. Finally, it will make it very hard for people to help out on the project, and also it won’t make Seph a possible language for people to use until they upgrade to Java 7.

Java 7 with extensions

It turns out that the interesting features coming in Java 7 is just the tip of the iceberg. There are many other proposed features, with partial implementations in the DaVinci project (MLVM). These features aren’t actually complete, but one way of forcing them to become more complete is to actually use them for something real and give lots of feedback on the feature. Some of the more interesting features:

Interface injection

This feature will allow you to say after the fact that a specific class implements an interface, and also specify implementations for the methods on that interface. This is very powerful and would be extremely helpful in certain parts of the language implementation - especially when doing integration with Java. The patch is currently not very complete, though.

Tail calls

Allowing the JVM to perform proper tail calls would make it much easier to implement many recursive functional algorithms easily. Since Seph will have proper tail calls in the language, this will mean that I will have to implement this myself if the JVM doesn’t do it, which means Seph will be slower based on this. The patch seems to be quite good and possible to merge and harden to the JDK at some point. Of all the things on this list, this seems to be one of things that we can actually envision see being added in the Java 7 or Java 8 time frame.

Coroutines/continuations

Both coroutines and continuations seem to be possible to do in a good way, at least partially. Coroutines might be interesting for Seph as an alternative to Kilim, but right now it seems to be a bit unstable. Continuations would allow me to expose continuations as a first class citizen which is never bad - but it wouldn’t give me much more than that.

Hotswapping

Hotswapping of code would make it possible to do agressive JITting and then backing out from that when guards fail and so on. This is less interesting when we have invoke dynamic, but will give some more flexibility in terms of code generation.

Fixnums, tuples, value types

We all want ways of making numbers faster - but these features might also make it possible to efficiently represent simple composite data structures, and also things like multiple return values. These are fairly simple features, but have no real patch right now (I think).

Light weight code loading (anonymous classes)

It is horrible to load byte code at runtime in Java at this point. The reason is that to be able to make sure your loaded code gets garbage collected, you will have to load each chunk of code in a new class in a new classloader. This becomes very expensive very fast, and also endangers permgen. Anonymous classes make this go away, since they don’t have names. This means you don’t actually have to keep a reference to older classes, since there is no way to get to them again if you lost the reference to them. This is a good thing, and makes it possible to not generate class loaders every time you load new code. THe state of this seems to be quite stable, but at this point JVM dependent.

The price

Of course, all of these lovely features comes with a price. Two prices in fact. The first price is that all the above features are incomplete, ranging from working patches to proof of concepts or sketches of ideas. That means that the ground will change under any language using it - which introduces hard version dependencies and complicates building. The other price is that none of these features are part of anything that has been released, and there are no guarantees that it will ever be merged in Java at any point. So the only viable way of distributing Seph would be to distribute standard build files with a patched OpenJDK so that anyone can download and use that specific JDK. But that limits interoperability and causes lots of other problems.

Somewhere in between

My current thinking is that all of the above choices are bad. For Seph I want something inbetween, and my current best approach looks like this. You will need a new build of MLVM with invoke dynamic and method handles to develop and compile Seph. I will utilize invoke dynamic and method handles in the implementation, and allow people to use Rémi Forax’ JSR 292 backport to run it on Java 5 and 6. When Java 7 finally arrives, Seph will be more or less ready for it - and Seph can get some of the performance and maintainability benefits of using JSR 292 immediately. At this point I can’t actually use defender methods, but if anyone is clever enough to figure out a backport that will allow defender methods to work on Java 5 or 6, I would definitely use them all over the place.

This doesn’t actually preclude the possibility of creating alternative research versions of Seph that uses some of the other MLVM patches. Charles Nutter have shown how much you can do by using flags to add features that are turned off by default. So Seph could definitely grow the above features, but currently I won’t make the core of the language depend on them.

• nottingham-http-cache-channels-01.txt
  (tags: cache squid http ietf)
• mnot’s Weblog: Cache Channels
  (tags: cache squid http)

Following the recommendations of Corey Haines, Michael Guterl, James Martin and Michael Hunger I decided to get Kent Beck's screencasts on Test Driven Development which have been published by the Pragmatic Programmers.

I read Kent's 'Test Driven Development By Example' book a couple of years ago and remember enjoying that so I was intrigued as to what it would be like to see some of those ideas put into practice in real time.

As I expected a lot of Kent's approach wasn't that surprising to me but there were a few things which stood out:

• Kent wrote the code inside the first test and didn't pull that out into its own class until the first test case was working. I've only used this approach in coding dojos when we followed Keith Braithwaite's 'TDD as if you meant it' idea. Kent wasn't as stringent about writing all the code inside the test though – he only did this when he was getting started with the problem.

  The goal seemed to be to keep the feedback loop as tight as possible and this was approach was the easiest way to achieve that when starting out.

• He reminded me of the 'calling the shots' technique when test driving a piece of code. We should predict what's going to happen when we run the test rather than just blindly running it. Kent pointed out that this is a good way for us to learn something – if the test doesn't fail/pass the way that we expect it to then we have a gap in our understanding of how the code works. We can then do something about closing that gap.
• I was quite surprised that Kent copied and pasted part of an existing test almost every time he created a new one – I thought that was just something that we did because we're immensely lazy!

  I'm still unsure about this practice because although Ian Cartwright points out the dangers of doing this it does seem to make for better pairing sessions. The navigator doesn't have to wait twiddling their thumbs while their pair types out what is probably a fairly similar test to one of the others in the same file. Having said that it could be argued that if your tests are that similar then perhaps there's a better way to write them.

  For me the main benefit of not copy/pasting is that it puts us in a mindset where we have to think about the next test that we're going to write. I got the impression that Kent was doing that anyway so it's probably not such a big deal.

• Kent used the 'present tense' in his test names rather than prefixing each test with 'should'. This is an approach I came across when working with Raph at the end of last year.

  To use Esko Luontola's lingo I think the tests follow the specification style as each of them seems to describe a particular behaviour for part of the API.

  I found it interesting that he includes the method name as part of the test name. For some reason I've tried to avoid doing this and often end up with really verbose test names when a more concise name with the method name included would have been way more readable.

  A couple of examples are 'getRetrievesWhatWasPut' and 'getReturnsNullIfKeyNotFound' which both describe the intent of their test clearly and concisely. The code and tests are available to download from the Prag Prog website.

• One thing which I don't think I quite yet grasp is something Kent pointed out in his summary at the end of the 4th screencast. To paraphrase, he suggested that the order in which we write our tests/code can have quite a big impact on the way that the code evolves.

  He described the following algorithm to help find the best order:

  • Write some code
    • erase it
    • write it in a different order

  And repeat.

  I'm not sure if Kent intended for that cycle to be followed just when practicing or if it's something he'd do with real code too. An interesting idea either way and since I haven't ever used that technique I'm intrigued as to how it would impact the way code evolved.

• There were also a few good reminders across all the episodes:
  • Don't parameterise code until you actually need to.
  • Follow the Test – Code – Cleanup cycle.
  • Keep a list of tests to write and cross them off as you go.

Overall it was an interesting series of videos to watch and there were certainly some good reminders and ideas for doing TDD more effectively.

One of the other neat ideas I was reminded of when watching Kent Beck's TDD screencasts is the value of 'calling your shots' i.e. writing a test and then saying what's going to happen when you run that test.

It reminds me of an exercise we used to do in tennis training when I was younger.

The coach would feed the ball to you and just before you hit it you had to say exactly where on the court you were going to place it – cross court/down the line and short/deep.

The point of this exercise is that it's relatively easy to just hit the ball over the net but to have control over exactly where the ball's going to go is way more powerful albeit more difficult.

This applies to TDD as well – it's easy to write a failing test but much more useful to write a failing test which fails in exactly the way that we expect it to.

I've written previously about the value of commentating when pair programming and calling your shots is a really easy way of keeping the conversation flowing in a pair.

I like to ask not only how a test is going to fail but why it's going to fail in that way. I think it's a pretty good way of making sure that you as a pair understand exactly what you're doing. It also slows the pace just enough that you're not coding without thinking about what you're doing.

I quite like Bryan Liles' take on this which he described in a talk at acts_as_conference 2009. Bryan suggests that we first write a test and then either make it green or change the error message that you're getting.

We should know whether or not the test is going to go green or the error message is going to change before we run the test and the idea is that we want to either get the test to pass in one go or at least get a step closer to a passing test.

I’ve been dropping a few hints and mentions the last few weeks, and I thought it was about time that I took some time to preannounce a new project I’m working on. It’s going to be much easier writing my next few blog posts if people already know about the project, and my reasons for keeping quiet about it have mostly disappeared. It’s also a moot point since I talked about it at the Emerging Languages camp last week, and the video will be up fairly soon. And I already put the slides online for it, so some things you might have already seen.

So without further ado, the big announcement is that I’m working on a new language called Seph. Big whoop.

Why?

I already have Ioke and JRuby to care for, so it’s a very valid question to ask why I would want to take on another language project - outside my day job of course. The answer is a bit complicated. I always knew and communicated clearly that Ioke was an experiment in all senses of the word. This means my hope was that some of the quirky features of Ioke would influence other languages. But the other side of it is that if Ioke seems good enough as an idea, there might be value in expanding and refining the concept to make something that can be used in the real world. And that is what Seph is really about. That blog post I wrote a few weeks ago with the Ioke retrospective - that was really a partial design document for Seph.

So the purpose of Seph is to take Ioke into the real world while retaining enough of what made Ioke a very nice language to work with. Of course, being the person I am, I can’t actually avoid doing some new experimentation in Seph, but they will be mostly a bit safer than the ones in Ioke, and some of the craziest Ioke features have been scaled back a bit.

Some features

So what’s the difference? Seph will still be prototype based object oriented, in the same way as Ioke. It will definitely consider the JVM its home. It will be homoiconic, and allow AST manipulation as a first class concept - including working with message chains as a way of replacing blocks. It will still have a numerical tower. It will use almost exactly the same syntax as Ioke. It will still allow you to customize operators and precedence rules.

The big difference. The one that basically makes most all other design changes design themselves is a small but very important difference: objects are immutable. The only way you can create new objects is by creating a new object that specifies the difference. This can be done either by creating a new child of the existing object, or creating a new sibling with the specified attributes changed. In most cases, the difference between the strategies isn’t actually visible, so it might be an implementation strategy.

Now once you have immutable objects but still focus on polymorphic dispatch, that changes quite a lot of things. It changes the core data structures, it changes the way macros work, it changes the flow of data in general. It also changes what kinds of optimizations will be possible.

Another side effect of immutability is that is becomes much more important to have a good module story. Seph will have first class modules that ends up still being simple Seph objects at the same time. It’s really a quite beautiful and simple scheme, and it makes total sense.

If you’re creating a new Object Oriented language, it turns out that proper tail calls is a good idea if you can do it (refer to Steele for more arguments). Seph will include proper TCO for all Seph code and all participating Java code - which means you’ll only really grow the stack when passing Java boundaries. This will currently be done with trampolining, but I deem the cost worth the benefit of a tail recursive programming style.

I mentioned above that objects are immutable. However, local variables will be mutable. It will also be possible to create lexical closures. I’m still undecided whether it’s a good idea to leave a big mutable hole in the tyoe system, or whether I should make it impossible for lexical closures to mutate their captured environment. Time will tell what I decide.

Stealing is good

Seph believes in reusing concepts other people have already made a great job with. As such, many pieces of the language implementation will be stolen from other places.

Just like in Ioke, the core numbers will come from gnu.math. This library has served me well, and I’ll definitely continue to use it. The big difference compared to Ioke is that the gnu.math values will be first class Seph object, and won’t have to be wrapped. Seph will also have real floats instead of bigdecimals. This is a concession to reality (which I don’t much like, btw).

Seph will incorporate Erlang style light weight threads with an implementation based on Kilim (just like Erjang).

As mentioned above, the core data structures will have to change. And the direction of change will be towards Clojure. Specifically, Seph will steal/has stolen Clojures persistent data structures, all the concurrency primitives and the STM. I don’t see any reason to not incorporate fantastic prior art into Seph.

As mentioned above, the module system is also not new - it’s in fact heavily inspired of Newspeak. Having no globals force this kind of thinking, but I can’t say I would have been clever enough to think of it without Gilad’s writings, though.

Basically everything else is copied from or inspired by Ioke.

Isn’t mutability the essence of Ioke?

If you have worked with Ioke, or even heard me talk about it, you might have gotten the impression that mutability is one of the core tenets of Ioke. And your impression would be correct. It wasn’t until I started thinking about what a functional object hybrid version of Ioke would look like, that I realized most of things I like in Ioke could be preserved without mutability. Most of the macros, the core evaluation model and many other pieces will be extremely similar to Ioke. And this is a good thing. I think Ioke has real benefits in terms of both power and readability - something that is not easy to combine. I want Seph to bring the same advantages.

Will you abandon Ioke now?

In one word: no. Ioke is still an experiment and there are still many things that I want to explore with Ioke. Seph will not fill the same niche, it won’t be possible for me to do the same experimentation, and fundamentally they are still quite different languages. In fact, you should expect an Ioke release hopefully within a few weeks.

So will it be useful?

Yes. That’s the whole goal. Seph will have an explicit focus on two areas that Ioke totally ignored. These areas are concurrency and performance. As seen from the features above, Seph will include several powerful concurrency features. And from a performance standpoint, Ioke was a tarpit - even if you wanted to make it run faster, there wasn’t really anything to get a handle on. Seph will be much easier to optimize, it’s got a structure that lends itself better to compilation and I expect it to be possible to get it to real world language performance. My current idea is that I should be able to get it to JRuby performance for roughly the same operations - but that might be a bit optimistic. I think it’s in the right ballpark though. Which means you should be able to use it to implement programs that actually do useful things in the Real World ™.

Is it available?

No. At the current point, Seph is still so young I’m going through lots of rewrites. I would like the core to settle down a little bit before exposing everything. (Don’t worry, everything is done in git, and the history will be available, so anyone who wants to see all my gory mistakes will have no trouble doing that). But in a nutshell, this is why this is a preannouncement. I want to get the implementation to the stage where it has some of the interesting features I’ve been talking about before making it public and releasing a first version.

Don’t worry though, it should be public quite soon. And if I’m right and this is a great language to work in - then how big of a deal is another month of waiting?

I’m very excited about this, and I hope you will be too! This is an adventure.

I've been watching Kent Beck's TDD screencasts and in the 3rd episode he reminded me of a mistake I used to make when I was first learning how to test drive code.

The mistake happens when testing collections and I would write a test which would pass even if the collection had nothing in it.

The code would look something like this:

[Test]
public void SomeTestOfACollection()
{
	var someObject = new Object();
	var aCollection = someObject.Collection;
 
	for(var anItem : aCollection)
	{
		Assert.That(anItem.Value, Is.EqualTo(...));
	}
}

If the collection returned by someObject is empty then the test will still pass because there is no assertion to deal with that situation coming up.

In Kent's example he is using an iterator rather than a collection so he creates a local 'count' variable which he increments inside the for loop and then writes an assertion outside the loop that the 'count' variable has a certain value.

In my example we can just check that the length of the collection is non zero before we iterate through it.

[Test]
public void SomeTestOfACollection()
{
	var someObject = new Object();
	var aCollection = someObject.Collection;
 
	Assert.That(aCollection.Count(), Is.GreaterThan(0));
	for(var anItem : aCollection)
	{
		Assert.That(anItem.Value, Is.EqualTo(...));
	}
}

It's a relatively simple problem to fix but it's certainly one that's caught me out before!

Reading time: 2 – 4 minutes

We are using FactoryBeans extensively to own the responsibility of interesting work in constructing our object graph. These are request scoped, because they may depend on other objects that were themselves created by FactoryBeans specific to this request. For example, we have a Customer which needs an Account, so CustomerFactoryBean depends on the field Account to be injected. (We use field or setter injection because otherwise creating a factory bean that depends on another object that is created by a factory bean may create a spring exception about circular dependencies.)

package com.example.dotcom;
 
import com.example.common.Account;
import com.example.dotcom.session.CustomSession;
import org.springframework.beans.factory.FactoryBean;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Scope;
 
// the scope is crucial, this controls the scope of the factory bean
@Scope("request")
public class AccountFactoryBean implements FactoryBean {
 
       // this is itself created from another factory bean
       @Autowired
       CustomSession session;
 
	@Override
	public Object getObject() throws Exception {
		// Sometimes this is going to return null, other times an instance.
                // The fix is to always return a non-null instance (despite what the
                // javadoc says). Use the Null Object pattern.
                return session.get(Account.class);
 
 
		// Note: you will need to implement caching in here or in 
		// a custom scope (which we did). Or every time you need
		// to inject the Account it will call getObject(), which
		// could be problematic if it was a slow operation.
	}
 
	@Override
	public Class getObjectType() {
		return Account.class;
	}
 
	@Override
	public boolean isSingleton() {
		return false;
	}
}

We are using autowiring by type, but we encountered a problem when the result of a factory bean’s getObject() is sometimes null. If the first call to that request scoped factory bean returned null, it would never call getObject() again in future requests. Upon investigation, you can see there is a subtle caching of null return values from factory beans. (Even if they are request scoped.) Might be a bug, I don’t know. But the work around is probably a good idea to implement anyways: use Null Objects instead of passing around null.

Here is the problematic code from Spring that caches the null return value: AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement. There is a field cached and cachedFieldValue. These are set if the value is null, and then on subsequent calls they will return the null value without calling getObject on the factory bean.

I recommend not returning null in factory beans, or else it might be cached by Spring and your factory bean will not be called again when that object is needed.

• The worlds longest photo exposure
  (tags: photography longexposure pinhole)

I’ve just got back from Tokyo, where I was honoured to be invited to give the keynote at Agile Tokyo 2010. Agile conferences have a short history in Japan – both Agile Tokyo and Agile Japan are only two years old – and there was a real buzz of excitement, with lots of smart people (280 delegates attended) and interesting conversations. So first of all I’d like to thank Yoshi Nagase and his wonderful team at Technologic Arts for hosting me, Yoko Yoshikawa for taking such good care of me during my time there, Gihyo for organizing the conference and for inviting me, and my fellow presenters for some great discussions.

Agile Tokyo 2010 by Yoko Yoshikawa

The most surprising discovery for me was that agile is considered new and somewhat subversive in Japan. Japanese IT companies have been doing waterfall on large projects for a long time now, and it is deeply entrenched. Indeed, everybody I spoke to confirmed that waterfall worked perfectly well for them as a software development methodology. This was staggering to me, since outside Japan large software projects run with waterfall routinely go over budget and end up with either cuts in scope or poor quality – several recent reports, such as this one, put IT project failure rates in the range of 60-70%. This appears not to be the case in Japan – it is perhaps the only country that could make waterfall work reliably and repeatably. As my host, Yoshi Nagase, commented, this is deeply ironic in a country that created Toyota and lean manufacturing.

However there is certainly an interest in agile now. Several large systems integrators have started running small agile pilot projects, and apparently the results have been successful. Why the sudden interest in agile if waterfall can deliver high quality software on time and on budget? The answer is easily extrapolated from this example of a project plan I saw at one of the companies I visited – the plan has been changed somewhat to protect the guilty, but not in any essential details – it is a pretty standard plan following the V-model.

For me, the most egregious problem with this plan is that the planning stage for phase 2 of the project comes before the testing stage for phase 1. So anything discovered during the testing and user acceptance stages cannot be fed in to phase 2 – it has to wait for phase 3. Effectively, that means that the feedback loop from discovering a problem in your requirements for phase 1 will have to wait until the end of phase 3 to be delivered to users – a cycle time of 1 year. This makes it extremely hard to respond to customer feedback. Even in the best case scenario, whereby you discover a requirement during the planning stage of a phase, you must wait 6 months for it to be delivered to users.

Japanese companies seem to be realizing that they simply cannot maintain a competitive advantage in the current economic climate with this kind of cycle time. They must get valuable software into the hands of users, and incorporate feedback from them, much faster. This of course is the value proposition of agile methods – the first principle of the agile manifesto, which also inspired the title of our book, is this: “our highest priority is to satisfy the customer through early and continuous delivery of valuable software”.

The risk is that Japanese companies simply have very little experience doing agile delivery at all, let alone on large projects. Adopting agile in an effective way involves serious organizational change. Cross-functional teams including developers, testers, analysts, testers, operations staff, and project managers must be created. Developers must be trained in practices like test-driven development, continuous integration, refactoring, and incremental development. Testers will have to start collaborating more closely with analysts and developers. Managers will need to be trained in agile delivery methodologies – and they will need to make mistakes and gain experience. Teams will need new tools.

However if there is any country that can make such a large change in an organized and efficient manner, it is Japan. I’m really excited to see what agile methodologies do for Japanese IT, and also to see what the Japanese software delivery community does to agile – no doubt we will see many new ideas and evolutions. It’s an exciting time to be in IT.

On the majority of the projects that I've worked on at ThoughtWorks we've held a showcase at the end of each iteration to show our client what we've been working on and finished over the previous one or two weeks.

The format of these showcases has been fairly similar each time but the people who attended has tended to vary depending on the situation.

As part of the project being worked on at ThoughtWorks University we've run a showcase at the end of each week which the whole team have been attending.

Toby pointed out that having everyone there didn't seem to be the best use of people's time since the majority of the developers in the room didn't actually have any input during the showcase.

I've often felt the same in that situation although we used a similar approach on a project I worked on last year and it seemed to be quite useful for addressing the human aspect of the project.

The client was able to meet all the people on the team and equally the developers were able to gain some insight into the types of conversations that the analysts were having with the client.

In that particular example we had around 10-15 developers so there was quite a big gathering but on the other projects where we've had the whole team at the showcase there's only been 3 or 4 of us.

On the last project I worked on we took it in turns to go to the showcase. I think it's quite useful to have at least one technical person in a showcase as they're able to give instant feedback on implementation details such as how difficult it will be to change the way a certain piece of functionality works.

A lot of the time we didn't have any input but it was interesting that quite often we'd come out of those showcases with more understanding of what the client was trying to achieve with a specific feature.

If we don't have all the developers in every showcase we do tend to have them all there for the first showcase at least as that tends to be the one where the most stakeholders will be present.

Overall though it's a call to make depending on the situation and the difficulty in judging the value of attending the showcase seems to come about because a lot of the benefits of attending are indirect whereas staying at our desk and coding has a direct benefit to the project.

One of the interesting side effects of using SmartGWT is losing the ability for people to select text on the page. SmartGWT has some pretty nifty widgets and the one to use when you want text selectable is HTMLPane. Be careful not to confuse this with GWT’s own HTMLPanel

I’ve been meaning to write about this for a while, since I keep saying this and people keep getting surprised. Now maybe I’m totally wrong here, and if that’s the case it would be nice to hear some good arguments for that. Here’s my current point of view on the subject anyway.

A specter is haunting the Java community - the specter of generics.

Java introcued a feature called generics in Java 5 (this feature is generally known under the name of parametric polymorphism in the literate). Before Java 5 it wasn’t possible to create a reusable collection that would ensure the type safety at compile time of what you put in to that collection. You could create a collection of for example Strings and have that working correctly, but if you wanted to have a collection of anything, as long as that anything was the same type, you were restricted to doing runtime checks, or just having good tests.

Java 5 made it possible to add type parameters to any other type, which means you could create more specific collections. There are still problems with these - they interact badly with native arrays for example, and wildcards (Java’s way of implementing co= and contravariance) have ended up being very hard for Java developers to use correctly.

Java and C# both added generic types at roughly the same time. The C# version of generics differed in a few crucial ways, though. The most important difference in implementation is that C# generics are reified, while Java generics use type erasure. And this is really the gist of this blog post. Because over and over I hear people lament the lack of reified generics in Java, citing how good C# and the CLR is to have this feature. But is that really the case? Is reified generics a good thing? Of course, that always depends on who is asking the question. Reified might well be good for one person but not another. Here you will hear my view.

Reified? Huh?

So what does reified generics mean, anyway? It is probably easiest to explain compared to the Java implementation that uses type erasure. Slightly simplified: in Java generics doesn’t exist at runtime. It is purely a fiction that the compiler uses to handle type checking and make sure you don’t do anything bad with your collection. After the generics have been type checked, they are used to generate casts and type checks in the code using generics, some metadata is inserted into the class file format, and then the generic information is thrown away.

In contrast, on the CLR, generic classes exist as specific versions of their class. The same class with different generic type arguments are really different classes. There are no casts happening at the implementation level, and the CLR will as a result generate more specific code for the generic code. Reflection and dynamic type checks is also possible on the CLR. Having reified generics means basically that they exist at runtime, that the virtual machine knows about them and handles them correctly.

Multi-language virtual machines

The last twenty years something interesting has happened. Both out hardware and software has gotten mature enough that a new generation of virtual machines have entered the market. Traditionally, virtual machines for languages were made for specific languages, such as Pascal, Lisp and Smalltalk, and possibly except for SECD and the Warren machine, there haven’t really been any virtual machines optimized to running more than one language well. The JVM didn’t start that way either, but it turned out to be more well suited for it than expected, and there are lots of efforts to make it an even better platform. The CLR, Parrot, LLVM and Rubinius are other examples of things that seem to become environments rather than just implementation strategies for languages.

This is very exciting, and I think it’s a really good thing. We are solving very complex problems where the component problems are best solved in different ways. It seems like a weird assumption that one programming language is the best way of solving all problems. But there is also a cost associated with using more than one language. So having virtual machines act as platforms, where a sharked chunk of libraries are available, and the cost of implementation is low, makes a lot of sense.

In summary, I feel that the JVM was the first step towards a real viable multi-language virtual machine, and we are currently in the middle of the evolution towards that point.

Solving the problems

So why not add reified generics to the JVM at this point? It could definitely be done, and using an approach similar to the CLR, where libraries are divided into pre and post reified makes the path quite simple from an implementation standpoint. On the user side, there would be a new proliferation of libraries to learn - but maybe that’s a good thing. There is a lot of cruft in the Java standard libraries that could be cleaned up. There are some sticky details, like how to handle the API’s that were designed for erased generics, but those problems could definitely be solved. It would also solve some other problems, such as making it possible for Scala to pattern match on type parameters and solving part of the problem with abstracting over primitive types. And it’s absolutely possible to do. It would probably make the Java language into a better language.

But is it the only solution? At this point, making this kind of change would complicate the API’s to a large degree. The reflection libraries would have to be completely redesigned (but still kept around for backwards compatibility). The most probable result would be a parallel hierarchy of classes and interfaces, just like in the CLR.

Refified generics are generally being proposed in discussions about three different things. First, performance, second, making it easier for some features in Scala and other statically typed languages on the JVM, and thirdly to handle primitives and primitive arrays a bit better. Of these, the first one is the least common, and the least interesting by far. JVM performance is already nothing short of amazing. The second point I’ll come back to in the last section. The third point is the most interesting, since there are other solutions here, including unify primitives with objects inside the JVM, by creating value types. This would solve many other problems for language implementors on the JVM, and enable lots of interesting features.

The short stick

I believe in a multi language future, and I believe that the JVM will be a core part of that future. Interoperability is just too expensive over OS boundaries - you want to be on the same platform if possible. But for the JVM to be a good environment for more than one language, it’s really important that decisions are made with that in mind. The last few years of fantastic progress from languages like Rhino, Jython, JRuby, Groovy, Scala, Fantom and Clojure have shown that it’s not only possible, but benificial for everyone involved to focus on JVM languages. JSR 223, 292 and several others also means the JVM is more and more being viewed as a platform. This is good.

Generics is a complicated language feature. It becomes even more complicated when added to an existing language that already has subtyping. These two features don’t play very well together in the general case, and great care has to be taken when adding them to a language. Adding them to a virtual machine is simple if that machine only has to serve one language - and that language uses the same generics. But generics isn’t done. It isn’t completely understood how to handle correctly and new breakthroughs are happening (Scala is a good example of this). At this point, generics can’t be considered “done right”. There isn’t only one type of generics - they vary in implementation strategies, feature and corner cases.

What this all means is that if you want to add reified generics to the JVM, you should be very certain that that implementation can encompass both all static languages that want to do innovation in their own version of generics, and all dynamic languages that want to create a good implementation and a nice interfacing facility with Java libraries. Because if you add reified generics that doesn’t fulfill these criteria, you will stifle innovation and make it that much harder to use the JVM as a multi language VM.

I’m increasingly coming to the conclusion that multi language VM’s benefit from being as dynamic as possible. Runtime properties can be extracted to get performance, while static properties can be used to prove interesting things about the static pieces of the language.

Just let generics be a compile time feature. If you don’t there are two alternatives - you are an egoist that only care about the needs of your own language, or you think you have a generic type system that can express all other generic type systems. I know which one I think is more likely.

曾经我认为，敏捷的各种实践，只要有了标准化的动作，加上一点点定制，加上PDCA/SDCA，就能做好。迈向敏捷之路，是可以唯一定义并且重复实施的。比如说持续集成，大师说 “在自己的计算机上启动一个自动化build”是重要的──我们把它叫做“本地构建”。做不好本地构建，提交构建失败率就会高，对持续集成的信心就会失去。这个问题，和它的解决方案，都是确定且可重复的。

可是这个团队让我吃惊了。他们一直没有真正意义上的本地构建。但他们真的相信持续集成──不仅仅是几个领导，是整个团队，真的相信：如果每天发现的小问题不能及时解决，那么到版本上线时一定会被客户骂得狗血淋头。他们不想被骂，他们也不想同一个战壕里的战友被骂，所以他们就把持续集成做好了。尽管没有真正意义上的本地构建，就靠着原始简单的代码评审，更认真的态度，他们就真的把持续集成做好了。

这个团队颠覆了一些正在我脑中渐渐成型的东西，让我想起了一些原本印象深刻但被渐渐遮掩起来的东西，比如敏捷宣言的第一条：个体与交互 重于 过程和工具。

面对巨大组织时强烈的无力感、无助感会让“敏捷推行人”们（包括我自己在内）不自觉地选择把自己的定位拔高──也许真是想帮助更多的人，也许只是想从残酷的现实中逃开，抑或两者兼而有之。但这些人不会做高层面的事情。于是离开低层面、离开具体事务的结果就是不知道该做什么，就是把敏捷推行简化为过程和工具的推行。

有一个敏捷推行人对我说，我负责几十个项目的改进，如果每次只到一个项目去做具体的事，对整体的帮助很小。我对她说，也许你救不了所有的鱼，但被你救的这一条鱼，它会在乎 ，它会得救。在反复思考如何拯救所有鱼的过程中，其实你什么也没有做，没有一条鱼真的得救。

敏捷的改变，最终是一个一个人的改变。过程和工具会帮助你。但如果把敏捷简化成过程和工具，你就在一步步远离敏捷，因为你已经违背了敏捷宣言的第一条原则。

Last year, I missed the announcement, and therefore the Agile Coaches Gathering UK organised by Rachel Davies (author of Agile Coaching). Fortunately I happened to join twitter late last year and saw the announcement for this year’s gathering shortly after its announcement. I bought a ticket as soon as I could. It’s a good thing too since I think I bought one of the last few tickets for a restricted sixty person gathering.

This year’s gathering, held at historic Bletchley Park spanned a day and a half with Friday evening opening the Open Space format to be used on Saturday. Bletchley Park ended up an awesome venue, and the open space format worked well with a number of spaces located outside. Given the UK actually had a summer this year, it turned out very nicely.

One of my favourite things about this sort of gathering is getting a whole bunch of people working in many different environments, contexts and getting them to share ideas and approaches about what works for them. We had a mixture of experienced coaches and coaches new to the role yet there was something about being there on a Saturday and a passion for it that brought people together. What follows are my notes from each of the sessions.

Experiences from the coaching discipline (not just agile practices)

My first session of the day explored other coach’s experiences working or exploring what coaches from other disciplines do. One person shared their experience working with a swimming coach and how they helped them get better at their practice.

This theme, looking at other coaching disciplines to see what they do, definitely ran throughout the day although I’m cautious of taking the role too literally. We discussed this during this session as most people being coached (life coaching, sports coaching, etc) tend to do so on a voluntary basis. Working as agile coaches for organisations, not everyone is necessarily there on a voluntary basis.

Another aspect that, I think differs, is that most coaches aren’t necessarily experts in the field they’re coaching in. For instance, sports coaches are often not people who’ve been the most successful in the world. In my experiences, agile coaches are really there to act as shepherds to help people get the most out of thinking and working with agile practices. Unlike other coaches, this often requires a level of mastery than what most other coaches have in their field. I’ve seen coaches act dangerously with not enough experience with agile practices.

I still think there is value in looking outside of our discipline for ideas and methods of working, yet conscious of the appropriate context to use them in (and there is always a context).

Presentation is not facilitation

Tobias Mayer is really well known in the Scrum community, and proposed a session to rant about the way that most training is done. I think there are plenty of examples where Death by Powerpoint is interpreted as effective training and whilst I agree with the premise, I’m not sure I agree with the end conclusion. I think presentations have a place, just maybe not in training. I also don’t necessarily think that training is solely about trying to trigger a complete mindshift in people.

When a coach gives up

I met Xavier Quesada Allue at XP2009 and again at Agile2009 so was interested to hear the premise around is it okay for coaches to give up and when to give up. This was one of those sessions located in the sun and some interesting stories about dealing with difficult teams or organisations where it doesn’t make sense to try.

We didn’t attempt to clarify, at the start of this session, what giving up meant to everyone. This probably made it difficult to get an agreement about when or why to give up, although we heard some very interesting stories.

My take on this, is that coaches end up needing to prioritise their work, just like everyone else and thus, not everything is going to get taken care of at the same time. Working with people also means you cannot predict how quickly people will move, or how they will react, therefore there is no way that you can always set completely achievable goals (it relies on others, not just yourself to make them happen). As a result, as a coach, you need to be comfortable with things not always going as planned.

I think that coaches also have the benefit of seeing the system that is driving a lot of the behaviour for the people on teams and unless something is done with that, then they will continue to behave as they always have. Both of these take time.

Game sense approach (what I learned coaching rugby)

Another inter-disciplinary coaching session that I attended briefly although a very loud bus made it difficult to concentrate and I ended up leaving because I had difficulty participating during this session.

Double loop learning + Defensive routines

If you ever get a chance to listen to Benjamin Mitchell in a safe environment, he’s quite the riot. If anything, perhaps it’s his self-deprecating and admitting his own faults and looking back at mistakes amidst his jokes that makes it so warming.

During this session, he talked about a book he’d been reading discussing how people react, and talking about different “models” in which the author classified people and using that to be able to project behaviour in different circumstances.

Once again, Benjamin reminded me of some key points I’ve learned over the years like:

• What you say and what people hear are completely different
• Avoid positional or solutions-based negotiation. Lay your interests on the table and you’ll often end up in a better position.

I’m not really clear about what double loop learning is, so I’m going to have to add that to my to do list as well.

New books

• Training from the back of the room
• The Skilled facilitator
• Situational leader
• Discussing the Undiscussable

Announcing Frank, a lightweight UI automation framework for iPhone and iPad applications.

[Click Here to watch the screencast in a more sensible size!]

Frank sews together several open source tools, notably UISpec, cocoahttpserver, and Cucumber. The goal is to automate basic UI-level acceptance testing of an iPhone or iPad application, integrated into a Continuous Integration system.

You can read more about Frank here.

One of the requirements that the ThoughtWorks University grads have been given on the internal project they're working on is to ensure that they leave the code base in a good state so that the next batch can potentially continue from where they left off.

The application will be deployed on Thursday and this means that a lot of the time this week will be spent refactoring certain areas of the code base rather than only adding new functionality.

When this was suggested Duda pointed out that it's often the case that we might accept a certain amount of technical debt in order to get the application out there.

While he is right and this is quite an unusual situation, we did see a similar situation on the last project I worked on.

On that project there was quite a tight delivery deadline for the first release so we knowingly incurred some technical debt in order to make sure that we met that date.

I've written previously about some of the technical debt that we incurred in that first release and while I think most of the time we made the right call I think there were still some occasions when we thought we were taking on deliberate prudent debt but were actually taking on deliberate imprudent debt.

Luckily it didn't really come back to bite us and in the second release we had a much more relaxed pace and were therefore able to go through the code base and refactor certain parts of it to make it more maintainable.

J.B. Rainsberger has a really cool analogy about refactoring where he talks about cleaning the kitchen and cleaning the garage.

Cleaning the kitchen is what we endeavour to do all the time such that we'll write a bit of code and then clean up after ourselves. Sometimes we don't clean up enough and we end up with a bit of a mess which takes much longer to clean up – i.e. we need to clean the garage.

I think we sometimes drift towards thinking that we don't need to clean the kitchen so often and end up cleaning the garage too often as a result.

This is something that Uncle Bob covered in a post he wrote around a year ago where where he points out that we're more likely to take on technical debt when we didn't need to rather than the other way around.

Finding the right balance between cleaning the kitchen and cleaning the garage seems to be something that comes from the experience of taking either approach too far.

Brian, Tejas and I (well mainly them) have been working on an application to give badges to people based on their GitHub activity at the Yahoo Open Hack Day in Bangalore and we've been making use of Bundler to pull in our dependencies.

Our Gemfile was originally like this:

gem "sinatra", "1.0"
gem "haml", "3.0.13"
gem "activesupport", "3.0.0.beta4", :require => false
gem "tzinfo", "0.3.22"
gem "nokogiri", "1.4.2"
...

For quite a while we were wondering why 'bundle install' wasn't actually resolving anything at all before we RTFM and realised that we needed to call 'source' at the top so that bundler knows where to pull the dependencies from.

Changing the file to look like this solved that problem:

source "http://rubygems.org"
 
gem "sinatra", "1.0"
gem "haml", "3.0.13"
gem "activesupport", "3.0.0.beta4", :require => false
gem "tzinfo", "0.3.22"
gem "nokogiri", "1.4.2"
...

bundler still seemed to have problems resolving 'nokogiri' whereby I was getting various error messages, eventually ending up with this one:

Installing nokogiri (1.4.2) from .gem files at /Users/mneedham/.rvm/gems/ruby-1.8.7-p299/cache with native extensions /Library/Ruby/Site/1.8/rubygems/installer.rb:482:in `build_extensions': ERROR: Failed to build gem native extension. (Gem::Installer::ExtensionBuildError)
 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby extconf.rb
checking for iconv.h... yes
checking for libxml/parser.h... yes
checking for libxslt/xslt.h... yes
checking for libexslt/exslt.h... yes
checking for gzopen() in -lz... no
-----
zlib is missing.  please visit http://nokogiri.org/tutorials/installing_nokogiri.html for help with installing dependencies.

I had to follow the instructions on their website to get that working which isn't ideal as it means not everything is being resolved through bundler.

I'm not sure if there's a way to get around having to do that so if anyone know a way let me know!

• Ian Tomlinson ruling: we must all fight this whitewash
• Quantum mechanics not in jeopardy
• The Daily Mail Song

After some teasing, tugging and pulling I finally got my head around what OJSpec was up to. After understanding what was going on, I stripped it bare and integrated the essential parts of the framework into OJTest. OJSpec matchers are now available for use! I am actually really enjoying writing tests using it and I will probably use OJSpec matchers in my tests from now on (not something that I expected!).

OJSpec

Before we dive into how to use the new matchers in OJTest, we should talk a little history. OJSpec was around before I even started using Cappuccino and Antonio Cardozo was the creator. It was always a framework that I was interested in but never had the time to explore. When we started the OJTest repository, I asked Antonio if we could co-opt his work into the OJTest repository. He graciously agreed and we are appreciative of his gift!

Enable OJSpec

In order to enable OJSpec matchers, you will need to add a -s argument to your ojtest command:

ojtest Test/*.j

becomes

ojtest -s Test/*.j

This is, however, only necessary as long as OJSpec is an experimental feature. When it feels like OJSpec has been used enough to rely on it working, we will enable it by default.[1] Until then, though, you'll have to add the extra parameter.

The Matchers

There are currently 10 different matchers in the repository. These matchers are

    shouldEqual:
    shouldNotEqual:
    shouldBeNil:
    shouldNotBeNil:
    shouldBeSameAs:
    shouldNotBeSameAs:
    shouldBeInstanceOf:
    shouldNotBeInstanceOf:
    should:by:
    should:

Each of these matchers are attached to every Cappuccino object (including toll free bridged objects like strings and arrays). They are built on top of the OJAssert library currently used by OJUnit and the OJTest test runner will catch everything exactly the same way as it does for OJUnit. The difference is in the syntax.

The Syntax

The best way to explain why this different approach might be better or worse is to show a side by side comparison.

The OJUnit way

    [OJAssert assert:expected equals:actual];

The OJSpec way

    [actual shouldEqual:expected];

I really like the second syntax. This is also a bit of a cherry-picked scenario, though. Let's try something a little more complicated. Let's try to verify that something loops 5 times.

The OJUnit way

    var i = 0;
    while([anObject shouldLoop])
        i++
    [OJAssert assert:5 equals:i];

The OJSpec way

    [0 should:"be 5" by:function(i) {
        while([anObject shouldLoop])
            i++
        [i shouldEqual:5];
    }];

In this case, the OJSpec way is a bit more verbose but also a bit more description. It'll also allow us to give more detailed readouts (not yet, though!) on what is happening within our code. If this is a common case, we could also build this into the library.

Matchers Spec

Each matcher operates on an object. The parameters, however, vary. Below is a rough Doxygen documentation draft.

    /**
     * Verifies that one object is equal to the other using == or isEqual
     * @param anObject the object to test against
     * @throws AssertionFailedError
     */
    CPObject-shouldEqual:(id)anObject
    
    /**
     * Verifies that one object is not equal to the other using != or !isEqual
     * @param anObject the object to test against
     * @throws AssertionFailedError
     */
    CPObject-shouldNotEqual:(id)anObject
    
    /**
     * Verifies that an object is nil
     * @throws AssertionFailedError
     */
    CPObject-shouldBeNil
    
    /**
     * Verifies that an object is not nil
     * @throws AssertionFailedError
     */
    CPObject-shouldNotBeNil
    
    /**
     * Verifies that one object is the same object as another using ===
     * @param anObject the object to test against
     * @throws AssertionFailedError
     */
    CPObject-shouldBeSameAs:(id)anObject
    
    /**
     * Verifies that one object is not the same object as another using ===
     * @param anObject the object to test against
     * @throws AssertionFailedError
     */
    CPObject-shouldNotBeSameAs:(id)anObject
    
    /**
     * Verifies that one object is an instance of a class
     * @param aClass the class to test against
     * @throws AssertionFailedError
     */
    CPObject-shouldBeInstanceOf:(Class)aClass
    
    /**
     * Verifies that one object is not an instance of a class
     * @param aClass the class to test against
     * @throws AssertionFailedError
     */
    CPObject-shouldNotBeInstanceOf:(Class)aClass

    /**
     * Runs a test with a bounded, named context
     * @param aDescription the description of the test
     * @param singleArgClosure aClosure with the object passed into it
     * @throws AssertionFailedError
     */
    CPObject-should:(CPString)aDescription by:(Function)singleArgClosure

    /**
     * Stubs a test and marks it as a failure
     * @param aDescription the description of the test
     * @throws AssertionFailedError
     */
    CPObject-should:(CPString)aDescription

Hopefully this is just a small step and we can get into doing things like a custom OJSpec test runner and all that fun jazz. In the meantime, start using this and let me know what you think! Thanks!

[1] It isn't enabled by default currently because OJSpec modifies the CPObject class which can be very dangerous and may have unknown consequences. In order to make sure that it doesn't effect you, you should first run your tests with OJSpec enabled. If you have poor test coverage or no tests, you can do a manual end-to-end test of your application by including @import <OJSpec/OJSpec.j> in your AppController.

• Go back to go forward: the resurgence of prog rock
• How significant is the 'new henge'?

The second day of Emerging Languages camp was at least as good as the first day. We also managed to squeeze in four more talks, since everybode agreed that the afternoon pause was too long and ineffective during day one. At the end of the day my brain was substantially melted that I didn’t even contemplate finishing these comments. But after some sleep I think I have a fresh perspective.

The sessions were a bit more varied compared to the first day - both in quality and how far out the ideas were. Because of how my interest in various subject vary, there might be some inconsistency in length of reporting on the different languages.

Anyway, here goes:

Kodu

Kodu is a language from Microsoft for creating games. It’s specifically aimed at kids to see if they can learn programming in a better way using something like this. The language uses icons and a backend text based syntax to make it easy for someone to program using structure instead of syntax. You get a basic 3d environment where you can modify and edit things in various ways. Another important part of the design is to get the game to quickly do something, so you get immediate feedback. Everything added to the language is user tested before adding it - including doing gender testing. They thought long and hard about whether they should add conjunctions or not - but ended up deciding for doing it. You work with an XBox when programming and running the game. It’s also free. Overall, Kodu looks like a really nice and innovative initiative, probably going back as far as Logo in terms of inspiration. Very nice.

Clojure

Rich didn’t actually talk much about Clojure in general, but decided to focus on a specific problem he is working on solving. His talk title doesn’t really say much about this, though: “Persistent, Transience, Persistents, Transients and Pods - invasion of the value snatchers”. It was a great talk with lots of information coming extremely fast. I found myself focusing more during this talk than during any other during the conference, just to follow all threads of thought.

Rich spent some time giving an introduction to persistent data structures so everyone knew how Clojure works with them - including how they are turned into transients - since that’s where the new feature comes in.

An important part of persistent data structures is that yu preserve the performance guarantees of a mutable equivalent of that data structure. Clojure uses bit-partitioned hash tries, originally described by Phil Bagwell. This allows Clojure to have structural sharing, which means it’s safe to “update” something - the old version is retained. It uses path copying to make it possible to udpate with a low cost. There is definitely cost to doing it, but it works well in a concurrent environment where other solutions would be more costly to get correct results.

Clojure has an epochal time model that allows Clojure to view things as values inbetween being “modified”. State is at one step higher than that, so you can see mutable change as a creation of a new value from an existing value that is then put into the same reference the original value existed in. Clojure has four different types of references with various semantics for coordination.

To get good performance, some Clojure functions will actually mutate state that is invisible to anyone else to efficiently create new data structures. To get performance that is acceptable to Rich Clojure, data structures are not implemented using purely immutable data structures (Okasaki style) from the Java side. Persistent data structures also doesn’t scale to larger changes, specifically multiple collections, several steps or other situations where you want to have functional end points but efficient mutation inbetween.

Transients is a feature that allows Clojure to give birth to a data structure. Clojures transients will accumulate changes in a safe way and can then finally give you a persistent value. Only vectors and hash-maps are currently supported. Lists are not, since there is no benefit in doing that. Transients also enforce thread isolation. Composite operations are OK, and so is multi-collection work and you don’t need any locks for this. This is already in Clojure, but they might be doing too much. They both handle editing and enforce the constraints on it, such as single-threadedness. Transients can sometimes return new values too, even on mutating operations.

Pods allow you to split out the policy from transients. Values go in, values go out. The process goes through the pod. Different policies are possible, such as single-threadness or mutexes. A pod knows how to make a transient version of a value. Functions to modify a pod will have to return a new thing (or the same thing). Dereferencing the pod allows you to get a new value from a pod at that point. This gives you the possbility to apply recipes on ordinary Java objects too. A good example is String vs StringBuilder. Pods can ensure lock acquisition order, but not lock composition - although pods can detect it at least. There are still a few details in the design that Rich hasn’t decided on yet.

All in all, a very interesting talk, about the kind of concurrency problems you wish your language had.

E/Caja

Mark Miller recapped the interaction models of the Web, starting with static frames going to the current mess of JavaScript fragments going back and forth, using JSONP, AJAX and Comet. He also talks a bit about  the adoption curves of languages and why some languages get adopted. Posits that a mess of features may be easier to get adopted. This means many languages succeed by adding complexity.

E is an experiment in expressing actors in a persistent way. He used some of the lessons from E combined with AJAX/JavaScript to create Caja, a secure language. Some of the features from Caja were then used  to start work on EcmaScript 5. They are currently working on a standard for SES, secure JavaScript. Dr. SES is an extension of this, that stands for Distributed, Resilient, Secure JavaScript. Object capabilities involve two additions to a regular memory safety and encapsulation model; effects only on held references, and no powerful references by default. This means a reference graph becomes an access graph. Caja can sanitize JavaScript to prevent malicious behavior, but preserve the semantic meaning of the program outside of that.

He showed some examples of how Caja can be used to sanitize regular JavaScript and have it running securely. Very interesting stuff, although the generated code didn’t look as amenable to debugging as something like CoffeeScript.

Fancy

Fancy is a language that tries to be friendly to newcomers, with good documentation, a clean implementation and so on. It’s inspired b several languages: Smalltalk (pure message passing, everything’s an object, dynamic, class based OO, metaprogramming, reflective),  Ruby (file based, embraces UNIX, literal syntax, class definition is executable script, fixed some inconsistencies with procs/lambdas/blocks), Erlang (message passing concurrency, light weight processes - not implemented yet). Fancy takes the opinion that first class is good; classes, methods, documentation, tests should all be first class. FancySpec is a simple version of RSpec. Tests for all built in classes and methods are there. These tests are not dependent on implementation. There are plans to port Fancy to a VM. Methods marked with NATIVE will have an equivalent method in Fancy and in the interpreter, to improve performance.

It’s got dynamic scoping and method caching. Logic can be defined based on the sender of a message, which makes it possible to do things like private and public.

Exceptions are taken directly from the implementation (ie C++).

The language seems to be pretty similar to Ruby in semantics, but more Smalltalk like syntax.

BitC

BitC is geared towards critical systems code. Resource contrained, CPU, memory, those kind of areas. One cache miss sometimes counts. Abstraction is fine, but only if it’s the right one. Variance constrained too. Predictability is very important, so something like a JIT can be a problem. Statically exception free. “Zero” runtime footprint. Non-actuarial risk model. Mean time between failures in decades. Problem is to establish confidence. After other failures in this area, the conclusion has been that BitC shouldn’t be a prover.

The language is an imperative functional language with HM-style parametric type system. You have explicit control of representation. State is handled in a first class manner. Inferencing actually infers mutability in lots of cases. Dependent range checking isn’t there yet, but is coming soon. “The power of ML/Haskell”, “The low-level expressiveness of C”, “Near-zero innovation”.

Trylon

Trylon is a small language, indentionation based and compiles through C. It’s object oriented, with prototypes under the class based system. According to the author, nothing really new in the language - he just did it for his own sake. There are no users so far except for the author.

ooc

The language tries to be a high level low level language. It mixes paradigms quite substantially and has some nice features. It’s class based, and mostly statically typed.

Coherence/Subtext

Jonathan Edwards started this presentation by showing a small example where the ordering of statements in an implementation is dependent on what representation you use for data, and shows that it’s impossible to handle this case in a general way. From that point he claims that there is a fundamental tension between styles in a language, and you can only get two of these three: Declarative programming, Mutable state and Data Structures. I’m not sure if I agree with his conclusions, and the initial example didn’t feel like anything I’ve ever had trouble with.

Based on the conclusion that you can only have two of the three, he goes on to claim that the thing that cases all these problems is aliasing. So in order to avoid aliasing, his system uses objects where instances are physically always contained within another object. This means you can refer to these objects without having actual pointers - and thus cannot do aliasing either. From that point on, his system allows declarative programming of the flow, where updates never oscillate back out to create more updates.

Lots of interesting ideas in this talk, but I’m not sure I agree with either the premise or the conclusions.

Finch

Finch is a small programming language, bytecode compiled with fibers, blocks, TCO, objects, prototypes, a REPL and Smalltalk style message selectors. In the feature, the author aims to add metaprogramming, some self-hosting, continuations and concurrency features.

Circa

Circa is a small programming language that allows you to get immediate feedback. It’s aimed at game programming, and achieves this by running the script many times (one time for every frame as far as I understood it). You then specify what state you have in your program, and this state will be automatically persisted between invocations, so that a specific invocation of a specific function will always get access to the same state it started out with. This was a very interesting but weird model. It seems to work really well for smaller prototyping of games and graphics but I’m wondering what can be done to expand it.

Wheeler

Wheeler is a proof of concept presented by Matt Youell. It’s pretty hard to describe, and I’m not even sure if there’s a computational model there yet. The project is apparently just a few weeks old, and the ideas are still in progress. The basic tenets of the language seems to be that you work with categories of things and establish transitions between them. A transition pattern matches things that it looks for, which means that things like syntax and ordering doesn’t mean as much. The author calls it mutual dispatch, because it uses the types/categories of everything involved to establish what transitions to use. At this point there is no time model, so everything happens in one sweep, but once a time model gets in there it might be very interesting. To me it looked a bit like a cross between neural networks and cellullar automata.

Interval arithmetic

Alan (Mr Frink) gave a talk about the problems with floating point numbers, and one way of handling that. Floating point numbers cause problems by making it possible to introduce small errors.

Intervals is a new kind of number. It represents a specific number by giving two end points and saying the real number is somewhere within that interval. You can see it in two different ways: “the right value’s in there somewhere but I’m not sure where” or “the variable takes on ALL values in the interval simultaneously”.

This was a very interesting discussion, and you can find out more about it from Frink’s web page (just search for Frink and interval arithmetic). At the end of the presentation, Alan gave this challenge to other languages:

for:

x=77617

y=33096

calculate:

((333 + 3/4) - x^2) y^6 + x^2 (11x^2 y^2 - 121 y^4 - 2) + (5 + 1/2) y^8 + x/(2y)

Ioke handles it correctly, both using ratios and using decimals.

Stratified Javascript

Stratified JavaScript adds some concurrency features to JavaScript based on Strata. It looked like a very principled approach to giving JS concurrency primitives that are easy to use at the same time as they are very powerful. The presenter showed several examples of communication, blocking and coordination working really well.

Factor

Factor is a very high level stack based language created by Slava Pestov. He went through some of the things that Factor does well and other dynamic programming languages handle less well, like reloading code from the REPL. Lots of other small tidbits of how powerful Factor is and how expressive a stack language can be. At the end of the day I still think it’s interesting how much Ioke code sometimes resemble Factor, even though the underlying semantics are vastly different.

D

Walter Bright showed D, his systems level programming language. He focused on showing that it can do several different paradigms in the same language - all of it looked very, very clean, but I got the impression that D is an extremely big language from these examples. To summarize, D can do inline assembler, class based OO, generative programming, RAII, procedural, functional and concurrent programming (and I probably missed a few). I liked the approach to immutability, but I must admit I’m scared of the sheer size of the language. It’s impressive how such a big language can get so good at compile times.

AmbientTalk

AmbientTalk is a language built on top of Java that puts communication in center. It is supposed to be used in areas where you have bad network connectivity and want to communicate inbetween different devices in a flexible way. Things like network outages aren’t exceptions because they will happen all the time in the environments AmbientTalk is built on. The language embraces futures to a large degree and also takes a principled approach to how Java integration works - so that if you send an AmbientTalk object in to Java, it will work as if you had sent it to a remote device, and the only way Java can interact with that object is by sending messages to it. Much interesting stuff in this talk.

And that was it. I can obviously not capture all the interesting hall way and pub conversations that were had, but hopefully this summary will be helpful until the videos come along in two to four weeks. I would call this conference a total success and I really look forward to next year.

A new release of SWTBot is now available at a the usual location.

This purpose of this release is to work with the helios release and contains a few bug minor fixes.

A few other improvements were done to improve the code coverage for some plugins that were not tested too well.

Forthcoming releases will improve on the code coverage. Well, the irony.

• How to Distinguish Fiction From Nonfiction
• Archaeologists unearth Neolithic henge at Stonehenge
  
  I'm most of the way through a book about Stonehenge, and they have to go and re-write it!
• The Numbers No One Wants to Talk About in the Android vs. iPhone Sales Battle
  
  "65,000 iPhone 4s sold each day, which is nearly 100,000 less, PER DAY, than Android phones"
• 12 Timeless Writing Tips from Mark Twain

I recently came a blog post written by Matt Ward describing some habits to make you a better coder and while he presented a lot of good ideas I found myself disagreeing with his 2nd tip:

2. Write Your Logic through Comments

When it comes to coding, there are many tenets and ideas I stand by. One of this is that code is 95% logic. Another is that logic doesn’t change when translated from human language into a programming language.

What this means is that if you can write it in code, you can write it in a spoken language like English or French.

…

Instead of just jumping into coding the function, I could step back and write the logic in plain English as comments.

I've tried this approach before and although it can be useful I found that quite frequently I ended up with a more complicated solution than if I'd driven it out with a test first approach.

The advantage of driving the code from examples/tests is that it helps to put you in a mindset where you only need to care about one specific way that a particular object may be used.

As a result we often end up with simpler code than if we'd tried to imagine the whole design of that object up front.

I find it more difficult to keep a lot of code in my head but having just one example is relatively easy. The less code I have to think about at any one time the less mistakes I make.

As we add additional examples which describe different ways that the object may be used I've often found that the code ends up becoming more generalised and we end up with a simpler solution than we might have imagined when we started.

Matt goes on to say:

This way, I can think through the full logic and try to iron out the wrinkles before I actually get into writing the code. I’ve found this to be an incredibly valuable habit that tends to result in fewer bugs.

Using a TDD approach "allows us to describe in code what we want to achieve before we consider how" so the examples that we write provide an executable specification of what we expect the code to do.

I don't have a problem with making mistakes when coding. I make mistakes all the time but having the safety net of tests helps me fix them pretty quickly.

Matt ends this section with the following:

As a bonus, since I will rarely actually delete the comments, writing the logic through comments also means that my code will already be documented, making it easier for others to follow my logic if they ever have to work on it, or even just for myself, if I have to come back to it several months or years down the road!

There are other ways of documenting code which don't involve peppering it with comments. We can write our code in a way that reveals intent such that instead of having this:

// FUNCTION: Lock On Time
// This function will accept two time values, indicating the range through
// which it should return an unlocked status.
 
  // Create a new data object
 
  // Using the data object, get the current time
 
  // IF the current time falls within the range passed to the function
 
    // Return false – meaning that we are currently unlocked
 
  // ELSE
 
    // Return true – meaning that we are currently locked.
 
  // ENDIF
 
// END FUNCTION

We have something closer to this:

public bool ShouldBeLockedBasedOn(DateTime startOfTimeRange, DateTime endOfTimeRange)
{
	var dataObject = CreateDataObject();
	var currentTime = dataObject.CurrentTime;
 
	if(currentTime.IsBetween(startOfTimeRange, endOfTimeRange) 
	{
		return false;
	}
	return true;
}

…where 'IsBetween' would be an extension method on DateTime. We could have that as a private method but I think it reads better this way.

Comments don't tend to be maintained in the same way that code is from my experience so as soon as the code around them changes we'll find that they quickly become misleading rather than helpful.

There are certainly times when it makes sense to put comments in code but using them as a substitute for writing intention revealing code isn't one of those!

One of the things Business Analysts do is to facilitate customers in shaping their release.  You could be shaping what the next immediate release be, or indeed, planning multiple releases that span months if not years.  A good practice that we foster here at TW is to guide our customers to a direction that will best help them figure this out.

To give a counter-example, picture a chaotic release planning.  Little considerations have been placed in selecting what features were to go into the next release, except for one rule: cram as many requirements as one could possibly squeeze (all taken from good old backlog, pick from largest estimated features).  There was no collaboration or group discussion between business stake holders but a checklist being sent out via e-mail afterwards, and all manners of chaos and pain followed.  Now back to reality and into the world of product development where this counter-example is not an option – release planning is no result of a random act, as the next release could well make or break your customer’s impression of your software that could either cause them to embrace your software, or leave.

So how do you help your customer figure out which features to have in a single release?  First, let me introduce you to the bucket.

The Bucket Analogy – Your release as a bucket

Imagine for a moment that your release is a bucket.   Now, you have a few bigger stones, a few bunches of pebbles and some sand to fill this bucket up.  The size of the stones and pebbles are relative to the size of features that go into your bucket, your next release.  A bigger stone is a bigger feature with a larger estimate, a pebble a smaller feature, and then you have the bunches of sand that make up the smaller features or tweaks that you know will be great to have, such as delighters that will help make things easier, faster or less painful.

So the goal is to fill the bucket up with the most useful things… no rocket science and straightforward enough, surely?

Prioritisation Tool – How to pick and choose to fill your bucket

Here comes the nitty gritty bit.  You know your bucket is not a bottomless barrel due to budget –  be it money, people, time.  Besides, a bigger bucket would probably not help you release any quicker – that’s a different discussion – the key to this point is figuring out how many, and what stones, pebbles and sand to fill your bucket.  And oh yes, don’t forget that each feature has to give you the most value in the most efficient, timely way.

In an attempt to answer this, I want to first borrow a leaf from a different discipline, Psychology studies.

There are 2 tools to help understand the basic concepts of this idea.  The first tool is the Herzberg hygiene/ motivator theory.  Herzberg says that there are different levels of job satisfaction.  Hygiene factor – having a safe and adequate working environment to carry out work, and the motivation factor – making work enriching for the people.  Apply this to software thinking: we need to make sure that the features in a release will not take away any hygiene factors, otherwise the people who use it will grind to a complete halt and not be able to do their job.  If you had taken away some of the enrichment part, they may not be able to do things quicker or easier, provided you give them workarounds, it will not stop them from completing a certain task.

The second tool is Kano model.  This is a way of helping customers think in terms of building quality.  This tool is the most apparent when applied in product development situations.  Kano model states that product attributes can be mapped to customer satisfaction levels: Basic, Performance and Excitement.  This model can be applied to requirements.  Imagine all requirements can be grouped in the 3 categories: Basic, Performance, Excitement.  You can build quality in a product release by choosing how much to invest in the 3 categories that will make your product meet your users’ most pressing needs.

The most difficult aspect of this is to do a balancing act of judging which features are to stay, and drop which ones in the release.  Which ones we can live without for now, provided that we get it later? You as BA for sure do not have all the answers and insight in this, so this needs to be executed in a workshop or a series of workshops to get you to your goal.  It also pays to verify the assumptions with your business stakeholders and indeed the end users.  Having said that, you are there to help shape their thinking so make sure that they understand every feature they are asking for has justified reasons, financial, strategic… Or perhaps not, in other cases.  Playing devil’s advocate sometimes help, as is re-playing conversations (“Last meeting we got all the stones in place for next release and we have pebbles next”) and key movements (“So last Friday we said we would want X and Y in the next release, and not Z because performance was not a showstopper…”) as pit stops throughout the session.

Whilst we are on the topic, there is also a clarification.  Be careful this is not to be interpreted as the Must have, Should have, Could have (MoSCoW) categories that is so commonly used.  By no means basic maps onto must-have, performance to should-have, and excitement to could-have.  This is because the brutal truth is that not all releases contain all Ms, Ss and Cs – some only have Ms and Ss and Cs never make it to the next drop.  However you need to be aware that all elements of basic, performance and excitement need to be present in a particular release.

Amalgamating 2 brilliant ideas, and turning it into one concept

Herzberg’s two-factor theory teaches us that there are some feature that you just cannot live without.  Kano model tells us what impact each feature would have on that release if you had taken those away.  By first introducing the bucket analogy, a simple conceptual model to imply the need to prioritise, and then combining the 2 theories behind the logic of what to pick and choose, then instil this to your client in order to help them gain the much-needed focus and the courage to prioritise effectively.  Try it, use that in your next release planning exercise.  It’s essentially 2 ideas amalgamated into one.

In a recent project I encountered this situation head-on, and this is how they have adapted the theory into practice and I thought I might share it here as an ending note.  This is what they have done:

Requirements, in the form of stories, are first associated with an estimate and a priority defined by the customer.  In addition to this, a parity field was added.  This field had a reincarnation of Basic, Performance, Excitement terms from the Kano model (known by different names, cannot disclose sadly)…  As soon as the idea of filling a combination of them in a release bucket was presented to the client, it took off almost immediately and each requirement were tagged with this.  Basic was the cornerstones of that release, in other words the hygiene features.  It was so well received that,  each release from then on, it has to have a mix of all basic, performance as well as excitement stories, period.  Thinking back the time at the beginning of the engagement which seems like a different era altogether, when the business stakeholders were adamant that almost everything had to be in the first release(!)  Then, slowly and surely, as the project and our relationship evolved, they have learnt to embrace the theory behind stones, pebbles and sand, and gradually realise that some features could wait until a later release.

• Unattended, Encrypted, Incremental Network Backups: Part 1
  (tags: s3 aws duplicity backup incremental sysadmin debian)
• Bash Script: Incremental Encrypted Backups with Duplicity (Amazon S3) « damontimm.com
  (tags: s3 aws duplicity backup incremental sysadmin)

Yesterday was the first day of the Emerging Languages Camp, a part of OSCON specifically organized for language creators and designers. You can read more about it at www.emerginglangs.com. The first day was fantastic, lots of very interesting talks and great conversations. The amount of brain power in this room is really humbling.

The format of the camp is that there are about 20 speakers and each speaker gets 20 minutes. This is a fairly limiting format and means the speakers will have to focus their talks quite substantially. I expected a few talks (including my own) to bomb completely because of this, but it didn’t happen during the whole day. All of the talks were very different but good in many ways.

All of the presentations are filmed by Confreaks and will be available within a few weeks.

I’ll try to write a few sentences about each presentation, with thoughts and impressions baked in.

Go

Rob Pike started out the day by talking about the history of CSP (communicating sequential processes) and the lineage of languages that led to Go. Most of the talk was based on using channels/goroutines to handle concurrency. It was definitely a good talk, but it didn’t get me more interested in using Go for anything.

Ioke/Seph

I had the second slot. I had twenty minutes to cover both Ioke and a new language I’m working on, called Seph. Against all odds, my talk went quite well and I managed to communicate the things I wanted to get said. Hopefully the audience wasn’t too bored.

Thyrd

Thyrd is a proof of concept visual language, focused on using tablets for programming - so it’s distinctly none-textual. In many cases you drag and drop operations instead of typing them. The actual development happens in a recursive grid of cells. I’m wondering what the audience for this language would be - it definitely looks intruiging though, and I like how some algorithms ended up being very easily readable and understandable.

Parrot

Allison Randall gave a talk about what’s currently happening with Parrot. It seems they are going for a new rewrite of most of the subsystems. One of the changes is going from a CISC style op code system to a RISC style. Parrot apparently has over 1200 op codes at this point, and they want to scale back everything to about 20-30 bytecodes instead. As a preparation for this, they have ripped out the JIT and will revisit most of the subsystems in Parrot to see what can be done. Allison also gave the audience the distinct impression that Parrot is still quite slow for user programs.

Ur

Of all the talks during the day, I think I understood the least of the Ur/Web talk. Ur is a functional limited programming language focused specifically on building web applications. It’s got dependent types inspired by Agda and allow you to statically check your whole program. The example shown was a simple CRUD app, and I didn’t get any impression of how complicated it would be to actually use it for a real world application. The speaker said the only real world web app he knows about is a hosting application for Ur applications that he is building himself.

Frink

I don’t think I can do this presentation justice. Frink is just incredibly cool and you should check it out. It’s a general purpose programming language, but it’s got units of measure and several other features builtin that makes it very easy to use it to calculate all kinds of interesting facts. As an example, he showed that if all people in China jumped at the same time, that would be equivalent to 4.7 on the Richter scale.

Newspeak

Gilad Bracha talked a bit about the basic ideas and principles behind Newspeak and what the current status is. Gilad focused on no global state, and all names being late bound (including class names). The first feature falls quite naturally out of prototype based OO, so it’s something both Io, Ioke and Seph has (and it’s really nice). The second feature is a bit more obscure, but I’m not sure if it gives as many benefits as the first one.

F#

Joe Pamer talked about what they had to do to take F# from a research language to something Microsoft could ship in Visual Studio 2010. Not something most of us really think about, but there are lots of challenges in doing that kind of transition. Joe covered this quite well and also gave us an insight into the current state of F#.

CoffeeScript

CoffeeScript is a language that compiles down to JavaScript. In comparison with GWT for example, it’s pretty close in semantics to JavaScript, and the generated code can be debugged and looked out without wanting to stab out your eyes. The syntax of CoffeeScript is very pleasant and looks very nice to work with (it’s indentation based, and focuses on getting lambdas to be as small as possible). Next time I’m reaching for JavaScript, I think I might just go for CoffeeScript instead. Good stuff.

Mirah

Charles Nutter covered Mirah (the language formerly known as Duby). It looks more and more complete and useful, and sooner or later I’m going to try switching most of my Java development to Mirah. The extensability features makes it possible to do metaprogramming tricks in Mirah that you wouldn’t even try in Ruby.

Io

Steve jumped in last minute to cover for the Objective-J guy who couldn’t be here. Steve covered the basics of Io, talking about concurrency and the other basic features.

It’s been a great first day, and now day two begins - so I’ll have to focus on that.

• How super? What cyclists make of superhighways
  
  "Colliers Wood, on the jam-packed A24 and near a huge supermarket saver-centre, seems an improbable place to start a revolution."
• What to do when PyPI goes down
• eSobi Mobile
• Android Day 1: SDK, Eclipse IDE, and device activation
• I come from Java and want to know what monads are in Haskell
• Outsourcing has become a bone of contention at the Guardian

I'm currently working as a trainer for ThoughtWorks University (TWU) and the participants have some Industrial Logic e-learning material to work through before they take part in the 6 week training program.

I've been working through the refactoring/code smells courses myself and while I've been finding it really useful, I think this was partly because I've been able to link the material to situations that I've seen in code bases that I've worked on over the past few years.

It would have been interesting to see if I'd have got as much value from going through the material 4 years ago before I started working at ThoughtWorks and didn't have a range of code bases to relate the patterns to.

My thinking is that I would have found it more difficult to see the value of the material and that the approaches described would just have seemed 'obvious' despite the fact that I've made pretty much all of the mistakes that the training looks to address!

About a year ago I wrote a blog post where I described the value of re-reading books and one point which I'd forgotten is that even though it may be hard to relate to some ideas the first time you come across them it's still valuable to read about them anyway.

It helps to prepare your mind for when you eventually come across some code where the idea can be applied and Krishnan pointed out that this was actually part of the feedback received from the current TWU participants.

I think this is probably a good example of a variation of confirmation bias at work in that since we've been prepared to see certain patterns/potential to use different refactorings in code when we do come across those situations in code we're much more likely to see those situations.

Krishnan pointed out that it would still be very useful for the TWU trainers to refer back to the Industrial Logic material when we come across those patterns in the project simulation that we run as part of TWU.

I think this is the most important part of the process as it will help to reinforce the learning.

I'm still curious whether there ever is a time when it makes sense to delay learning about something until we have more context.

I find that when I'm learning about something which goes way over my head I'll often stop doing that and pick something else to look at which I can relate to a bit better.

I try to go back to the original material again later on but what I find often happens is that I'll come across it in the future more by coincidence than design and this time it will make more sense.