Category Archives: Software Development

Thoughts on software development practices, events, etc.

Can I Get Some Support Here?

I’ve been chasing my development comrades around for a year or so, convincing them that test-driven development is a good thing.  That continuous integration is a great way to show what  a great job we’re doing by being transparent – posting our success and coverage numbers for everyone to see.

All the developers I can get to try it come back saying, “You know, I wouldn’t have caught this bug before it went to production without TDD.”

Score!  A convert!

We set up a CI server running JetBrains’ TeamCity.  Dead easy and it’s been working great.  We have about four development teams using CI this way now – we’re still working on the testing – and even the configuration management group likes what they’re seeing.

So what’s the problem?

Memo from the standards group.  “Uh, we don’t have TeamCity on our list.  Your choices are, CruiseControl, Rational Buildforge, or Team Foundation Team Build.”

My response was, “Okay, but we’re in the middle of a transition here.  Do you suppose we could just go ahead and change over later (once the habit is good and engrained)?”

Their answer?  “We’re not comfortable with that”

I am getting a lot of support from the local management, so hopefully things will brighten in the next day or so.

Next.. Can I get you guys to estimate in some TDD time?

Advertisement

What? A Post About Software Development?

When I started this blog, my intention was to regale the reader with pithy and insightful pronouncements on the state of software development. 

Since I didn’t want to maintain multiple blogs, I thought I’d put the odd stray commentary about life or books or politics on here too.

Naturally, what happened was that I didn’t do as much of the software stuff, and a lot more of the commentary stuff.  Software topics became the stray..  Astray to be honest.

So, for once in a long while I thought I should do a post on software.

It should be plain to my readers (the two of you out there) that the company I joined in late 2007, was bought out by the Bank of America.  Nice, right?

Absolutely!

For a bank, it is surprisingly flexible about things like open source (there’s an official FOSS group in the bank) , agile development (we have two standard and supported SDLCs — one is agile; guess the other) and personal involvement (we have an internal open source-style repository and user groups).

Great!

The one thing that irks is that we aren’t allowed to have mini test databases on our individual machines, whether laptop or desktop.   Security is the thing, and not unwarranted.  We don’t need some goof leaving a laptop in his car with a few hundred thousand customers’ data on it.

What we need is a way to sanitize the data so that it’s useful for testing, but useless for outsiders — or even larcenous insiders.

Gerard Meszaros’ book, xUnit Test Patterns lists using a centralized testing database as an anti-pattern.  Agreed.

So, how to go forward?

The database can help if it is aware of the relationships between tables.  A semi-sophisticated script can replace any identifiable information with bogus but readable chaff.  The keys used can be replaced with matching but unuseful values.

These exist, right?  I mean, why don’t we hear about them more?

Is this another open source project waiting to be started?  I haven’t looked, but maybe I will tonight…

Donald Knuth Doesn’t Get It?

I was surprised to read excerpts from an interview that Andrew Binstock did with the pre-eminent Donald Knuth regarding aspects of software development methodologies and the state of programming generally.  The summary was provided here: http://www.artima.com/forums/flat.jsp?forum=276&thread=229705

The actual interview is here: http://www.informit.com/articles/article.aspx?p=1193856

What I found interesting is that, perhaps not surprisingly, the reviewer (Frank Sommers) doing the summary picks out supposedly controversial topics from the detailed interview leaving the impression that Dr. Knuth is out of touch with software development topics — at least from where I stand.

So I went and read the entire interview.  What I was left with was a brilliant, opinionated academic computer scientist who by his own admission is more interested in the science of computing than in software development per se.

He doesn’t elaborate on what in eXtreme Programming that he finds to be ‘exactly the wrong way to go’ which leads me to assume (perhaps incorrectly) that his objections are more due to the hype often associated with XP than with proper practise of it.  This article comes to mind: http://www.xprogramming.com/xpmag/jatBaseball.htm

There are a number of things on Dr. Knuth’s website that are interesting.  I’ll be playing with MMIX simulators well into the night..

Then I went to Dr. Knuth’s homepage and discovered his “Infrequently Asked Questions” page: http://www-cs-faculty.stanford.edu/~knuth/iaq.html.  Dr. Knuth is thinking about other things in a rather lucid way..

Thanks for looking in.

Spoiled!

I worked as a consultant for 3 years doing Agile Development (yeah, I used caps for emphasis), then I worked at ThoughtWorks for 2.5 years doing Agile and coaching clients in Agile Methodologies.  It was great and empowering as a developer.

Before ThoughtWorks, there was a large software company creating applications for municipal energy cooperatives and a large supplemental health insurance company, then about five clients while I was with ThoughtWorks.  All of them were impressed with the real efficiencies that agile development techniques — properly implemented — provide.

Now I’m working at a large mortgage lending institution in southern California.

Today, a demand came that the project I’m working on produce a technical specification for a process that we don’t really know, and to document a solution that we haven’t demonstrated is a real solution other than theoretically.  Why?  Because someone is pressuring the BAs for these documents without regard for their value (or lack thereof).

I think it’s pretty clear that there is really no value in the the specific documents that are being requested (after a little pushback, it became a request and not so much of a demand).  It’s classic CYA.

I will be doing everything I can to demonstrate to those who really count, that the only real measure of successful development projects is, surprise!, working software that satisfies their requirements.

I’m all for standards.  We will be using the .NET tools that are mandated (and some that are better).

We will demonstrate that we have excellent test coverage (and good depth of coverage too).

We will practice the art of simplicity — ‘the art of maximizing the amount of work not done.’ (that’s straight from the Principles behind the Agile Manifesto) — so that we complete our assignments without wasting time on features we won’t need.

I only hope that they — all of the non-believers — finally get it, at least to the point that we can do our jobs and feel empowered, like before.

It is frustrating having to plow the same ground again, but I guess it’s another example of The Dip.  I could quit, but I believe that the rewards of not quitting will be worth it.

Domain-Driven Design and the Evil Necessity..

I’m not completely serious but I genuinely see the database as a means for persisting object state.  I used to take the completely opposite position too, but I grew up a while ago ;-).

My preferred M.O. is to essentially ignore the database design early in the development process, and to gradually pay more attention as the performance begins to suffer.  Unless you have a really simple application, data retrieval will gradually slow things down to an unacceptable level of performance.

I have had my teams use simple text files, and object serialization instead of RDBMS’s, but ultimately you need to consider (real) persistence.

It’s a good idea to add performance smoke testing early and keep an eye on it.  When you can extrapolate some numbers in terms of transactions required and elapsed times, do some analysis and see if it’s within the tolerable range.

This is where a mapping tool like iBATIS is great.  You can control the actual database access methods and map them to your classes in arbitrarily complex ways, so it leaves your options more open than an ORM like Hibernate might.  iBATIS tends to handle legacy databases well too, as you hopefully migrate to something more manageable. 

In my current situation, we’re looking at the daily production of tens of millions of rows and the attendant maintenance of 100s to 1000s of millions over time, and there is legacy data to consider.  Ouch.

It would be great if you could count on the ActiveRecord pattern exclusively, but you really can’t.

Behavior-Driven Development is a Specification Technique.. It Should Use the Ubiquitous Language

In a post in his weblog, Greg Young posits that there is, and should be, a separate “[Shared Language]” that is distinct from (or orthogonal to) the Ubiquitous Language (UL) we discuss is the context of Domain-Driven Design (http://codebetter.com/blogs/gregyoung/archive/2007/10/23/bdd-and-the-shared-language-the-stakeholder.aspx)

The premise is a statement in the entry, as follows:

A stakeholder may or may not be a domain expert. If a stakeholder is not a domain expert do they belong in the inner-circle of your team? In other words to they deserve to be privy to your ubiquitous language?

My answer is: Of course they should be privy.  The Ubiquitous Language is not a secret that the development team keeps to itself.  It is meant to be a shared vocabulary for discussing elements of the application regardless of the level of the contribution; from Sponsor, to Stakeholder, to Analyst, to Developer, to Tester.

Speaking of stakeholders, Greg goes on to say:

With this customer you should absolutely NOT discuss the model in the ubiquitous language as it only exists within your development team (whatever your code is at any given point is your defacto ubiquitous language as you all understand the code) and it likely does not represent reality in the slightest.

I can’t disagree more. 

(Well that’s not entirely true. Dismissing my desire to speak directly to the target users, I had a business analyst tell me that “There is no way you can trust the users to know what they want.  I’ve seen too many cases of ‘Lost in Translation'”.  “Exactly!”, I said.  But I digress).

If there’s one way to get stakeholders on board, and keep ’em there, it’s to adopt a language that’s close to the one they know about their domain (it’s their domain too) and to be able to discuss their issues, concerns and desires for the application in that language.  I’d go so far as to say that if the ‘Stakeholder’ doesn’t understand your UL, they are a stakeholder in name only.

I believe very strongly that that language needs to span all aspects of the development effort.  In fact, if the code isn’t expressed in the Ubiquitous Language, I’m going back to my developers with a refactoring tool and a set of instructions to rename appropriately.  By-the-way, code = application code + test code.

To me, using the UL everywhere is every bit as important as test coverage (with depth, before I get the ‘coverage isn’t everything’ comments).

Behavior-Driven Development (BDD) is nearly orthogonal to Test-Driven Development too.  If you are doing TDD properly, it largely renders BDD unnecessary — at least if you do TDD the way I do.  And the tests are expressed in the UL as well. 

BDD is an executable specification technique.  If one is disciplined with TDD you are writing executable acceptance criteria and requirements.  They are similar, but different.

In either case, they should use the Ubiquitous Language.

Can We Value Tools? Maybe..

We are in a transition here at the largest mortgage lender in the country, hereinafter CW – maybe you’ve heard about it 😉

That said, my boss is an avid supporter of agile techniques, as am I (ThoughtWorks to CW).  We are evaluating a couple of small, inexpensive options.  Those being (so far) ThoughtWorks’ Mingle (free 5 user license) and VersionOne (Community Edition). We currently have 3-4 interested parties in house so this level of license is okay for now.

If this catches on then we will need a significant number of licenses.

I am by nature and training leery of big management tools being called agile.  Remember the clause in the manifesto, “Individuals and Interaction over processes and tools“. 

I’m more comfortable with a story board, and an Excel spreadsheet for reports than a comprehensive software package.  That said, we are in a large mortgage company that’s been doing waterfall for years.  I have a feeling that seeing a tool such as VersionOne somehow legitimizes ‘Agile Development’ to those who aren’t ‘in the know’ already, so I’m willing to be flexible (or, as the title suggests, pragmatic).

Mingle has a cool feature that scans the comments in a Subversion repository and links Story Card references to the Revisions History. If you apply your comments properly, you can see which revisions affect the story cards. Nice.

Does VersionOne have an equivalent feature?

Domain-Driven Design — Practical to Tactical

..and dare I say, Strategic?

 On the Domain-Driven Design group (http://tech.groups.yahoo.com/group/domaindrivendesign/), Bil Simser lays out a common question:

“It’s all great to talk about repositories and aggregates and context boundaries, oh my, but the real challenge is making that connection from theory to practice. Then taking it even further (because presentation style and book samples can only go so far) how to you apply it to real-world apps?”

A fair question, and one that in recent weeks has been much on my mind (apologies to the late Graham Chapman).

The key to transitioning from book knowledge to practice, is perseverance.   Perseverance and cover, if you work in a big IT/development organization.  It’s absolutely required to have executive sponsorship in order to move the organization in any particular way.  It’s especially important to allow you to move towards any methodology that hints of ‘Agile’ as DDD does.

My approach (to move developers) has been to recommend “The Pragmatic Programmer: From Journeyman to Master” (ISBN: 020161622X).  This one book changed my programming life.  It led to my joining ThoughtWorks and it led to my current position trying to bring agile sensibilities to a large, developmentally conservative department in Southern California.  There are several other technique books that are well written and great for developers.

As for the management (development and business), it really requires patient conversation and the gentle application of the five ‘whys’.   There are also some great books available in this snack bracket.  Craig Larman’s “Agile and Iterative Development: A Manager’s Guide (ISBN-10: 0131111558) is a good one, but there are others. 

Eventually, if it’s possible, the management will hear what you are saying and sponsor a trial.  You need to select your team carefully.  As I’ve written before, occasionally there will be saboteurs assigned (wittingly or unwittingly).  You really need to be disciplined, and perform the project ‘by-the-numbers’.

One key strategy I’ve found is to get to the users directly.  You can start having conversations with them along the lines of “What bothers you about your job, day to day?” or “What one feature would make your work life easier?”  These conversations break the ice, and if you’re doing it carefully, they will start talking to their management about things they like/hate about the applications they use.

I don’t use subterfuge though.  I say what I’m after and why.  I’ve found that the most important thing that management should consider is business value delivered, and because of that, I’m interested in the business reasons for the users’ actions.  Very often, they are performing functions that have nothing to do with delivering value to the business.  They are forced to do some arcane tasks to support an inflexible legacy application, or surprisingly, because no one asked what they were really trying to do, and why.  Developers (usually under the gun to do something quickly) simply made assumptions about the technical way to accomplish what the user asked for, probably filtered through one or more intermediaries. 

Ever heard of the childrens’ game “Telephone”?  It’s not a surprise that users don’t get what they want most of the time; but I digress.

Once you have access to the users, get as much of their time as you can.  Ask all the questions that can help you start to build that “Ubiquitous Language”.  Iterate over the language and the concepts with a tight feedback loop, and you’re on your way.

If you’re having trouble getting to this stage, perhaps you need an agile coach to help educate the apprpriate people.  I know a few really good ones.  Drop a line if you need a referral.

Seriously.

A Whole New Thing..

So I’ve taken a job in sunny Southern California with a company that I originally encountered about two years ago.  I had a great time with the main folks there, doing some mentoring and coaching in the agile methods.

Joining a company is NOT the same as consulting.  There are some frustrations; I was 1 of 800, now I’m 1 of 60,000+.  There are bureaucratic issues, political issues, etc.  I see things that are obvious to me but I can’t say them out loud (yet) and there are some things that are not obvious, but will need to be addressed.

My family is staying in the South for a while, so I may increase my posting frequency as I sort through the day-to-day of being a salaried company man, instead of a salaried agitator from a consultancy.

Let’s hope these ruminations are useful to us both.

Thanks for looking in..

.NET vs Ruby on Rails.. Overdue

So, many months ago I said we were doing a Proof of Concept (PoC) to see what sort of  productivity gains could be had using Rails vs. ASP.NET.

We implemented 3 user stories around logging in to a site and accessing a web service for user privilege information.  The PoC showed that there could be as much as 2.5 times the productivity using Rails in that simple example.

Over the course of actual development, it looks as though 10-20% improvement is more likely as one gets to the actual complexities of a modern web site.  It appears that Ruby encourages more refactoring and refitting as the code base grows and the team learns new patterns and stretches old ones.

As Rails matures, practices around it coalesce, and developers gain experience, I’m sure that real productivity will improve but at the moment, it ain’t no silver bullet.