View Sidebar

Archive for February, 2010

Yuck, I Got Data on my Hands

Yuck, I Got Data on my Hands

Once upon a time I wrote a computer program that did not require data.  It was called helloworld.exe and it was awesome.  It was also a wee bit useless.  The essence of useful software is taking input, doing something with it, and spitting something out.

There is lots of data out there.  My first professional gig after university was consuming text files dumped out by a COBOL app.  (They were delicious.)  You can parse XML, munge HTML, manipulate images, or decipher networking protocols.  But despite the variety of possible data sources, we end up sticking a lot of our data in relational database systems.

And then we write a lot of code for moving data in and out of the database.  And then we complain about how much data access code we have to write and maintain.  In Survival Skills for Developers, the first item for your basic survival pack is a data access toolkit (homegrown or open source or commercial).  The reason is not to be trendy or to sound up-to-date at developer conferences.  The point is to make better use of your time by relying on frameworks and libraries to do some of the heavy lifting for you.

Do Not Be Afraid

In the .NET programming world, ADO.NET is the underlying data access technology.  Many data access patterns and frameworks have been built on top of ADO.NET and yet scores of developers still write ADO.NET data access code the way they learned nearly a decade ago.  What a waste.

There is no reason to be afraid of modern data access techniques.  You do not have to rewrite your existing codebase.  .NET programmers can still use classic ADO.NET when it’s expedient.  You can mix and match tools and frameworks.  I have a small project that I’m transitioning from SubSonic (used during prototyping) to NHibernate.  Currently the code features a mix of SubSonic and NHibernate as I transition and it works just fine.

Frankly I think developers who refuse to explore alternatives are being irresponsible.  I’m not holding up any specific frameworks, toolkits, or approaches as best.  I’m simply saying that a failure to be well informed about options is intellectual laziness and we cheat our employers, clients, and stakeholders when we insist on writing everything by hand every time we need a record from the database.  We would question the judgment of a house builder relying on all manual tools, yet we mindlessly churn out the same data access code over and over and over again.

Stop Freaking Out About Inconsequential Performance

A mental roadblock for many folks is a worry about performance going down the toilet by relying on something like an ORM (object-relational mapping) tool.  If there is one serious mental shortcoming amongst programmers, it is our obsessive need to prematurely optimize everything even when there are ample computing resources to deal with our less-than-completely-efficient code.

Stop freaking out.

The JVM and the .NET runtime have proven that letting the computer handle some tedious work is totally worth it even if it’s less efficient than what you could (theoretically) write by hand.

CPU time is cheap.  Programmer time ain’t.  Spend time optimizing code only when it becomes necessary.

Do You Really Need a Database?

Simple lists of objects can be similar to tables of records.  The relationships between objects can be similar to the relationships between tables.  Nevertheless, there is a mismatch between object oriented programming and relational data (perhaps you’ve heard of the object-relational impedance mismatch).  Depending on your situation, you might not really need a relational database.  Check out some of the technologies in the NoSQL side of the world for alternatives to relational database storage engines.

What’s Available?

There are a lot of options out there for working with relational data.  Check out Barry Gervin’s article All I Wanted was My Data for some options.

What are your suggestions?  Drop them in the comment box.

February 24, 2010Comments are DisabledRead More
Domain Distortion

Domain Distortion

50216300_6c80682292_m Remember the game Telephone?  You whisper something in your neighbour’s ear, he whispers to his neighbour, she whispers to her neighbour, and so on all the way down the line.  The last person announces the message she heard and everyone laughs when they discover how mangled the final message is from the original.

An eerily similar phenomenon occurs in software development as requirements flow from domain experts to business analysts to system designers (architects) to developers.  Each communication channel in the process introduces new opportunities for information to be distorted.

I was digging through some old papers this week and found a sketch I made nearly a decade ago on the back of a Dilbert comic.  I’ve prettied it up here as a reminder to all of us to always be watchful for potential distortions as requirements flow from users and stakeholders to the people who build and test the software:


The importance of good communication in the software development process cannot be overstated.

If you are passionate about building software and want to thrive in your career, hone your communication skills.  Ask questions.  Write things down.  Repeat back what you’ve heard to validate it.  Ask more questions if things don’t make sense.  Communicate, communicate, communicate.

In 2009 I had lunch with an executive with a lot of experience managing developers.  When I asked for his biggest piece of advice for professional developers outside of learning technology, his answer was to learn to communicate effectively.


(Image credits: malias,, Salvatore Vuono, Danilo Rizzuti, Maggie Smith)

February 17, 2010Comments are DisabledRead More
Survival Skills for Developers

Survival Skills for Developers

3822866409_c113336eb3_m Let’s suspend reality for a moment and pretend you’re heading out into the woods this weekend. Set aside the fact that you are a software developer and have no business tromping around out in the wilderness.  You’re going to want to take along a few basics:  maybe some matches, a tent or shelter of some sort, a bit of food, and some dry clothes.  To survive you need the right tools / supplies and the skills to use them.

(I bet you see where I’m going with this.)

Back at the office, where your mouse and keyboard fret over your safe return, there awaits an entirely different survival scenario:

Survivor: Cubicle.  Outcode.  Outbuild.  Outlast.

Surviving as a software developer is more than stringing together some lines of code that read and write from a database.  Sure, those are basic skills.  To survive in the woods you obviously need to walk and breathe, but you also need to start a fire and build a shelter.

The following 8 items form a basic survival pack that can get you through most modern software development forests:

1. Data access toolkit (homegrown or open source or commercial)

If reading and writing from a database is a basic skill like walking, then a good data access framework is your walking stick.  I’m a .NET guy so I’ve worked with NHibernate, SubSonic, Entity Framework, and a couple homegrown solutions.  An ORM tool is not necessary.  Use the core ADO.NET classes directly if that works for you.  What you absolutely should avoid is writing all your data access code from scratch every time you need to work with data.

Personally I recommend you learn the basics of several modern data access toolkits and learn about some of the code generation solutions available on the market so you can make informed choices in your projects.

2. Regex

OK, let’s crank the controversy meter up one notch.  Regular expressions are like waterproof matches.  Sure you don’t need them, but they can make things a heck of a lot easier in some circumstances.  When it comes to text processing, there are lots of approaches that can work, but a regular expression can turn an arduous coding exercise into a single line of code.

If you use Visual Studio, I highly recommend you explore the regular expression syntax available for the Find dialog box.  It is amazingly useful once you learn the wonky syntax.

3. Unit testing

Let’s try one more notch higher on the controversy meter.  You must be able to write and execute unit tests for the language and platform you are developing on.  Unit testing is like a compass.  You don’t always need it and you won’t need it constantly, but it can be priceless in many situations.

Unit testing has its fair share of zealous proponents and vehement haters.  I like to think I have a nice moderate middle-of-the-road perspective.  Unit testing is incredibly important in some circumstances.  I am not an adherent to the Church of Test Driven Development.  I do not usually attempt to achieve complete code coverage in unit tests.  But without the ability to write and execute unit tests, you risk wasting precious time walking in circles as you test the same things over and over again manually.

4. Basic printing, reporting, and charting

Not every system has to print or produce charts and reports, but lots do.  Even web site developers have to consider how their pages will look when printed and perhaps generate PDF documents for printing.  Regardless of whether you write desktop apps, web apps, or middleware, you should be able to print and generate some basic reports.

If you are a .NET developer like me, check out some of the vendor solutions like Telerik Reporting, ActiveReports, LogiReport, and XtraReports.

Printing, reporting, and charting are about user needs.  Users need to print invoices and put charts of projected vs. actuals into the shareholder report.  Sometimes you need paper to get your business done.

5. Internet: sending email and downloading HTTP content

Back in the last century, we invented this thing called the Internet.  It’s nice.  It lets software programs on different systems communicate with each other.  And the Internet has this way of creating new opportunities so you never know when your software will have to become Internet-aware.

Sending email and accessing HTTP content are pretty easy tasks to accomplish in most languages given the plethora of libraries available (e.g., classes built into the .NET framework).  And even if your current coding project does not require Internet access, you might see new opportunities once you’ve learned how easy it is.  For example, you might decide to automatically email crash reports to your help desk.

If you pass other hikers in the woods, you might not need to talk to them, but then again, maybe you will.  Better to be prepared.

6. HTML and basic CSS

Regardless of the type of software you build, I firmly believe you should know a little bit about HTML, XHTML, and CSS (and some XML would be good, too).  If you don’t build web applications, you don’t need to master these technologies, but a grasp on the basics will come in handy.  Examples:

  • creating HTML formatted emails to send to users
  • updating a development team web site
  • customizing a product wiki
  • writing end user documentation
  • updating the company web site (happens in small companies)
  • setting up a blog for the marketing department

(X)HTML + CSS is the lingua franca of the web so just learn it already – no fancy survival metaphor required.

7. Scripting or command line development

Many software components need to work with standard input / output streams or simply do not require a graphical user interface.  If you only know how to build Windows or web applications, you are severely limiting your ability to efficiently handle the many small tasks that often accompanying development work like parsing a file or deploying software updates.

If you work in Windows, learn to use the command line along with some basic VBScript and batch (.bat) file "programming."

If you work in a Linux / UNIX environment, you probably already know the things you need to learn.  In case you don’t, I suggest digging into some shell scripting, sed, awk, and grep after you’ve mastered the command line fundamentals.

If you work in an OS X environment, you just need Photoshop.  No, I’m kidding.  Learn the UNIX utilities.

Decent command line and scripting skills are your ferrocerium – more hardcore than a match and much more durable.

8. Services, daemons, and cron

Sometimes software just needs to run on its own.  If you’re a Windows developer, learn how to build a Windows Service and how to run programs with the Task Scheduler.  If you’re in a Linux or UNIX environment, learn how to write a daemon and schedule cron jobs.  Even if you don’t specifically need a Windows Service or a daemon process in the foreseeable future, understanding the concepts will make you a better programmer.  Unlike desktop apps that can be easily restarted or web apps in which code executes in short bursts, daemons and services are long-running processes that require careful creation.

Thinking about responsible resource management in a long-running process will make you a better programmer, just like building a campfire in the wilderness from collected wood will make you that much better at building your next campground sing-along-and-make-some-s’mores campfire.

Mmmm…  s’mores…

Let the Comments Flow

Agree?  Disagree?  Think I missed something?  Leave a comment and let’s discuss.


(photo credits: Alaskan Dude)

February 10, 20103 commentsRead More
Blogging Refocus and Page Size

Blogging Refocus and Page Size

As part of my planning for 2010, I decided to write new vision statements for some of my web sites and put myself back on a content creation schedule.

This site ( will now be updated every Wednesday, so stay tuned via the RSS feed or subscribe by email.

New Design

I decided to simplify the design, opting for a clean white background, more whitespace, and less colour.  Hope you find it easier to read.


Page Size Optimization

One of my goals in redesigning the site was to reduce the page size a bit.  I like to include images in my posts so going overboard on optimization wasn’t going to make total load time much better, but small improvements are always welcome.

Thanks, YSlow

To measure my optimization progress, I used YSlow.  YSlow is a Firefox add-on that integrates with Firebug.  Firebug is a web development add-on for Firefox.  I highly recommend you hook yourself up with the Firefox + Firebug + YSlow triad of awesome if you do any web development.

My quest for page size reduction began with cleaning out the old WordPress theme.  For example, by removing Recent Posts and Recent Comments from the footer, I reduced the amount of HTML and gained the happy side effect of reducing the number of database hits required to build a page. serves its HTML content compressed.  At the time of the design switch, the old theme was sending down 13.7K of compressed HTML for the home page.  The theme changes reduced that to 12.4K.  Dropping the number of posts shown on the home page from 10 to 5 brought things down to 7.7K.  That’s 6K less than the original (not including images from posts no longer visible on the home page).

I did a clean up of the CSS files for the site as well.  The site has a CSS file for web browsers and a CSS file for printing.  With some editing and some whitespace removal, the CSS files went from 4.6K down to 3.2K.  It was not a super thorough scrub of the CSS, but I trimmed another 1.4K.

The previous theme referenced nine images via the CSS file.  I dumped all those extraneous little images, which knocked  29.8K off the page size.  I then introduced a new logo that added back 2K.  That left a 27.8K reduction, not to mention 8 fewer HTTP requests.

I also dropped a banner ad that removed 0.4K of JavaScript and an additional HTTP request (plus another subsequent HTTP request to download the ad itself).

Net Gain (Well, Net Loss Really…)

The total reduction in page size came out to 35.6K as well as 9 fewer HTTP requests (not counting the request for the banner ad).  For a broadband Internet user, the optimizations should make the home page load 1/10 to 1/4 of a second faster than before.  Not a massive savings but a rewarding exercise as I searched for ways to reduce the bulk of the home page.

Everyone’s mileage will vary but please leave a comment about your own page optimization experiences.

Previous YSlow Stats


New YSlow Stats


February 3, 20104 commentsRead More