Ran across an article about software that’s improving intelligence analysis and has already saved lives. Although interesting for many reasons I think it highlights the significance of a few trends in software development:
- Building software is an iterative process. To get it right you need frequent input from the end-users. This is a key distinction between agile methods and classic waterfall design. According to the article, “Every other week for about two years, the engineers returned to Washington with a revised product, based on analysts’ requests.”
- Rapid application development. You can’t revise a product every 2 weeks unless you’ve got tools capable of supporting that level of productivity. I would be very surprised if the company’s flagship product didn’t have large components written in one of the more productive languages (e.g., Java, C#, Visual Basic, etc…).
- Software Engineers have to be able to work with end-users and subject matter experts to create better software.
- The user interface is important. A key feature of the software is something that anyone who uses google takes for granted: the ability to search several databases conveniently.
- Lightweight categorization (aka tagging) can produce better results than up-front, deep hierarchies. Tagging is easy for users to do (so they are more likely to do it) and flexibly accommodates changing categorization needs. Instead of having a team of experts build a deep semantic hierarchy that’s out of date within a week tagging lends itself to a bottom up approach. All the end user has to do is tag the information using whatever terminology makes sense to them. Software is then used to identify clusters in the data.
I’m sure there are other lessons to be learned from the success of Palantir. It’s great to see software developed using modern methodologies put to such positive use in such a short time.