I’ve been around long enough–and seen enough disastrous computer crashes–that I have a pretty careful backup strategy. I’ve been the nagging voice of nagginess when my friends have lost data due to hardware failures.

“You should ALWAYS make backups of your data. You never know when you computer will take a permanent siesta.”

I commit all of my source code to a subversion repository on my local machine, and I regularly take snapshots of that development machine, which I keep on an external hard drive. In case of a failure I’d suffer two or three days of lost data. A week, tops.

Lately, my computer’s primary hard drive has been making funny noises, like a tiny helicopter flying into a narrow fjord, deep within my laptop. So I’ve been extra diligent about backups.

On Friday, it finally died.

During the boot-up process, it went into a ten-minute chkdisk loop before ultimately succumbing to a horrible BSOD.

After that… nada.

No problem. I bought a new drive and prepared to transfer everything over. After installing the operating system onto the new drive, I connected the backup drive (in its own external USB enclosure) and listened, horrified, as it made a few pathetic clicks and whirring noises.

Dead.

Huh. That’s weird.

This drive had been chugging along without complaint just a few hours beforehand, and suddenly, it’s a complete brick. The operating system doesn’t recognize it at all. It may as well be a can of sardines.

I fiddled with the connections, finally taking apart the enclosure, swapping the cables. Still nothing. The drive is definitely kaput.

Keep in mind, this is not just my source code. This is my entire life. Software projects, art projects, letters, stories, email archives, proposals, invoices, tax documents, from the last seven or eight years.

I felt like my house had burned down.

Panicked now, I connected the old laptop disk, to see what I could salvage.

As it turns out, there was still a lot there.

Even though the operating system wouldn’t boot, and some entire directory hierarchies had vanished in a puff of corruption, I was able to salvage nearly all of my personal documents (in fact, as far as I can tell, all of my photos and documents survived).

But the source code is riddled with holes. Entire projects are just completely gone. (And about a third of my MP3 collection. The horror!)

The application analytics project–which I’ve been working on since June–is particularly decimated. From a total of probably 30,000 lines of code, I’d say about 25% has vanished. The entire server project is nowhere to be found. (The entire embeddable library project and about 90% of the reporting client GUI survived.)

I was able to recover an old, crusty snapshot of the server code (from September) by manually slogging around in the raw subversion data files.

Also completely gone: a set of ActionScript support libraries, with data structures and algorithms for the charts & graphs in the reporting client. Luckily, at the end of December, I started the process of moving my source code–starting with this particular library–to a repository on a remotely hosted (and independently managed) server.

Thank God for small miracles, right? At least if I could recover that code, I’d save myself the trouble of reimplementing about three or four thousand lines of code.

So I tried doing a check-out, but guess what!

There’s nothing there.

Browsing remotely with Tortoise SVN, the server happily responds that the repository is empty. And the web SVN interface says that I don’t have access to the repository. Which is odd… since it also lists a record of my most recent check-ins.

Wouldn’t it be funny if that repository had somehow evaporated?

Is someone playing a joke on me? Am I being punk’d?

I’m going to send my (two!!) failed hard drives to a data recovery company tomorrow. Maybe they can do better than me at recovering some of this lost files. Yikes.

Anybody want to make a small wager that the data recovery facility will burn to the ground the moment my FedEx package arrives?

In the spirit of me actually spending the bulk of my time writing code, I’m going to dispense with the meandering long-winded stories and just show a few details of the upcoming implementation of the Desktop Analytics platform.

Within the next couple of months, I’m going to release the full product suite. The server is written in Java and the reporting GUI will be deployed as an Adobe AIR application, but those aren’t what I want to talk about today. Today I just want to provide a sneak-peek into the embeddable library that developers will link into their own applications to enable statistical reporting.

Read the rest of this entry »

The wooden doors of the saloon have long since fallen off their hinges.

A sun-bleached steer skull gleams under the hot desert sky.

And tumbleweed creeps across the road, bumping against the old hitching post.

It’s been a ghost town around here lately, since I posted my last message in June, and the lost signs of life have not gone unnoticed a few of you out there.

Three or four times a month, I get an email from someone curious about my Desktop Analytics project, who maybe saw the GUI mockup, and wondered whether I was still around and still actively working on an implementation.

As it turns out, I’m still here. And I’m still working hard.

In fact, the ongoing emails from interested bystanders have really kept me motivated and have reaffirmed my confidence in the strength of the concept. As my way of saying thank you to the people who have sent me their questions, I’ll take a little time tonight to give you a long-overdue status report. Then, sometime within the next few days, I’ll post a sneak peak into the architecture I’m developing and some of the cool features that I hope to unleash on my users early next year.

I’m thinking…mid to late Q1…

First, a quick refresher, for those of you who might not have been around when I originally floated this idea.

I’m building an application analytics platform, analogous in many ways to Google Analytics or Omniture. But those solutions focus on analyzing web traffic to answer questions like “which search engine keywords provide me with the most traffic?” and “which of my content resources have the highest click-conversion ratio?” Those types of metrics were invented my the advertising industry for messuring consumer responses to ad campaigns. Long before Google emerged from the labs at Stanford, ad executives asked Nielsen analysts for their conversion numbers.

The web analytics software on the market today still reflects that mentality: the internet is a medium for delivering content. Some of that content attracts a lot of eyeballs. Some of those eyeballs tell their mouse-fingers to clicky on an ad. And some of ad-clickers buy hilarious cat calendars from Amazon.com.

Web analytics is all about maximizing throughput of the content/eyeball/mouse-finger/cat-calendar pipeline.

Developers of applications, rather than publishers of web content or cat calendars, get short shrift in the whole equation. Sure, we can get detailed statistics about which types of web surfers are more likely to download our products. But when the last byte finishes downloading, our visibility into that user’s behavior evaporates.

I want an analytics package that tells me how users actually interact with my software.

How many people are using that new spellcheck feature I spent so much time developing? How often do my users fire up the application? Every day? Once a week? Do they use the software all day long, or just for a few minutes in the morning? How many menu clicks does it take to find certain features? Are people getting lost in my user interface? Do they invoke the HELP system? Do the “Premium Edition” users actually use any of the premium features, or do their behavior patterns look just like the “Free Edition” users? How many people are still using a legacy version, and which features should I improve, to motivate an upgrade decision among those users?

Answering questions like that will help me develop better software.

And I like to think that “better software” is almost as important as “good keywords” in driving new revenue.

Anyhoo…I hope I’ve gotten you a little sweaty and breathless…

Now: status report. Tomorrow: architecture details.

The software comes in three pieces: an embeddable library, a server, and a reporting client.

The embeddable library is a tiny piece of code that developers will link into their own “host” applications. The code is written in a snazzy little programming language called D, which compiles to native machine code, providing C linkage on Windows and on x86 Linux and OSX. No matter what language or platform you’ve used to write your application, as long as it provides some mechanism for linking with C libraries, you’ll be able to embed the analytics library in your code.

As of today, the Windows version of the embeddable library is about 95% complete (the OSX and Linux versions are about 75% done).

The server is implemented in Java, so you can run it pretty much anywhere (I personally recommend a “Mr Coffee” deployment environment). The server is about 85% finished, and can handle about 100 requests per second on my dinky little 2-year old laptop. It doesn’t use Tomcat or JBoss, and it doesn’t depend on a gazillion application frameworks. It’s a standalone HTTP server, talking to a MySQL backend, and you can get it up and running within minutes (hint: no XML configuration).

Finally, there’s the reporting client, with all the sexy charts and graphs. I’m only about 10% done here, after making a bold decision to switch my development environment. I started out working in C# with WPF/XAML, but I recently switched to Flash/Flex/AIR (for reasons that I’ll describe in more detail later) and now I’m playing catch-up.

That’s all for now :o)

I’ll be back soon with more details. But right now, I just wanted to check in with you guys, let you know that the project is still rolling forward, and that I’m anticipating a Q109 release.

Thanks, everyone, for your feedback about the Desktop Analytics software.

I’ve made the following platform choices:

Initially, the embeddable client library will strictly provide a C interface for Windows. In future versions, I’ll also provide wrappers in popular high-level languages, starting with C# and Java. At some point, the library will also be available for Linux and Mac OS X.

The server will be developed as a standalone Java server (i.e., without requiring Tomcat or JBoss or anything like that) with a MySQL backend. Future versions will enable more database choices (probably starting with SQL Server), though I won’t develop a non-Java version.

In a subsequent version, I’ll also release a “reflector” module in PHP/MySQL. The reflector will be a dumb server component that listens for events from the embedded client library and stuffs them into a database. Periodically, it will connect to the main server and send a batch of data. The purpose of the reflector module will be for developers who don’t want their users to see an outgoing HTTP request to an unfamiliar domain, but who don’t want to actually host the main server themselves. At some point, the reflector will probably also serve as a load-balancer and/or a failover mechanism.

The reporting client, with all the fancy charts, will be developed in C# using the .Net 3.5 framework, for maximum WPF goodness.

Thanks again for your help in choosing platforms!

I know not everyone is interested in getting a truckload of business intelligence about the users of their software. But for those of you who think the Desktop Analytics project is pretty cool, tell me a little bit about your development platforms.

Do you use Java or .Net to develop your desktop software? Or are you maybe using C++ with MFC or WxWidgets? Anyone selling desktop software built with Ruby or Python?

Do you have a server? Is it Windows or Linux? Do you have your own server, or do you use a VPS account? Or just a shared web server, with PHP/MySQL? Do you have permission to install your own software into your server environment? Are you already running a Java app server? Tomcat, or something else?

Whatever platforms people seem interested in are the ones I’ll focus on in my development, so express you preferences here!

I’d like to solicit some feedback on a GUI mockup I’ve put together, for the deskop analytics project (click to enlarge):

Without any additional commentary on my part, what do you think of that design? I’ll answer questions in the comments section.

First of all, let’s start by assuming that you love hilarious tee-shirts.

Something, perhaps, to express your knowledge of contemporary video-game pop culture, while simultaneously making an ironic commentary about being a complete social misfit. Not just a run-of-the-mill computer nerd. But maybe a former a member of an actual high school band. Brass section.

Maybe something like this…

Read the rest of this entry »

A handful of steadfast readers out there may remember that, a few years back, I challenged myself with developing a list of 30 unique business ideas in 30 days, with the goal of launching a product and building a business.

During that time, I did a lot of blogging (sometimes as late as 3am, to write a blog entry while on vacation, after everyone else had fallen asleep). The whole endeavor really got me into a mindset of wide-open possibilities, to the point where it was hard to stop thinking of new ideas and actually focus on just one.

(I’m going to make a bold statement: The people who complain about a lack of software ideas have never really spent a significant chunk of time going through a methodical brainstorming process.)

Now, after two years, I’m finally making good on the implied promise of that brainstorming month.

At the end of June, I’m going to release a new product.

Later this afternoon, I’m going to write my first line of code.

Read the rest of this entry »

So, I finally bit the bullet and started my own company. I’ve been a free-agent for about three weeks now, and I love it.

Hooray for me!!

The new company is called Wry Research, and at least at first, I’m mostly just doing AI research consulting for a couple of clients. I’m trying to also fit in some original product development, when I have spare time. But I don’t have a whole hell of a lot of spare time, so I’m mostly just consulting right now.

But life is very good, and I’m enjoying the freedom and flexibility of being a solo software developer.

In the near future, I’ll talk about some of the work I’m doing as a consultant. I’ll also talk a little bit about a really cool research project that I’m doing, in conjunction with the Columbia Medical School Neurovascular Research Lab. Depending on the success of the research, it could lead me to developing a very cool product/service for the medical research community.

I’ll keep ya posted :)

Last week, the Eternity II puzzle (which I’ve talked about here and here) went on sale. I can finally wipe up the drool-puddle-of-anticipation that’s been forming on my desk and get to work cracking this nut.

Unfortunately for those of us in the United States, the puzzle is not available here, and there are very few online retailers willing to ship the puzzle to the USA.

But I did track down the puzzle at Hamley’s and placed my order. Their online shopping cart is a little bit crappy (it got confused verifying my non-European billing address), so I had to call them on the phone ( (+44) 800 2802 444) and a very friendly girl with an Irish accent took my order.

I placed the order at noon last Wednesday, and the package was delivered (in Boston, MA) by about 2:00 pm on Friday.

Holy crap.

At first, I thought they must have used an American order-fulfillment location, but the package had international shipping stickers and a return address in the UK, so it seems to be an actual international delivery. Fulfilled in just two days. I’m very very impressed. I had no idea that it was even possible to ship something overseas so quickly.

The puzzle cost £35, plus another £25 for shipping. With today’s lousy exchange rates and the weak dollar, that’s a total of $120. Unfortunately, when I win the $2 million prize, that means I’ll really only be netting $119,999,880.

Bummer.

Anyhow, if any of you American algorithm enthusiasts have been sulking because you can’t get Eternity II here, sulk no more!!