Friday, June 20, 2008

Postmortem

So, here's the collection of lessons learned. I'm going to frame these as advice, but everyone should remember Buchheit's Law: "Advice = Limited Life Experience + Overgeneralization". This is one startup, and not even a successful one at that. There are many ways to succeed, and even more ways to fail, and so there will likely be exceptions to every single one of these rules.

If your idea starts with "We're building a platform to..." and you don't have a billion dollars in capital, find a new idea. Now.

This seems to be a really common mistake among first-time founders, particularly those from a technology background. My last employer made it, and has been working on their platform for the last 7 years with little to show for it (nearly all their revenue comes from a single application on their platform). Early in their development, they had decided that "Well, the only thing all our use cases have in common is that they all involve writing code, so we'll provide a way to write code and handle the tricky business of parallel processing ourselves." The didn't realize that by making code their user interface, they made Java/C++/Python their competition, and those languages have had literally thousands of man-years of development behind them.

When I started GameClay, I vowed I wouldn't make the same mistake, and so Mike and I decided there would be no code in GameClay. Everything would be done through a GUI. I didn't realize that the problem wasn't code, it was the lack of focus that came with allowing that sort of flexibility. So as we were building the GameClay UI, I kept getting dragged into all these "Well, we really need this feature to let people do what they want, but it interacts in weird ways with that feature, so we need to rethink the whole thing." The search space of possible designs was just too huge to tackle.

Solve a problem, not a class of problems. Only after you've thoroughly solved that problem can you work on expanding it. Really, it's okay to not have your product do everything, as long as it does something well.

Interestingly, when you think of many common platforms in use today, they started out that way. Linux started as a terminal emulator. The web started as a way to share scientific documents. Rails started as a collection of code extracted from BaseCamp. Django started as a collection of code extracted from the Lawrence Journal-World. PHP started as a way for Rasmus Lerdorf to manage his personal home page.

The exceptions all seem to be giant corporate projects with money to burn - hence the "and you don't have a billion dollars in capital" caveat in the heading. UNIX at Bell Labs, the Macintosh at Apple, Windows and then .NET at Microsoft, and Java at Sun. Big corporations can afford to spend a billion dollars and wait 10 years, though, while startups usually can only afford a few tens of thousands and about 6 months. You have about zero chance of building and getting people to adopt a platform in 6 months.

It's a marathon, but it's a marathon made of sprints

I didn't realize how slowly I was working until I worked with a YCombinator company. I was making about 6-8 commits/day; they setup a new Subversion repository 2 days ago and it's already on revision 100.

I was going to title this "Beware the halfhearted effort", but I think that'll be misinterpreted. When people hear "halfhearted effort", they think "Oh, I'll just work harder." But there's a limit to how hard you can push yourself. Programming is a mental activity, and the brain has its own timescale. If you keep thinking to yourself "Must work faster, must work faster," you'll just end up slowing yourself down.

Instead, I think that it's absolutely essential to set things up so you get that rush of accomplishment as you finish things. With GameClay, a lot of my work was basically "Oh, I need to finish this UI widget, and then there's that bug in auto-updating the game when properties change, and...shit, it's not working." Even when I finished something, I just wasn't that impressed with what I'd done, and it was sorta ho-hum to see it working. I'd feel drained and spent after coding 2 hours instead of energized.

A lot of this comes down to picking a problem that's a.) worth addressing and b.) doesn't require a lot of support code to address it. Early in GameClay's development, I decided that every action the game developer took would instantly update the running game, so they could see their changes immediately. I thought this was absolutely essential for usability, and it probably was. But it meant that every single UI widget, whether 3rd-party or written by us, had to be wrapped with functionality to capture all of its events, update the game code, compile it, and then reinstall the new version into the running game loop. And this all had to be done in the browser, using nothing but AJAX calls. This was hard, and a lot of my time was spent dealing with problems related to this.

Initial conditions matter. A lot.

When I started, I'd read everywhere that execution was the important thing and the initial idea wasn't all that important. I still think that's true, but like most other advice you read on the Internet, it's not the full story.

The initial idea does change, and it's almost certainly wrong. The thing is, the initial idea determines how the initial idea will change, which is crucial to all the execution that follows it. And the devil is often in the details: little decisions made early on can crucially impact big decisions made later.

I gave an example above, with the decision to have all updates reflected immediately in the game. Another choice was the decision to start with arcade games, which was done very early on (even before I joined, I think) without much thought at all. It was just the first thing that came to mind when we thought of casual gaming, so we said "Let's start with that, and then expand into other archetypes when we have more experience." We didn't realize that a.) just getting arcade games customizable in a compelling way would be a huge undertaking and b.) we'd ditch the archetypes concept entirely, and so our initial archetype choice became the basis for our product. Meanwhile, most of the growth in the casual gaming industry has been in puzzle & word games, so we were positioned wrong from the start.

A third example is the choice to go into gaming at all. When we started this, a year and a half out of college, that was what we were familiar with and passionate about, and so we didn't really think much about other areas. But the gaming industry is incredibly overcrowded, and most areas other than game creation have too much competition to be worth entering. There are hundreds of Flash game hosting sites, you really don't want to be a game developer, and MochiAds has a lot of momentum in the game advertising space. So that's why this is the end of GameClay instead of a transition: if it were a less crowded and more lucrative space, I'd instead think of repositioning things to an overlooked niche.

All this is not suggesting that you overthink every possible decision at the beginning of the project and make sure you have the perfect idea. You'll never get started if you do that. But realize that your initial decisions become the base that you take for granted, and you usually never even think about them past that point. And there's a high, high probability that those decisions are wrong. It's worth reexamining them after 3-6 months to see if that's still what you want to do. (We actually did reexamine them, several times, and the answer was still yes, but that didn't make the decisions any less wrong.)

Developing in a vacuum never works.

If you look back at early blog posts here, you'll see that the idea hasn't really changed. That's because it hasn't been in front of many people. Oh sure, I demo regularly to my former cofounders, and Mike had been showing it around to people at his workplace and a couple potential investors. But it hasn't really had the sort of rigorous critique that you need to achieve product/market fit.

The reason for this is that I didn't want to repeat one of the mistakes of my last employer. They had products that they actually put before a customer and got feedback on - but then the customer's feedback was viewed through the lens of "What does this mean we have to include in our platform?" not "How can we solve the customer's problem in the simplest way possible?" This was a huge drag on development, since everything had to fit into the original vision.

I believe that if you have a strong vision, you should go for it. You may be right or wrong (you're probably wrong ;-)), but at least you find out and can approach subsequent problems with a clean slate. Don't waste 10 years trying to "work up" to your vision that was probably wrong anyway.

The flip side of this is that if your vision takes more than a year to execute (I'd actually say 3 months, since that seems to be the limit on your initial enthusiasm), assume it's just not possible in the current environment, and find a new problem to work on. You want to prove yourself wrong as soon as possible, not spin your wheels forever. Then you can start the hard work of getting to right quickly.

I should probably clarify this: this doesn't mean you should shy away from big goals. Only shy away from big goals where you have to solve lots of thorny problems before you can even put something useful in front of users and get their feedback. Invent the WWW, not Xanadu.

Beware the chicken and the egg.

A couple of the side-projects we launched had serious chicken-and-egg problems. Bootstrapacitor, for example, would only be useful if other entrepreneurs used it, yet it needed to be useful before people used it. Diffle and GameClay had minor chicken-and-egg problems, but we solved them in Diffle by finding lots of free-for-the-taking games on the web and uploading them ourselves. inAsphere.com, a startup I worked at way back in the first dot-com boom, also had a chicken-and-the-egg problem that prevented the site from really getting off the ground.

The easiest way to avoid chicken-and-egg problems is simply to have a product that is useful on its own. Del.icio.us, for example - it's just a bookmark manager that happens to be more useful as more people use it.

It's also possible to beat chicken-and-egg problems through sheer determination - after all, Reddit, YouTube, FaceBook, Snaptalent, and various other Web 2.0 companies have achieved it. The way to do it seems to be to make sure you're passionate enough about your own product to use it yourself (like submitting your own stories to Reddit, or how we all created 3-4 sockpuppets at inAsphere and had conversations with ourselves on the forums) and to get out there and put it in front of lots of other people. It's that last point where we failed: we just kinda built the product, launched it, and let it die.

It really helps to have a passionate, committed cofounder here. It's pretty difficult both to improve the product (which is critical if you want to show users that you really care enough to be in this for the long term) and to evangelize it.

Prototype any 3rd-party libraries that you'll be depending upon, before you base your product on them.

We did this with JavaScript libraries - we ran through quick prototypes with Prototype, Mootools, and YUI before finally settling on JQuery. And we haven't regretted the choice at all.

We didn't do this with Python web frameworks, and wasted a lot of time as a result. When I started, I picked web.py because it was the only one I could understand while working within the time constraints of my day job. Totally the wrong choice - there were a lot of things that web.py just didn't handle that we had to spend a lot of time implementing ourselves.

Then I went to Pylons, because I had this irrational aversion to Django. I think Pylons actually would've been okay - it's not a bad piece of software - but it was a mistake to make the decision based on a quick glance at the Django website. When I actually started using Django for the project, I found it was much more convenient than Pylons had been.

Another thing to keep in mind: by "prototype", I mean actually write code that exercises a lot of the features of the framework, don't just read the website. A serious look at the platform turns up many more issues than a superficial glance at a feature list. Get to know your software before you pick it.

Actually, one of the things I regret is not taking a deeper look at Ruby on Rails and ExtJS for this project. I did do a quick Rails prototype for my previous employer, so I'm not completely ignorant of it, but I wish I knew it better when deciding. But there is limited time for evaluating software, and so the non-Python solutions tended to get short shrift.

If you're doing anything other than building your project and getting users, it's premature.

This was actually something we did pretty well. We had help, of course: nobody would give us money, so we didn't have the resources to do anything else.

But when we did deviate from the product/user focus, it was a mistake. We nearly incorporated too early, back when diffle.com first launched; there was no reason to (nobody was uploading anything anyway), and the bookkeeping involved with being a corporation would've slowed us down significantly. I also spent two weeks or so load testing the server and doing some performance tweaks, which was completely unnecessary: we never hit more than about 5% CPU usage anyway. Some of the deployment tools we built never got used; wait till you have something to deploy before you start automating updates to it. Internationalization support was also a total waste of time.

Some things we did well on this front:

We didn't spend all that much time talking to investors. They probably wouldn't have invested anyway, but if they had it would've been a disaster. Our idea was flawed from the beginning; I don't think we could've pulled it off without putting 5-10 years into it. If we'd gotten money, it would simply have allowed us to waste other people's time in addition to our own.

We didn't worry about corporate trappings - titles, board of advisors, offices, furniture, etc. - at all. This let us focus on the product and keep our overhead low.

We didn't spend a lot of time on hypothetical design decisions. Basically, when one of us proposed a change or new feature, our design process was "Let's try it out and see how it looks." Oftentimes, the answer was "Terrible. Let's change it back," but we'd find that out in a day or two (or sometimes just a couple hours) instead of arguing over it forever. And many times, the process of looking at it suggested a new way of doing things that was better than both the old and the new.

We didn't spend lots of time trying to partner with big names to offer us credibility. Okay, we also failed, so it's hard to say whether this would've been helpful or not, but since we failed due to lack of a product, I suspect it'd just have made the failure more public and embarrassing. Also, big companies tend to be very conservative about lending their brand to support any sort of new startup; it can be very difficult to get them to sign on at all.

The product will take longer than you expect. Design for the long-term.

I need to put this together with the last point, because otherwise I know someone will misinterpret one or both.

Whenever I had a choice between doing things quick and doing things right, I chose to do things right. And in nearly all cases, this was the correct choice in hindsight. This seems to contradict the previous point, so I'll offer a possible clarification:

Do anything you possibly can to avoid bugs.

Bugfixing takes unbounded time; it's usually not possible to predict how long it'll take to track down and fix a given glitch. Moreover, fixing a bug in a dirty code base often introduces 2-3 other bugs that then have to be fixed, creating a nonterminating loop and stack overflow. I've worked in organizations where 90-95% of programmer time was spent tracking down bugs. Just think, they're moving 10-20x slower than they could be.

I should probably mention that this is heavily context-dependent, i.e. the reason why avoiding bugs was such a big win for us was because our product was so damn hard to build, and having a product that was that hard to build was a net loss. So maybe a company that gets the initial product idea right wouldn't benefit from solid development. Reddit, for example, is famous for just releasing their code and letting their users QA the product. But the Reddits of the world seem to be the exception - I'd guess that most people are working on products hard enough that they may actually screw them up, and their customers actually care when they do screw up.

People have an incentive not to crush your dreams. Take everything they say with a grain of salt.

When we did talk to investors, the response was almost universally "It sounds interesting, but I'd need to see a prototype before I invest". Almost nobody flat-out said "This idea sucks, and you'll never be able to complete it." (Paul Graham did, but even he couched it in language that was less blunt.)

Everybody in the startup world knows that big ideas come from unexpected places, and we've all heard the stories about the guy who told YouTube "Nobody would ever want to share videos online" or the guy who told Apple "Nobody would ever want to have a computer on their desk." They don't want to be that guy. So they'll encourage you, but say that it's not yet time for them to invest.

If you really want critical, honest feedback, you need to read between the lines. Concentrate on proving yourself wrong, not proving yourself right. It's easy to fall victim to confirmation bias, particularly if you're as passionate about your idea as an entrepreneur needs to be.

This of course presupposes that you want critical, unbiased feedback, and this is debatable. If I had an accurate assessment of my chances when I started, I would've stuck with my day job, and I'm so much better off now for having tried and failed.

I guess I'm trying to get across that you should assume that everyone you talk to is being nice, analyze their remarks critically to figure out what they're really saying, and then assume that every problem they raise can be solved. You want to prove yourself wrong, but then use that as a springboard for finding out what the right answer is instead of assuming there's no solution.

Know your limitations.

When I started GameClay, I was like "I'm a smart guy, I work really hard when I have a sense of ownership of the project, I've got a few years of work experience in fairly related areas, why not? I'm just as qualified as anyone else to start a startup."

It turns out that brilliance and dedication alone cannot create a market or solve an intractable problem. They can help you find out faster if your idea isn't viable, and they may allow you to reach markets that others pass up as too difficult, but there will be some markets that remain out of reach until the environment changes enough to bring them within reach of a dedicated, passionate team.

That doesn't mean you should shy away from working on hard problems, otherwise there'd be no Apple, or Sun, or Google. But even the founders of those companies knew their limitations and concentrated on the areas that they were experts in. Page & Brin, for example, knew they were experts in search and decided that Google would do that one thing well and not worry about anything else. Wozniak knew he was a damn good engineer and didn't join Apple until it was clear he would only be an engineer. Sun...well, the founders of Sun were each experts in their various domains, and it happened that when you put them together, they basically had all bases covered.

My problem with GameClay was that there were 3 really hard problems (game design, end user programming, and AJAX-based webapps) and I was an expert in maybe 1 1/4 of them. Next time I start a startup, I'll perhaps approach it with a little more humility, and pick something that's definitely within my sphere of competence.

On the plus side, picking a project way outside your limitations is a good way to expand your limitations. I know so much more now than I did when I started this project.

Conclusion

Ultimately, GameClay failed because I gave up. Up until that point, it's just a startup that has "not yet succeeded", and so I feel like I should explain why I'm giving up:
  1. I don't think I can do this without a cofounder. It's very, very difficult to wear both the developer and the evangelist hats at the same time: being a developer requires that you be very pessimistic, so you can see and fix all the problems in your design, while being an evangelist requires that you be very optimistic, so others can feed off your passion. I suspect that if I tried to do both, the cost would be my sanity, literally, and that's not a price I'm willing to pay for the startup. Cofounders also help even out the emotional highs and lows inherent in doing a startup, since you're rarely in phase.
  2. I've exhausted the pool of potential cofounders I know. Amherst College is not really a hotbed of entrepreneurial types, and most of my friends are now either lawyers, in grad school, or have secure corporate jobs. And I've found that you can't just jump into business with someone: you really need to forge the relationship in a low-stress setting before you subject it to the pressures of a startup.
  3. We're moving too slowly. This was a problem at my last employer, where it took 7 years and counting to build their platform. The risk isn't really competitors; most markets develop far more slowly than you'd think. It's that the whole technology ecosystem changes over time, which makes your initial design decisions a disadvantage against startups that start fresh. Woe to the companies that started building desktop GUIs in 2002, for example.
  4. There's little outside indication that people want what we're building. When I run it by friends, most find it interesting, but they find it interesting because I'm doing it and I'm their friend and not because they really understand the idea. Also, competitors have been significantly less successful than I thought they would be.
It's not really the end of my startup journey - I suspect there will be other startups in my future, both as an employee and hopefully as a founder. But it's the end of this startup. I'm a little sad about that, but I have no regrets about having started it.

The end, I think

Didn't work out with the YC startup. Their offer was actually quite reasonable, and I think that they'll likely succeed, but I just had a strong gut feeling that this was not the place for me. I don't want to put myself or them in a situation where I'm just terribly disgruntled six months down the line and then my leaving screws the company.

The experience of working with a YCombinator startup, even for just a week, was great though. It's amazing how much energy is in that place - it's like a full notch higher than I was working at, and I thought I worked hard.

That brings me back to GameClay, which at this point I think is on life support. And I think it's time to pull the plug. I remember Sam Altman's response to "How do you tell if you really are doomed?" - it was "If nobody cares about what you're doing and nobody on your founding team cares enough to change that, then you're probably in trouble." Well, I think that's basically the point we're at now.

I'm going on vacation in two days, so I think the plan is to create a personal home page, package up all the Diffle stuff that should be open-sourced and open-source it. Then I want to make this blog public, possibly linking to Subversion and Trac (though since my home page will likely be on a shared webhost, I dunno if I can host the live Trac app). I'll also write up a postmortem with some of the lessons learned.

Then, I guess I'll e-mail Paul Buchheit and see if FriendFeed is still hiring. Actually, I want to work a bit on the programming language I abandoned to start Diffle, see if I can get the hard parts done before I start work again. So maybe by August/September, I'll be employed again, and that's the end of (this phase of) my startup journey. Hopefully it won't be the end forever, though - there are second acts, I'm just not ready for them right now.

Wednesday, June 11, 2008

Updates

Woah, a month and a half between updates. A lot has happened since then, development-wise. I'll try to reconstruct it from the svn timeline.

When I left off, I was testing performance. Game performance is *still* inconsistent; I think that O(n^2) collision detection algorithm is really biting us, and we'll have to replace it with something saner. However, widget performance itself is quite good (except under VMWare), which surprised me a bit. It's definitely possible to do rich JavaScript UIs with JQuery UI - their speedups in 1.2.6 have helped a lot too.

I also hooked up the AJAX-compiler infrastructure and image uploading to the backend, so they make real AJAX calls and actually change the behavior of the game. It was cool to see this working - I don't think anyone else has done anything like this. Unfortunately we don't have all that many behaviors available - mostly you can just change trajectories, collision behaviors, properties, etc, which is still cool but doesn't really showcase the full power of the technique.

I had to redo the validation infrastructure to work out the bugs on this. Lesson learned: error messages matter, and if you can't output pretty, precise error messages, you'll waste far more time in debugging.

I also made a quickie library that mimics the interface of Justin Azoff's Python MogileFS library but uses local filestorage instead of actually talking to MogileFS. Been meaning to do this for a while; I was sick of setting up MogileFS every time I did a new install, and it's totally overkill for our current requirements. So now I can write files to the local hard disk, but just swap out a library and have it work with the real MogileFS.

The UI is much more fleshed out - nearly all the customizer panels are now done. I completely redid how they're laid out - there's only a single flat list on the left now, containing all sprite names. This was done to reduce confusion and eliminate a clickthrough on the sprite category; the result seems to work much better. I also moved the scrollpane inside the right-hand accordion, to prevent headers from falling off the bottom of the page and requiring scrolling.

I think we may be within a month or a month and a half of launch.

Now, I got an e-mail from one of the YCombinator SFP08 groups. Apparently PG gave my name to them. Am off to give them a call and see what they want.