Breaking things up

The winner of the last blog entry was Django. Reading over it again, it seemed that my main objection to Django was that the problem structure - hierarchical data, stored in Mogile, with lots of functions that do things other than CRUDscreen operations (eg. compilation & asset management) - didn't really fit a web framework meant for CMSes. Other than that, Django would be great, and there were a bunch of 3rd-party Django apps I wanted to use.

But there's another option - I could make the structure of our project fit Django, by pulling all the other stuff relating to the game creation language or compiler out into a separate project. And I should probably do this anyway, because it leads to a more flexible, more decoupled architecture. For example, I could run command-line tools that make & publish games and yet don't need Pylons installed.

So I just spent a couple days pulling everything out into separate projects. I have one for the compiler (input: a JSON data structure representing the game, output: a Flash file), one for storing things in Mogile, and eventually one for the website itself. Former 2 are standalone Python projects installed via setuptools, the latter one is a Django project that hopefully will also have a setuptools script. The compiler passes all its existing unit tests for both Python and Actionscript already, the storage module will be tested in a bit, maybe after dinner.

I'm kinda surprised (and very glad) that it only took 2 days. When I did the same thing at my first employer and my last employer (hah, no matter where I work, I end up doing the same job), it took like a month, and in my last employer's case was never finished. Granted, there I needed cooperation from the other engineers, and they all had other things to do.

Pylons and Django

Thinking about switching the webapp portions of GameClay from Pylons to Django. This would be our third framework (web.py -> Pylons, Pylons -> Django), and Pylons is mostly doing acceptably, though it's pretty annoying in some places and needs some code to fill in functionality that Django provides for free. So this is not something I'm going to do lightly.

Here's a list of the various aspects and advantages & disadvantages of the frameworks:

Templates

For Pylons, I'd stick with the default of Mako. Django comes with its own templating language, but it's not terribly hard to swap it out. However, the existing contrib apps expect templates to be written in Django's language, so if we do any customization of contrib apps, we'll need to use Django's. It's probably better to stick with that for all web templates, even though we'll be using standalone Mako in the game compiler.

Advantages for Pylons:

• I don't have to rewrite our existing Mako templates
• Mako is significantly more powerful than Django, with arbitrary expressions, defs, template inheritance (beyond layouts)
• Mako filters get pulled from the template namespace automatically, which is a useful convenience when using potentially lots of filters
• Same template system for website and game compiler
• I can extend it with defs, in the template, without needing to write a separate Python module to write new tags
  

Advantages for Django:

• I like the syntax better; it seems cleaner and easier to remember
• Filters can take arguments; useful for things like indent()
• The inclusion_tag syntax provides cleaner-syntaxed and more flexible blocks of reusable content than Mako's defs. The pitfall is that these are not defined with the other templates; they live in the template_tags directory.
  

Models

For Pylons, I'd use our custom Diffle DB database wrapper. For Django, I'd use its ORM.

Advantages for Pylons:

• Simpler; I know SQL already, don't need to learn a custom model definition language
• Can handle multiple databases reasonably easily. It seems very difficult to add this to Django; there's been a branch with it since 2006, but no movement on it. It seems like it'd be a real pain to adapt save() and the query methods to a horizontally partitioned database.
• Gives finer-grained control over queries; I can often avoid some extraneous SELECT or UPDATE statements that Django's ORM adds.
• Supports multi-column primary keys, something I've found pretty useful in past DB models.
  

Advantages for Django:

• The model definition classes are actually kinda convenient to use; would eliminate the mental context switch needed to go between SQL and Python. (I can get similar benefits by moving to SQLAlchemy + Pylons, but then I lose the other advantages of using only a thin wrapper).
  
• The admin interface is very convenient; it'd let Mike respond to routine complaints from users instead of requiring my intervention. (I might be able to build something similar by introspecting the database, but it'd take time.)

Controllers
Pylons controllers are basically WSGI apps, setup by the "paster controller" script. Django controllers are functions that take a Request and return a Response. Pylons uses WSGI pretty extensively throughout; in order to understand the choices it's made for controllers, it helps to understand WSGI really well (I don't, yet).

Advantages for Pylons:

• Each Pylons controller is its own WSGI app and in theory can use any other WSGI middleware. (In practice, I'm not sure this works, since Pylons constructs the controller objects with each request.)
• Provides an easy way to attach data to controllers, like a database or mogile connection
• Routes syntax is a little cleaner and more flexible than Django's URL-based dispatching.
  

Advantages for Django:

• Don't have to use the ugly c, request, session, etc. global variables. Since WSGI apps have a defined signature, they can't take additional parameters, which can be quit inconvenient in Pylons.
• Don't need to mess with StackedObjectProxies. Globals themselves aren't suitable for use in a webapp (they aren't threadsafe), so Pylons has its own version of a threadlocal. Unfortunately this is one hell of a leaky abstraction; you can't do dir() or vars() on it and get back anything sensible, and they break when used in generators.
• More explicit. Less to keep in mind.
• Less verbose. Controllers are just functions, and they can all live in views.py (though you can break them out into separate modules if necessary). Presumably they could be callables if necessary, though that breaks the next point.
• Can be decorated. Django provides a bunch of useful decorators, like @login_required.

I much prefer Django's controller system - it's simple, intuitive, and doesn't have lots of boilerplate. Personally, I think that Pylons' approach of using globals is a big mistake; it was one of the main reasons I moved away from web.py

Sessions/Login/Registration
Pylons gives you AuthKit by default, but that looks so unpolished, undocumented, and unsuitable to GameClay that I'd probably have to write my own. Django offers the contrib.auth application, which depends on the Session middleware.

Advantages for Pylons:

• Completely flexible, since I'm writing my own. Can include things like IP addresses in the session table, or reference the sessions table to see who's playing which games, or check out what your friends are playing.
• Integrates easily with the rest of the app, eg. templates and other controllers
  

Advantages for Django:

• Provides usable default controllers for login, logout, and registration
• User objects integrated with request; no need to explicitly check for login
  
• Can retrieve all sessions through Session model; however, session models have only the key and data, and so if we want to search by user or page, we have to do it ourselves. We probably need to use custom middleware for this; we can't do it as a session backend because we don't know the username at that point, and it'd be a pain to do it as an authentication backend because we'd also need to change the Session model and that's not directly accessible.
• Handles details like tamper-proof sessions, hashing, generating unique sessions, etc. transparently
• Has convenient @login_required decorators for marking posts that require logins. Also allows filter functions; this could be useful once we have shared games and such

Other goodies

With Pylons, you're basically on your own. You can use any Python libraries you need, but you can do that with Django too. Django provides a bunch of nifty features that we might like to use:

Advantages of Django:

• CSRF detection. We need to add this to all our forms otherwise; this is pretty handy middleware.
• Redirects. Particularly if we change the ID or URL scheme. We don't want to ever break URLs; this could be a handy way to avoid that.
• Syndication feeds. There are Python libraries to do this, but the contrib app is likely easier to use

Configuration, Packaging, and Testing
Pylons is based on Paste, setuptools, and nosetests. Django has its own config and packaging system, and uses doctest and unittest for testing. The two are roughly equally easy to use - they both provide single commands to setup most things.

Advantages for Pylons:

1. They handle multiple config files very well. Each config file is a separate .ini, so you have a test.ini for testing and a development.ini for development and a production.ini for production. Each of these can be checked in, and you specify the configuration to run when you execute Paster. Django lets you have multiple settings modules, but you have to switch between them with an environment variable or command-line switch, which is kinda annoying.[edit: the command-line switch turns out to not be that annoying in practice, though a little confusing if you don't know about it.]
2. Searches for doctests in all locations, not just models and a specific test module. When you have lots of non-DB code, as we do, there will likely be lots of other stuff.
3. Easy to specify foreign dependencies via setuptools, and create command-line scripts, and all the other great stuff that setuptools does. Using setuptools with Django is somewhat clumsy; I hope it's possible.

Advantages for Django:

• Using a Python module as a config file keeps everything Pythonic, and gives some added flexibility
• The concept of separate "sites" and "apps" results in a cool architecture where we can potentially utilize other people's contributed apps. I doubt this will be terribly relevant for us, but it may turn up something neat.

Overall, I think Pylons has the advantage for packaging and distribution. I'm disappointed that Django seems to be moving away from setuptools, even dropping it for packaging the Django distribution itself. Pylons also really thought through deployment in multiple environments, while Django seems to have done a quick & dirty solution so they could go on to other things.
Other reviews

http://pythonmag.blogspot.com/2006/02/pylons-vs-turbogears.html
http://jesusphreak.infogami.com/blog/why_django
http://jesusphreak.infogami.com/blog/vrp1
http://www.cmlenz.net/blog/2006/08/the_python_web_.html

Unit testing

Finally got AsUnit to work with MTASC. There is basically zero documentation on the web for this; I should probably write something up sometime. Goes on the bottom of my TODO list, after I get everything done for Diffle.

I rigged it up so we have a Python script that builds the game archetype, then links that against AsUnit and our unit tests and runs all the unit tests through the standalone flash player. It's pretty convenient - I would've preferred a completely command-line solution that automatically checks all output and doesn't require a GUI, but this is still a single command.

It found 7 (!) bugs, just in the ActionScript skeleton we've got so far. Most were silly typoes, forgetting an element of an object hierarchy or passing in an object when we meant a string ID. But without some sort of testing, we'd never have been able to track them down. Flash is notoriously debugger-unfriendly, particularly on Linux without the Flash IDE.

Mental Centering

I'm feeling like my mental center of gravity has been shifting more towards Diffle over the past 2-3 weeks. I remember back in October I felt like I hadn't really "jumped" yet, that something was missing. It wasn't really a matter of jumping, more a gradual acclimatization as I "detoxed" from my previous day job.

I kinda want to do a tool that'd graph Subversion commits over time, just to see if this is reflected in actual productivity. It feels like it is; I think I've done something like 50 check-ins in the past week. And I'm actually moving fairly quickly on the rewrite. In the week and 2 days since the last update, I've got a blank archetype all compiling, including expression compiler and all, and the code is much cleaner than the not-incrementally-tested version. This time around, it's got full unit tests and documentation and I've figured out how to use setuptools and paster to automate the whole build process.

Some notes on the issues in the last blog posting, so I know why I made certain decisions:

1. scheme2js was rejected because:
1. It can't call out to arbitrary JavaScript functions; they have to be declared, in a way I couldn't figure out
2. It doesn't generate readable code at all - everything is temps, and it's like it's machine code.
3. I don't think there's any way I can get it working to generate MTASC code.
2. haXe was rejected because external libraries need to be declared, which is a huge extra burden when we use as much 3rd-party JavaScript as we do. Also didn't seem particularly JQuery-friendly, and the mailing list posts indicated that they had no plans to support it because it was too functional and not OO-enough.
   
3. I'm keeping the _root.fn nonsense. I still hate it, but we don't have a better alternative at this point, and we can always get rid of it via global find & replace.
4. Leaning towards invoking the compiler via AJAX for the editor issue, though I still need to see how it works in practice.
5. Canvas tag prototypes went very well - it was quite easy to use, generally cross-browser, and we got some neat effects like lasers and rotation working.
6. Didn't get around to prototyping the expression language, but I did think more about its design. There will be an actual parser for it. However, I'm going to limit to arithmetic/boolean expressions and function calls. Thinking about making it strongly-typed, with typechecking running via AJAX to tell you if you've made an error as you type. As for variable scoping, it's pretty simple: I'm introducing keywords "this" and "that", and the name of each sprite type (subscriptable) will also be a variable introduced into the current scope. Until we have user-defined variables, I don't think we need more.
7. Didn't prototype worlds larger than the stage; really should, I guess. I figure that we'll just make sprites children of a 'world' movie, and then move that around to scroll the world.
8. I got Mike's new layout up and running under Mako. It looks pretty cool, with all the drop shadows and rounded corners. Was pretty simple to write too; the HTML really is dead-simple.
   

Next steps are importing games into Mogile and building the editor. I expect some fun surprises with the editor.

Oh, and I wanted to outline the sequence of events that led up to the rewrite, in brief form, because it's instructive of the kind of design pitfalls that hit startups:

1. Working on the editor, found that I couldn't edit trajectories and have them immediately applied to the game.
2. Fix would require that the editor have knowledge of individual game archetypes, which would introduce too much complexity.
3. Dropped the concept of game archetypes with special-cased code.
4. Because of this, the base game structure needs to have many more options.
5. This changes the primary data structure we're storing games as
6. Which touches nearly every aspect of the system, from how games are stored in Mogile, to how archetypes are specified in the codebase, to the type of UI required, to the compilation mechanism.
   
7. Which means we might as well rewrite everything.

I'm generally much more satisfied with the new architecture though - it's much more flexible and elegant, and since I've eliminated the whole archetype degree of freedom, I can make the editor much more user-friendly and robust.

Rewrites

Bit off more than I could chew. It's now been a week since my last commit, which is usually a signal that it's time to do a revert and start over again. I got most of the code done and was working on debugging, but then I realized there's no way I can verify the quality sufficiently to have confidence in the code I just wrote. So it looks like I'll be scrapping it and starting with a simpler feature addition.

No matter how many times I make this mistake, I always seem to make it once more...

Actually, I may back up all the way and start from a fresh Pylons project (keeping the bottom-up widgets libraries I've developed, of course, which now account for over half the code). Now that we have a better idea of the data structures involved, many of the early attempts at a game compiler just seem like extra cruft. We can also keep the editor, compiler, and game archetypes reasonably in-sync instead of having the compiler and games get way ahead of the editor.

There are a couple of weak spots that I'm still feeling shaky about and want to prototype some more before I build something that'll hopefully be the final version:

1. Worlds larger than the stage. I haven't really played with this in Flash, yet I think it's really important for our initial version. Games are much more fun when there's more of them offscreen. We also need to figure out how to edit them in JavaScript - am thinking OpenLayers or Canvas, maybe.
   
2. Canvas tag in general. Nobody seems to use it, but it may be pretty useful for what we're doing.
3. Expression language. This is a big one; the existing system is pretty ad-hoc, consisting of simple regexp replacements on what is basically ECMAScript. We should have some sort of formal language with error checking and a real parser itself. The existing version isn't even correct in some cases.

I also need to figure out how to reflect our changes immediately in the editor, now that we offer full, composable statements. I figure our options are:

1. Create an interpreter in JavaScript for the JSON "language". Don't really like this one, because then we need to update both the compiler and the interpreter when the language changes, which seems like a pain.
2. Duplicate the compiler in JavaScript, then eval the resulting function string and place it in the appropriate event handler. Same issues
3. Call the backend compiler and have it return a result via AJAX. This seems like the best option in terms of code duplication, but will result in a delay before the change is apparent. This may be acceptable, though, as long as it's only a couple seconds.

Also, I wanted to look at scheme2js and see if there's any usage for that. I'm guessing there won't be - we want to serialize our data as JSON for its cross-language benefits, so we'd just have to compile the JSON version to Scheme. But we do need some way to write library functions that can be used with both MTASC and JavaScript, preferably without the _root.fn nonsense we've got now.

Maybe I should look into haXe too. Last time I looked, there were issues with library support - whatever we use for JavaScript needs to support JQuery and probably Canvas. But at least it'd eliminate the need for MTASC.

Platforms

X-posted from a planWorld entry:

I just followed a link to a Worse Than Failure entry, and the article was almost an exact description of my last employer. Well, not quite, since they programmed in Java and ended up reimplementing a half-assed version of Java, while the article is all SQL. But close enough.

Maybe it was a good thing that I left.

It's funny though, when I started my own company, I vowed I wouldn't make the same mistakes as my two previous full-time employers. And now I'm either making or have narrowly averted making at least a half dozen of those same mistakes. It's funny how easy it is to fall into those traps.

Note to people who are thinking of starting a software company: do not start out thinking "I'm going to build a platform for X." Because X will very quickly become "everything". And then you'll have a poor imitation of the programming language you used to build it.

At least I know a bit more about programming language design than my last boss did, and have a better idea of the tradeoffs involved. And Mike's been pretty good at reigning me in and saying "No, we're not going to support that, let's just get a basic game done and worry about extending it later." So hopefully we can find a happy medium where it's still usable to users without programming experience and yet can build cool things.

Since this is a private blog, I can get into a little more detail here. The resolution to my last blog entry was to abandon the idea of archetypes. Trajectories are just one type of stateful interaction that needs to be reset as the user edits the game, and different archetypes may have different ones (shields, for example, or other game properties). These would all differ in different archetypes, so there's no way to adapt the editor software to handle all possible cases other than to hard-code behavior for each into it.

Besides, coding up all the different archetypes would be a huge programming burden on me, and we just don't have manpower for it. Much of the code was shared between them, anyways, yet we have no facility other than our template-based shared libraries for writing routines that can be used by multiple archetypes. And it limited user freedom - if they wanted a hangman where the body parts moved and could be shot at, they were out of luck. This opened us up to user confusion, since given flexible enough customization, games could drift quite far from its original archetype. Users would be asking "Why can't we make this game do what we want, when this other one which is nearly identical can do it?"

So I ditched the idea entirely.

I haven't told Mike yet - normally we make all decisions by consensus, but in this case, the alternative is that we can't finish a real piece of software in a reasonable amount of time. I figure I'd just have to overrule him anyway on technical grounds, and I hate giving people a choice when we don't really have one.

Unfortunately, that means I've got to be doubly-vigilant about things getting too complicated and technical. So far the UI seems okay, and aspects of game design actually got simpler from this. But now I'm finding there are so many such aspects, and each has to be handled. The one that prompted the above blog entry was sprite creation - it's fairly easy technically (we've already got createAt/createOn/createIn actions that handle the various cases), but the UI offers lots of potential choices. Do we give a separate Creation tab for each sprite, where the user can specify where, when, and how many? Do we make it an action attached to a timer on the game? How do we handle randomness in creation?