My co-founder and I built a non-trivial web site on Django. When the next version ships, there might not be a single Django module imported.

We’re not trying to drop Django; it is just sort of happening. Piece by piece, it is failing to meet our needs. Despite the marketing copy on the Django site, most components of the framework are tightly coupled enough to make customization frustrating. It is often easier to rewrite core framework components than to implement them on top of the existing extensibility points.

What follows is a loose chronology of our migration away from Django.

URL Routing

A flat list of patterns violates the DRY principal when creating nested URLs. Trees are a superior representation. Having a tree of views also enabled us to optionally associate a “binder” function with each node. These bind functions are executed for each URL component from left to right, filling the template context as they go. Breadcrumbs are automatically generated as each node binds, but only the last node executes its full view logic.

Authorization

Our site enforces permissions on every resource, but Django’s database ACLs would have been prohibitively numerous. Instead, views or their URL binders may raise an AccessDenied exception. Upon catching such an exception, a middleware layer serves a login form. This ensures users have permission to access the current resource, as well as all ancestor resources bound to the URL.

Authentication

Both of Django authentication’s key extensibility points are flawed. These two extensibility points are “user profiles” (storing additional per-user data) and custom credentials (such as for logging in via email address instead of username). Django’s documentation and numerous internet sources cover both topics, but all of the guidance lacks important caveats. The admin UI, in particular, is very easy to break with either extensibility mechanism.

Extending the User model with the ORM requires a one-to-one database relationship. This relationship can be implemented with a “user profile” setting, an explicit foreign key, or model inheritance. Each approach has its own strengths and weaknesses in terms of performance, API semantics, subtle behavioral changes, and outright bugs.

Enabling custom credentials requires implementing a trivial authorization “backend” object. Unfortunately, it is non-trivial to replace usernames with email addresses. The admin UI’s login form refuses to accept email addresses without hacking the template. Even if you hacked the template, the User model would still enforce a non-null constraint on the username field and the generated database schema enforces a uniqueness constraint as well. It turns out to be easier to fill the username field with a dummy value and “support” both forms of authentication with your backend, but you won’t come to that conclusion until your head has already bore a hole in your desk.

Templating

We do our best to keep view and template logic separate. Django’s templates are targeted at designers, who aren’t implementing any real logic anyway. However, we’re a pair of hackers. Sometimes it is just more convenient to put a little bit of logic in the views. Besides, templates are code; code needs to be reviewed and tested. We wouldn’t ever hire a designer who couldn’t pass a code review for some trivial template logic.

We needed a pragmatic template language to replace Django’s idealistic one. Any template language with greater expressive power would have been welcome, but Jinja2 fit the requirements and provided the easiest migration path. Ultimately, we’d prefer to use something like HAML, but there doesn’t seem to be a Python equivalent besides the inactive GHRML project. We are, however, using SASS. I will never write CSS by hand again.

ORM and Admin UI

One of Django’s most touted features is the Admin UI. For simple “active record” style database models, the Admin UI is a huge time saver. Sadly, it struggles a little bit with nullable fields and is tricky to customize. You’ll definitely need to write custom UI for complex models, but by and large the admin solves the problem at hand: viewing, creating, updating, and deleting database rows.

After using the Admin for a little, I found myself missing Microsoft Access. I never thought I’d say that, but it is true. Django’s admin does not support sorting, filtering, or other impromptu queries. Edit: It turn’s out I was mistaken about sorting and filtering, but I stand by the core message of this section. I found myself writing impromptu queries in the database and Python shells. After a while, I just gave up and installed a desktop client. I haven’t visited the Admin UI since.

Django’s ORM has shortcomings with respect to querying, especially for joins and aggregation. It has been improving over time, but it will likely never reach the capability of projects solely focused on databases, such as SqlAlchemy. With the admin having fallen into disuse, the Django ORM lost all advantage. Beyond Django’s specific weaknesses, I’ve come to believe that the schema-generative ORMs paradigm is fundamentally flawed. That is a topic that deserves an entire (Django-agnostic) post of it’s own. We are now using SqlAlchemy via schema reflection; no declarative layer.

Form Validation and Generation

Here is where our chronology meets present day. We are still using Django form validation, but never used form generation beyond scaffolding. Nearly all of our templates customize labels and display of errors. Additionally, embedding widget information in the Python code is cumbersome during template development. Django forms is a quality validation library, but there are some inconsequential style things that I like better about FormEncode. Preferences aside, the difference isn’t large enough to justify switching.

While I like FormEncode, I’m still not sold on its anti-form-generation companion, htmlfill. I think there is a middle ground with form generation that provides scaffolding during development, smoothly transitions to production use, and cooperates with validation. As we implement more complex client views, I’ll be on the lookout for ways to improve our form development toolbox.

So, ugh… What’s left?

Besides a few isolated helper functions, not much is left of Django.

The last big ticket item is the HTTP framework and WSGI server. We could continue using Django as if it were CherryPy or Paste, but Django has this nasty habbit of insisting on running your code for you. The settings and manage.py infrastructure are fiddly for deployment and don’t really add any value over simple scripts using our application like a library. Might as well use a simpler WSGI library, and replace those over-engineered management/commands/foo.py files with vanilla scripts/foo.py files.

Moral of the Story

I’m sure there are numerous lessons to be generalized from this journey. Personally, I’ve developed a moderate fear of the word “framework”, as well as altered the way I think about software abstractions. I think the most important lesson, however, is one I already knew: choose the right tool for the job. Unfortunately, we had no idea what the right tool was when we started. I’m not sure we know any better now.

Imported Comments

Austin

Maybe it worked out exactly like it should have. Django bootstrapped your app to a certain point. Got you further faster than you would have if you implemented everything from scratch. Then from there, you identified the things you considered inadequate and replaced them. If it all goes away who cares. You have learned something, shared it with us and moved on.

You seem to have successfully moved away from it to your own custom code. I am impressed that it sounds like your decoupling your app from Django was not as traumatizing as it might have been.

Brandon Bloom

Yeah, everything seemed to work out OK. Django was definitely a big help, especially with my initial limited web experience (previously, I was a game and desktop application developer).

If I were to do it all again, I probably wouldn’t choose Django. That’s not to say Django wouldn’t be the right choice for someone else. I just think that I’d choose a collection of tools that are less likely to get stuck glued together.

Eric

I think Pylons would be a great fit for you. It matches the way you want to work.

Varikin

Your URL routing sounds intreging. I like have been looking at Werkzeug, Routing and Django. I prefer Django to all three, but still I feel like something is missing.

Could you elaborate on your solution?

Juan_Pablo

Take a look at Werkzeug. Despite the weird name is the most modular and yet easy to use wsgi wrapper and dispatcher I know. Combine it with Jinja2 and SQLAlchemy… and voilà.

WhatDoYouThink?

Good thing there are people like you who would share your improvements back to the community!

Brandon Bloom

Thanks Juan, I’ll take a look at Werkzeug.

Eric: I’ve been looking at Pylons, but I think I want to go with something more like Juan’s suggestion.

Varikin: I was able to implement my URL routing over Django by simple matching (.*) and then writing a view which splits by ‘/’. I then loop over each component and perform a depth-first search of a tree structure. Each node has an optional bind(request, template_context, path_component) function. After I find the node which matches the path, I call the ancestor chain’s bind functions and then finally the current node’s view function. We’ve got some decorators and helper functions to make building this tree easy. I’d share it, but it is too tightly tied to our application. Maybe I’ll write more about this later.

WhatDoYouThink: My improvements are application specific. I’m sharing my experiences to help the community. That is far more than most people do.

Jeremiah

This comment has been removed by the author.

gaspode

The tree stuff and permission scheme you’d like perfectly matches up how repoze.bfg works with traversal and security based on traversal schemes (also known as object dispatch).

I’d highly recommend giving it a shot as it seems to mesh quite well with the way you’ve laid out the app.

Tucanae Services

Think Web2Py…..

Chris Shenton

Ditto what gaspode said. I’ve been fighting Django for an app whose content is basically hierarchical and BFG’s traversal would really be a relief.

Cesar

I’d like to add my 2 cents. Restish has no threadlocals (compatiable with eventlet/spawning) and focuses on resources/content negotiation. I believe the URL routing is similar to what you want

handi

Enabling custom credentials requires implementing a trivial authorization “backend” object. Unfortunately, it is non-trivial to replace usernames with email addresses. The admin UI’s login form refuses to accept email addresses without hacking the template. Even if you hacked the template, the User model would still enforce a non-null constraint on the username field and the generated database schema enforces a uniqueness constraint as well. It turns out to be easier to fill the username field with a dummy value and “support” both forms of authentication with your backend, but you won’t come to that conclusion until your head has already bore a hole in your desk.

We wanted to let users log into our app using their email address through contrib.auth.view.login. We don’t care about having email addresses log into the Django Admin.

You have to dig around a bit, but it can be done with a few lines: http://gist.github.com/170691

nassrat

I think you were looking for Pylons not Django. Each has its own advantages. I learnt that Django should be taken as is, once you start stripping it apart, IMHO you should consider using Pylons.

reedobrien

Sounds like you should look at http://docs.repoze.org/bfg/current/ http://bfg.repoze.org is a nice microframework which stays out of the way as much as you want it to.

Jonathan Ellis

If you like SQLAlchemy, you will probably like FormAlchemy for form generation from SQLA models.

adeel

I’ve written my last few web apps on top of Berry, which just lets you map URLs to methods (and lay them out however you want). And obviously you can import SQLAlchemy, Jinja, and whatever else you want yourself. I’ve found this gives me the most flexibility possible without actually writing a raw WSGI app.

Michael Galpin

You really need to read this http://terrychay.com/blog/article/challenges-and-choices.shtml. It will help you understand why you were doomed from the beginning :–)

Brandon Bloom

Thanks Michael. Despite the distracting laughter that was an interesting read.

gunzip

After using the Admin for a little, I found myself missing Microsoft Access.

it still happens to me every time i start a new data-entry project on the web. sometimes i think something went wrong in the latest 20 years…

Dead Man

I’m also building a project on Django (a medium sized, multi language web encyclopedia about a Northern-European state Estonia) and I have to admit that I’ve run into mostly the same problems. What surprised me the most is that I’m implementing a nearly identical to yours solution for URL mapping as our content is also in a (generic) tree structure. The difference being that I’m going to use class based views that allow nesting/composition of other views for the purposes of chaining that you described.

I’m also thinking of open sourcing the code as the project matures more as it’s already written in a clean way and decoupled from project specific code.

ars

Beyond Django’s specific weaknesses, I’ve come to believe that the schema-generative ORMs paradigm is fundamentally flawed. That is a topic that deserves an entire (Django-agnostic) post of it’s own. We are now using SqlAlchemy via schema reflection; no declarative layer.

Well you caught my attention with that. Are you planning to write such a post in the near future, say in the next two weeks?

Andrew Shultz

One of the big things I appreciated about django is the unit test supports. What’s your plan to replace that stuff?

proteusguy

Almost all of your technical feasibility issues are easily addressed by Django. URLs, authN/authZ, etc are all as flexible or more than any other python environment out there because it is so simple to leverage python to implement this things in an incredibly simple manner. This blog entry demonstrates it better than I could. It’s really just a matter of perspective and context. Django may not be the context that best fits your perspective but it most certainly has the capability and power to address all the technical concerns you brought up. Good luck finding the one that fits you best. I actually thought I’d be a TurboGears guy but ended up Django. Both are great and others are good as well.

Alexei

I am working on building a medium size application in Django too. It is indeed a good way to start. But I am hitting issues too. I have multiple “agencies” so need to have users with multiple roles in agencies. These are within single site. That also means multiple “admins” with one “super admin.” So Django Admin interface is out of the question and role/permission infrastructure is built from scratch. Another thing is importing data. I started writing scripts within Django framework (using settings.py, etc.) but need some tables/entities created during import and then dropped. Oops. Have to look at how South does it.

I realize that I am moving out of Django domain just because of the nature and complexity of the application I am building. Django cannot and should not be everything for everybody.

If you are looking for a “glue” framework, I recommend checking out Cherrypy: I like its configuration and request/response handling better than Paster + WebOb. It also has several types of routing, pick what you like or mix it (I would not recommend the latter though), one of them is Routes.

My next move will most likely be to Cherrypy + SQLAlchemy + FormAlchemy + Jinja2.

By the way, repoze.bfg is not microframework either, it pulls quite a large chunk of Zope code, primarily for managing configuration but for templating too.