Rethinking Everything: Perl and the Web in 201x

Aug 20, 2010 by chromatic

In 1999 and 2000 I worked on the Everything Engine, a website management system written in Perl 5 which runs sites like Everything 2 and PerlMonks. The system has its flaws–many of them due to our inexperience building and maintaining large systems with the Perl of the time.

We could build it much better today, with modern libraries and techniques in Perl. In truth, building such a system today would solve a lot of uses for MVC applications.

How Everything Works

The fundamental concept of Everything is that everything is a node. A node is an individual entity containing attributes, including a unique identifier and one or more addressing mechanisms expressible through a URI. Each node also has a nodetype (itself a node) which contains information about the node’s characteristics, such as the attributes it contains, the types of operations it supports, and any mechanisms to view node information.

In other words, a node is an object, a nodetype is a class, and there’s a metaobject protocol governing the relationships of nodes and nodetypes.

The Everything Engine provides a sort of routing system which might remind you of routes in one of the new heavy-model MVC systems. In one sense, Everything provided its own implementation of MVC, where URIs and URI components helped the controller find the appropriate model (node) and model operation (nodemethod), which eventually caused the rendering of a series of templates, or the view.

The system provides node inheritance for attributes and behaviors such as creating, reading, updating, and deleting nodes. Because every node inherited from node, every node has at least general behavior through any interface into the system. If you create a new nodetype, you can do useful things with nodes of that type without having to write any specific views or mess with routing or the like. You have the option of making prettier or more featureful or better or at least different views, but the default inherited views and behaviors are sufficient for the basic CRUD operations.

What Everything Didn’t and Did Well

Not everything went right. Remember, this system came about in 1998 and 1999, just as mod_perl was the shiny new technology in the world of Perl web development. Object-relational mapping was still difficult (if even understood much), CGI.pm was still the king, and Template Toolkit hadn’t taken over as the obvious first place to look for your templating needs.

Our hand-written ORM has its flaws, as dealing with node inheritance and lookup of node types and (yes) serializing code to the database to eval into a running instance of the system at initialization time. The XML serialization scheme for bundling core nodes and custom nodes was even worse, not only due to our use of the DOM.

Without a well-understood mechanism and framework and example of doing MVC well in Perl 5, Everything’s custom interpretation of MVC was odd. Where MVC helps separate application responsibilities into loosely-coupled layers, passing around a CGI object to models and templates violates that encapsulation. (The less said about the custom method dispatch strategy the better.)

Most of the problems with the system are obvious in retrospect, over a decade later (especially with a decade of experience creating new systems, maintaining existing systems, and stealing ideas from other projects which have made different mistakes).

Even with all of those mistakes, the system worked reasonably well in many cases. Adding new features was easy (even if deploying them is less so). Creating new behaviors by reusing existing primitives makes building simple systems easy. The usability provided by inherited defaults made it easy to iterate and experiment and refine new behaviors.

We also eventually produced a system to bind UI widgets–HTML, in our case–to node attributes. That made displaying and editing mechanisms much, much easier.

I wouldn’t start a new site with Everything as it exists today, but I’ve wanted a modern version of it for a long, long time. Modern Perl has almost all of the pieces needed to rebuild it.

Doing Everything Better

The Moose object system provides an obvious way to define nodes. A node is merely a Moose object, and a nodetype is its class. This provides a well-understood and robust mechanism for dealing with attributes and methods and roles and the like, and the metaobject protocol provided by Class::MOP allows introspective capabilities which will become obviously important soon.

Plack support is obviously the current best way to deploy a web application in Perl 5, as it allows reusable middleware and offers many deployment strategies.

DBIx::Class is the first ORM to consider in Perl 5 right now. I’m partial to KiokuDB for applications where I need persistent objects but don’t need to provide a relational interface to interact with the data. The ability to use either one as a serialization backend is important.

Any of the modern frameworks or libraries or toolkits for providing the controller part of the MVC application will do. In particular, all this layer of the application needs to do is to map requests to nodes, manage concerns of user authentication and logging, invoke the fat model objects to perform their business actions, then dispatch to the view to render the appropriate information in the user interface. I like the simplicity of Dancer, but anything compatible with Plack will work for web applications.

A New Architecture for Perl Apps

Here’s my vision.

I run a publishing company. I want to manage a web site with information about existing and upcoming books.

I start by defining some models: a Book has a title, an ISBN, a price, one or more Authors, a short blurb, a full description, and publication date, and a cover image. An Author has a name, a biography, and an image.

For each attribute of each model, I choose two types: the type for the object attribute itself (an ISBN is a ten- or thirteen-digit number, an author is an Author object, et cetera) as well as the type for the UI (Authors display as a list of authors and they require a multiple selection UI widget to display). That author selection widget is interesting because it can be a parametric role: a role which knows how to display multiple *something*s, where that something is a parameter in this case constrained only to Authors.

Obviously I need some mechanism to define new UI and attribute types, but assume that the system comes with a rich set of types (password, phone number, long input, short input, et cetera) from which I can build more.

My serialization layer already knows how to serialize these models. That’s part of the magic of using KiokuDB, and there are mechanisms available for DBIx::Class to perform the same mapping.

Given these models, the controller layer can create default URI mapping routes for basic CRUD operations. The HTTP verbs PUT, GET, POST, and DELETE map nicely. As well, the first URI component beneath the application itself can map to the type of model required, such that http://example.com/awesomebooksaboutperl/authors/chromatic obviously returns my author page. Without a unique identifier, http://example.com/awesomebooksaboutperl/authors/ could list all authors.

With the UI information associated with models, I don’t even have to write any templates to get a bare-bones UI. The system can use Moose’s introspection mechanism to read all of the object attributes then bind them to HTML widgets for the appropriate display type (reading, creating, and updating). This is particularly easy with a system like Moose where another parametric role can customize the appropriate elements to render based on access controls. That is, users can update their own passwords and administrators can update anyone’s passwords, but users cannot even see the password entry fields for other users. The model object decorated with this UI role can decline to make inappropriate information available to the template rendering system at all.

Even better, the UI decoration role can provide different destination output types as well, such as JSON or XML or even serialized objects themselves suitable for transportation between similar systems.

I care most about HTML for this web application, but it’s nice to have a JSON or Atom view of my data, especially because I can define another route (or perhaps you get this by adding an attribute to your models) which generates syndication information automatically; if I add a new book, it’s part of the http://example.com/awesomeperlbooks/comingsoon.xml Atom feed. Internally, the controller might map that URI to http://example.com/awesomeperlbooks/books/?view=atom;sort_by=desc;limit=10.

Whatever the other output options, I want my HTML to make generous use of CSS selectors, such that I have the option of customizing the result purely in CSS, where possible. (I don’t mind writing custom HTML, but the less of that I have to do the better.) This is because it’s possible to build a big page out of several models rendered together: each model should be able to render as a fragment, which makes Ajax applications easier to write.

Perhaps the real benefit of this system is that it can host itself, in that it’s very easy to write an administrative console which allows people to define their own models and types and widgets using the system as its own interface. I’m still likely to write my own models by hand in Vim, but I don’t mind customizing an HTML template fragment in a web browser window on a development server, at least if I can deploy the entire system as if I’d written it by hand to files in the filesystem.

I’ve discussed this system with a few people and various projects exist to make much of it work. Consider modeling, serialization, and deployment solved thanks to Moose, DBIx::Class and KiokuDB, and Plack. What’s left is some middleware, some conventions for routing and mapping, and a little bit of magic UI widget binding and default templates. I wish we’d had this great technology a decade ago, but now with modern Perl we may be able to create a postmodern fat-model MVC web framework that makes the easy things trivial, the hard things easy, and the annoying things wonderful.