Derby blog

Full-stack MVC framework making it easy to write realtime, collaborative applications

Derby v0.5.0

We’re happy to release the brand-spanking-new Derby 0.5! We’ve completely rewritten Racer, Derby’s realtime model layer on top of ShareJS. Derby 0.5 is the first full-featured web app framework with all data synced via Operational Transformation. It is also the first framework like it to support horizontal scaling to multiple servers. And while not new, Derby is still the only realtime framework with full support for server and client rendering with no extra code.

OT—not just for Google anymore

Operational Transformation (OT) is the technology that powers Google Docs, Google Wave, and a handful of other cutting-edge web apps. With OT, multiple users can simultaneously make changes to the same data. They can add, remove, and edit items in a shared list or edit the same text. Any conflicts are resolved in a way that is reliably consistent and aims to preserve each user’s intention.

Since all data in Derby uses OT, creating a realtime, collaborative text box is literally just <textarea>{_page.text}</textarea>. Every textarea and input binding uses text OT by default, and every shared object can be collaboratively modified without any extra work.

Now, any developer can put together a collaborative app that would have taken a large team months just a few years ago. To demonstrate, Ian and Joseph (with CSS loving from Sarah and Andreas) hacked together a collaborative story writing game yesterday afternoon:

Play it here!

Then, check out the source and see how it was made:

Big steps toward production ready

Derby and Racer are still very young, and a great deal of bug hunting, benchmarking, and hardening is in order.

However, this release is our first big step toward supporting production apps. With an appropriate proxy, Racer can now be used on multiple servers. It does require that clients are sticky to a given server, but it can quickly and gracefully reconnect to a new server if a connection is disrupted for any reason.

Racer now also has powerful ways to clean up client memory in complex apps. As users navigate between client-rendered pages, Derby automatically unsubscribes from and unloads documents no longer being used. It also removes old event listeners and state from the previous page. This is done intelligently, so data from the previous page still being used on the new page isn’t re-fetched from the server.

This code is hot off the keyboard, so it’s not for the faint of heart or IE. Expect bugs, and please help us to write tests!

For those of you already on the bandwagon

If you are already a Derby user, check out the previous post for information about migrating.

Migrating from Derby 0.3 to 0.5

From Derby 0.3 to 0.5, Racer has been completely rewritten. Please see the documentationn for more detailed information, but this post should help you get started with upgrading your app.

Migrating data in MongoDB

The first thing you’ll need to know is that Racer now requires keeping a journal of all operations, which is stored in Redis. If this journal gets out of sync with the data in MongoDB, it will likely break your app. The journal is the main source of what is the current state, and the database is used to store a cache of the data at a particular snapshot. When fetching documents, the snapshot stored in the DB provides more efficient retrieval and query support. To get started, you’ll need to install Redis 2.6, then initialize the journal with the data stored in Mongo.

To make getting started with existing data a bit easier, we’ve written a script to initialize the journal in Redis based on a MongoDB database. There is also a tool to inspect your data once it is initialized. For more info on using these tools, check out this video:

Conceptual differences

Overall, the core concepts in Racer remain mostly the same. However, we have made a few significant changes based on our experience with the previous version.

  • Queries use the native database format. Racer 0.3 worked by defining a named queries on the server in advance. It used a special format that was then mapped to the database’s native query language. In Racer 0.5, queries are simply expressed in the native format of the database. Currently, we have implemented a MongoDB adapter, though it is possible to implement adapters for other databases as well.
  • The previous concept of the “speculative model” has been removed, since all remotely synced data is updated via operational transformation. The speculative model was a frequent source of frustration, and operational transformation is a superior approach to solving eventual consistency.
  • Private paths have been replaced with local and remote collections. Previously, private paths were any path containing a segment that started with an underscore. This allowed you to nest private paths underneath objects that were synced, but this was always buggy and complicated the implementation. In 0.5, every document is either local to a model or remote and synced back to the server and other clients. Local collection names (the very first path segment) start with an underscore (_) or dollar sign ($). All other collection names are remote. Collections starting with dollar sign are meant for internal use by Racer, Derby, and framework extensions. Your application should define local collections starting with underscore.
  • Subscriptions are only available at the document level. To greatly simplify the implementation of subscriptions and fetches, models may only subscribe or fetch an entire document. For now, this is also a limitation of access control, and a given client may only be allowed or prevented from getting an entire document. Eventually, access control will support filtering certain fields from certain users, but this is not completed yet.
  • Racer is now much simpler and more modular. It relies on ShareJS to handle remote data syncing, allows use of any connection layer instead of depending on Socket.IO, and has much less abstraction internally.

Specific API changes

  • Derby automatically cleans up data stored underneath _page immediately before each full page render on the client. This means that all private paths and refs should generally be moved to be underneath _page
  • It is no longer possible to store data at a root private path, and it must be stored within a local document, now most likely underneath _page
  • The MongoDB adapter adds _v and _type to keep track of internal state
  • Documents may have a type other than an object at their root. If a document is an object, its MongoDB document will correspond directly to the document. If it is another type, the Racer document will be nested underneath _data in MongoDB. This means that it is now possible to use paths like hello.message, which would be stored as a document such as {_id: 'message', _data: 'Some text', _v: 1, _type: 'http://sharejs.org/types/JSONv0'} in the hello collection
  • Document id properties are no longer mapped to and from the _id in Mongo
  • Derby automatically unsubscribes from all queries and documents subscribed on the previous page unless resubscribed in the next route
  • model.subscribe and model.fetch only return an error argument. They no longer return scoped models corresponding to the inputs
  • Model events no longer correspond directly to method names. The model events are now change, insert, remove, move, stringRemove, stringInsert, load, unload, and all
  • The path wildcard synatx for model events has changed slightly, and there are now both single segment wildcards (*) and multiple segment wildcards (**) at the end of a path
  • Model functions are defined based on names rather than bundled as strings. Similar to view helper functions, model functions are now defined based on a name and initialized via model.start(). This means that model functions can use closure scope as expected
  • Stores no longer have getter and setter methods. It is now necessary to create a model from a store to perform gets and sets
  • The previous afterDb hooks have been removed
  • Access control APIs have been changed
  • model.ref no longer has a third key argument
  • Derby no longer generates and saves a file to disk for an app script bundle. Instead, the script is cached in memory and made available via middleware
  • The server configuration options have changed substantially. See the examples or generate a new project for reference

If you have other migration questions or tips, please leave them in the comments to this post.

Preview of Racer 0.5 and ShareJS for scalable Derby apps

Joseph and Nate have been sprinting on Racer 0.5 and updates to ShareJS, which will soon be powering Derby applications. Racer has been completely rewritten to make it suitable for a production environment; it now scales horizontally across multiple servers and properly cleans up memory as new data is subscribed and unsubscribed. Based on our experience with the first version of Racer, we have also redesigned the code to be much more performant for complex application models.

We are still polishing off a few features, but want to show off an early preview of the upcoming hotness:

The new version is still undergoing rapid development and needs lots more testing, but you can watch our progress and play around with it by checking out the Racer 0.5 branch on GitHub. Derby has not yet been updated to work with the new Racer API, so you can’t use Racer 0.5 with Derby just yet.

Getting Derby ready for prime time

At Lever, we believe that all web apps should be realtime. When you are looking at a page and it no longer reflects what someone else is seeing, that is a bug. In addition, we believe that web apps shouldn’t feel inferior to their desktop or mobile counterparts—they should load super quickly, respond to interaction instantly, and work offline without interruption.

We are creating Derby and Racer, intending to build the best technology to power the next generation of innovative web applications. We also aim to make their development accessible for individual developers, startups, and large teams.

Welcome, Joseph!

We’ll get back to our plan for achieving these ambitious goals, but first want to introduce Jospeh Gentle, who recently joined the Lever team. Joseph has already built ShareJS, perhaps the easiest open source library for adding Operational Transformation (OT) to a web app. OT is a method for enabling collaborative text editing and other forms of collaborative editing. Similar algorithms power Google Docs, Google Wave, Etherpad, and a handful of other advanced applications. Joseph was also previously a member of the Google Wave team, and is leading development of the next generation of our realtime infrastructure.

In case you haven’t been following along yet

Looking at the landscape of web frameworks and libraries a couple of years ago, we realized that while more and more applications were integrating realtime features, developer options were lacking.

We started by building Racer, the realtime data synchronization engine at Derby’s core. We focused first on innovating in the APIs and capabilities of the system, since there was little precedence for a generic realtime app data engine. There were many fantastic examples of realtime applications, such as chat, Google Docs and Wave, multi-player games, and financial platforms. However, they were by and large created by large organizations with massive resources. In contrast, clear patterns, standards, and frameworks had been established for server-rendered web apps and REST-backed client apps. Such tools were widely available to a variety of developers, including individuals and small companies.

On top of Racer, we built out Derby as a full-stack application framework with the capabilities of both a client-side single page application and a server-rendered Express app. Derby remains one of the very few solutions for taking advantage of full client and server code reuse, and it enables super fast page loads paired with immediate client-side updates.

We are now building a next generation application for companies to better manage their hiring. Our technology enables us to provide features unlike any other application in our market. At the same time, personally developing a full enterprise application helps us to test and improve the platform in a tight iteration loop.

Establishing a solid foundation

As with most new projects, we couldn’t foresee everything that would and wouldn’t work well. We wrote a whole lot of features and came up with a many new concepts as we created Racer and Derby. Many of these new ideas were fantastic, and many of them didn’t work well or made the implementation too complex. We also didn’t write the first version of the code to scale across multiple processes, since our focus was on innovating at the API level.

The next step in our development is simplifying down to the good ideas and making a horizontally scalable system. At the same time, we will be coordinating our work on ShareJS and Racer to better both projects.

The biggest part of the simplification work is moving away from Racer’s current fully arbitrary JSON structure object and replacing it with a mirror of Mongo’s collections and documents. The model API will stay the same, but we’ll be enforcing a collection and document topology consistently. This is already a requirement when using the Mongo adapter, and it is similar to how most other MVC frameworks structure their models.

Each collection will be a map from document ID to document data. Aside from access control filtering, each document will be present in the model or it won’t. Instead of trying to select out subsets of documents like a server-side Mongo query does, Racer will simply fetch each entire document that matches a query. We’ve learned that merging arbitrary document fragments from multiple queries introduces an excessive amount of complexity, and will likely be a big source of bugs without being particularly useful.

Over the next few months, we’re rewriting a great deal of Racer and refactoring ShareJS. Joseph is leading this effort and working on it full time. ShareJS will remain an independent project for those who only want to use its OT capabilities alongside their current stack. Integrating OT will help us to simplify some of the gnarlier parts of the model code while maintaining Racer’s simple API. What’s more, this will bring us much closer to proper offline support by default.

Racer will sit on top of ShareJS. ShareJS will provide the database layer, communication between servers and clients, and OT algorithms, and Racer will provide the model interface. Horizontal scaling support will be added into ShareJS itself, which is going to undergo a huge refactor to make the pieces fit nicely together. The highlights are:

  • Splitting ShareJS into three smaller projects: OT types, a scaling database backend, and a glue project to expose an OT database to a web browser.
  • Moving to a collection/document view instead of a flat set of documents.
  • Support for live database queries.

You can find a much more detailed description of the backend changes to ShareJS on Joseph’s blog.

These changes will be implemented in a bottom-up manner, so the changes to Racer will land last. In the meantime, we have been fixing bugs as they come up, but much of the current code is going to be removed or completely replaced over the next few months.

Let’s meet!

We’d also like to engage in a more open dialog with our community. We’ll be hosting a meetup next Wednesday, April 3 at our office in San Francisco. Hope to see you there!

Now hiring: Join the Derby core team at Lever

In addition to our continued development of Derby, Brian and I have started a company called Lever. Lever is redefining talent software, starting with applicant tracking for hiring. With Lever, we aim to set the standard for next generation web apps, and we are hiring software engineers who share our ambitions.

Innovate in web technology while solving real world problems

We are building our product with Derby both to benefit from its unique capabilities and to put the technology to the test. Working at Lever means moving the ball forward on a product that our users love at the same time as contributing to the future of web app development. A big part of why we are hiring is to continue innovation on the framework itself. We have learned an incredible amount about how we can make Derby better by building a sophisticated app on top of it, and we are excited to make improvements.

You might wonder why we chose to write new software for tracking hiring. After interviewing managers at Square, Twitter, Yammer, AirBnB, Stripe, and other great companies, we learned that applicant tracking tools are ripe for disruption. Limited visibility into hiring data frustrates engineering managers trying to grow their teams, and difficult to use tools frustrate interviewers and recruiters alike. This is a unique opportunity where companies are already eager to adopt new enterprise software.

Lever’s culture is rooted in pairing technology innovation with user-driven design. We work right next to 50 of our users, directly learning how we can make each of their days a little better with every code push. Rather than defining our product by a list of features, we have reimagined applicant tracking around the candidate-company relationship.

Sweet office in downtown San Francisco

We are located in our pilot customer’s office in San Francisco. In addition to the best user feedback a team could ask for, we also get three free meals a day, free snacks, an office full of talented engineers and designers to learn from, and a spectacular roof deck.

Learn a ton, and build a fantastic company

As one of the very first hires at Lever, you will take a defining role in the company. You will have flexibility to contribute across the technology stack.

  • Design and implement a wide variety of product features
  • Be a primary contributor to Derby, Racer, and other open source projects
  • Optimize search quality
  • Architect backend infrastructure
  • Administer servers
  • Create innovative data visualizations
  • Test and deploy every day

Technology

Derby & Racer, Node.js, HTML/CSS, Socket.IO, MongoDB, Solr, Git, and AWS

Location

San Francisco, CA

Apply

jobs+blog@lever.co

Derby v0.3.1

Derby v0.3.1 is out with support for LESS compilation, Express 3, easier to install examples, and a number of bug fixes.

LESS

Thanks to contributions from Zach Smith, Derby will now compile styles written in LESS as well as Stylus! We’re super excited to have new contributors, and it is great to see people using and improving upon the framework!

Express 3

We have updated Derby to use Express 3, which has recently been released in alpha. Express 3 is different in a number of small ways, so we wanted to get new projects started with the current APIs as soon as possible.

Easier to get examples installed

A number of people ran into issues running the derby examples on their machines, because of how they were included. They are now in a separate GitHub repo with much better installation instructions. Hopefully we have resolved all of the issues running Derby on Windows as well.

Our take on Derby vs. Meteor

We have received a number of requests for a comparison. I’d like to thank Nick Retallack, who already did a great job summarizing a lot of these points on our Google Group. Many of his observations are included here.

A bit of the origin story

First of all, I should mention that Brian and I first met the Meteor team last November when we demoed an early version of Derby at the Keeping it Realtime Conference hosted by &yet. When we met, the Meteor team had already started on their framework, and the similarities were obvious. Some time before, I had also met with David Greenspan (who recently joined Meteor) to pick his brain and learn more about his experience writing Etherpad. Our two teams like each other, we’ve been keeping in touch, and we have all learned a lot from each other.

Our teams share a similar vision for a world where all applications are realtime and collaborative by default. We as well as a number of other other developers all had a revelation about a year and a half ago—the way that web apps are currently built makes it painfully difficult to create the best user experience, where data dynamically update everywhere.

Brian and I first discussed this vision with each other a year ago. I was coming from working as a Product Manager on the Google Search team, and he had been working on a number of open source Node.js projects including Mongoose and everyauth. I was interested in writing a framework, because I believed it was the best way to write an app with the kind of performance that I desired, and Brian felt that the Node.js community needed an easy to use framework for those developers who preferred tools like Rails and PHP.

The Holy Grail: server and client code sharing

From Google Search, I learned that in order to have a responsive web app, it is critical to both render pages on the server and in the browser. Google can afford to do this, because it has so many engineers and speed is of paramount importance to search. To achieve fast page loads as well as fast page updates, Google implements pretty much the entire Search front-end in C++ as well as JavaScript. They have been working on a number of projects to make this easier over the past couple of years, but needless to say, such a process is too expensive and slow for a startup or a complex app.

Gmail, Twitter, and other sites rendered only in the client were painfully slow to load. Twitter went through the full circle on this—they started as a server-rendered Rails app, rewrote to render everything in the client, and then realized it took their users 4 seconds to be able to read a 140 character message. Now, Twitter renders half the page in Scala on the server, and they render the other half in JavaScript in the client. This both makes it more difficult for them to maintain a code base in multiple languages and means that the least important items on the page pop-in a few seconds after page load. People are very sensitive to movement, so in effect, they are distracting their users with the least important information after every page load.

Thanks to the awesome re-invention of JavaScript performance starting with V8 and the ability to easily create JavaScript servers with Node.js, lots of developers are excited about the notion of using JavaScript for both server and client development. But despite finally using the same language, few Node.js applications to date deliver on the amazing ability to share code. Even with a common language, there are still a lot of issues including very different latency and connection stability, separate ways of dealing with URLs on the server and in the browser, the existence of a DOM in the browser and not on the server, and direct access to a filesystem only on the server.

Let’s get down to business!

So after all that, our teams decided to go in similar directions in some ways, but very different in others. Derby’s goal is to make it possible for any developer to write apps that load as fast as a search engine, are as interactive as a document editor, and work offline. And to quote Geoff, Meteor’s goal is to create “a ‘mass-market’ app platform that could be used for 90% of the sites on the web, and for that 90%, make web development faster and within reach of more people.”

GPL vs. MIT

Currently, Meteor is only available under a GPL license. This means that you must contact them and arrange for a commercial license if you do not wish to release your source under the GPL or another compatible license.

Derby, Racer, and all other components of our framework are released under the permissive MIT license. Thus, you can do pretty much anything you want with it (other than sue us), and we can’t stop you or change our minds later.

I’m no lawyer, so don’t take any of what I just said as legal advice; I don’t have a crystal ball or anything. (Yes, a lawyer once told me to say that.)

Meteor packages vs. npm

Meteor has created their own package system and means for distributing packages.

Node.js modules are clearly defined by the CommonJS format and the built-in module APIs. However, it does not include a means for distributing modules. In the early days, Node.js had a few competing package managers, but pretty much everyone has now finalized on using npm. It is so universal, that npm is now distributed directly with the official Node.js binary distributions.

Frankly, this is the main thing that concerns me about Meteor. It is likely that many people trying Meteor will never have used Node.js before, and they won’t appreciate the great benefits of distribution via npm. I and many other developers view npm as one of Node.js’s core strengths, and I would be very sad to see it weakened by an incompatible package system.

Like pretty much every other Node.js project out there, we primarily distribute Derby, Racer, and all plugins for these projects as npm modules. We will continue to break out our projects into smaller modules so that they can be better reused by others. Perhaps you don’t use Derby, but you need to parse some HTML. We had to write a simple and fast HTML parser for our templates to work, so you’ll be able to use just that in any Node.js project. Connect is great example of a project that has a rich ecosystem of middleware designed to work with it that are all distributed via npm, and we hope to follow in the same pattern.

Compatibility with other libraries

Meteor makes it possible to replace parts of it with other libraries. It is especially flexible when it comes to substituting in client-side libraries, though those libraries must first be wrapped as a Meteor package.

Meteor mostly manages the creation of a server in a custom way. This makes it easy to create and deploy a Meteor app to their hosting service, but it makes it much harder to use Meteor as a module of a more traditional Node.js server.

In contrast, Derby is just a normal npm module that you can add to any Node.js server. If you want to use any of the 8900 packages in npm (at the moment), simply add them to your package.json file and npm install away!

Derby does not provide any sort of hosting solution, though there are lots of great hosting services for Node.js out there. We will provide better instructions for how to get Derby apps up and running quickly on a public server in the future.

Racer (the realtime engine powering Derby’s models) is a separate module, and it can be used independently of Derby. You could hook pretty much any UI layer up to Racer and handle the methods that it emits as the model is updated. Racer is built on top of the very popular Socket.IO module, which you can use directly if you need to.

Derby also uses popular modules at its core, especially Express for routing and Browserify, which can bundle up most Node.js modules for the browser automatically.

MongoDB API vs. Racer methods

Meteor takes the MongoDB API all the way to the browser. The MongoDB API is powerful and easy to use from JavaScript. It has also been around for some time, so there are well established patterns for using it do most things that an application might need to do. A lot of people seem to think this is a fun way to build apps, and it reduces need to understand multiple layers of abstraction. Some have expressed concerns over the security implications, but I think we should reserve judgment until the Meteor team has more time to address what they think is the best approach for authentication and authorization.

Racer has its own API based around a set of mutator methods and paths. This API maps pretty much 1:1 with any document store, including MongoDB. This has some pluses and minuses, but we believe it is the right choice for a few reasons:

Conflict resolution

Paths and methods map well to conflict detection techniques that we think will be one of the major benefits of using Racer. For now, our default mode is last-writer-wins, which is equivalent to how Meteor saves data. However, we have preliminary implementations of conflict resolution via Software Transactional Memory and Operational Transformation methods. Such techniques will make it possible to use a Derby app offline and then resync correctly. It will also make it possible to easily add features like Google-Docs-style realtime collaborative text editing in any text box.

Datastore portability

Our paths and methods are granular enough to take advantage of the capabilities of most datastores, but it is possible to switch from one datastore to another or to use them with multiple datastores simultaneously. Swapping in Riak, Postgres, CouchDB, or another service should be straightforward without modifying application code.

Efficient PubSub

Paths map well to PubSub, which is how we propagate changes in realtime. In contrast, Meteor’s LiveMongo implementation simply writes to the database and polls it frequently for the data in use by every connected client. The advantage to polling is that it can support subscriptions to pretty much any kind of Mongo query, but we have implemented most queries and query PubSub without needing to poll the database. We have no real-world evidence yet, but we expect PubSub to scale much better than database polling.

API plugins

It will be possible to create plugins that act like datastores, even though they are communicating with a backend service or a 3rd party API. Database adapters are built on top of a system of routes that can be used in a custom manner.

Server rendering and shared routes

Derby is one of the only frameworks designed to run all rendering and routing code on the server and in the browser. It’s even possible to use Derby to render static pages that share templates with dynamic apps.

This means you get crazy fast page loads with no effort. Even for a simple client-rendered app, pages typically take a second or two to load and then be displayed to the user. Simple Derby apps can fire the onload event in less than 200ms. As apps get more complex, server rendering has an even more drastic effect.

Lots of big companies including Google, Amazon, and Facebook dedicate massive resources to optimizing page load time, because it directly translates into better conversion rates and more revenue. I died a little bit inside the day Gmail added a loading progress bar. Don’t do that.

In addition, server-rendering is critical for search engine optimization, and it provides accessibility to screen readers and browsers without JavaScript.

Derby exposes routes as an Express middleware on the server, so you can use it alongside other server-only Express routes for tasks like uploading files, and you can even write multiple apps that handle different sets of routes on the same Express server. For example, you might write a separate admin app or a separate mobile app that doesn’t load all of the code for a desktop browser app.

While Derby gives you all of the speed and accessibility advantages of a more traditional multi-page app, it also creates fully optimized single-page applications that can render any template or route in the browser. Client-side routes simply render any links that they can without doing a page refresh; you don’t have to call methods on a special browser-side routing API. Just put a normal HTML link on the page, and Derby will render it client-side.

Meteor does not have these features yet, though they recently rewrote their templating engine to use strings so that it is now possible for them to add.

Model-view data bindings

Meteor uses a reactive functional programming approach to updating templates that could potentially work with any template engine. As a template is rendered, all of the data that is used to render that template is assumed to be an input to that chunk of the UI. Later, when that data changes, the template is re-rendered, and its output is compared with the DOM. Meteor then patches up the DOM to apply the minimum required changes. Effectively, every single variable and function in a template is bound. The big advantage to this approach is its simplicity and ability to support any template language.

In contrast, Derby has its own template language based on Handlebars. Bindings are declared explicitly, and the HTML of the template is parsed in order to figure out which parts of the DOM need to be updated. Unlike Meteor’s, Derby’s bindings are two way; when form elements are bound, the model is automatically updated as users type in text inputs, check checkboxes, and select items in drop downs. In addition, it is possible to bind everything or to optimize the performance of templates by only binding what is necessary. Outputting non-bound data may also be preferred if a developer wishes to update the view via manual DOM manipulation in special cases.

DOM event handlers

It is a relatively minor difference, but Meteor uses CSS selectors on specific templates, and Derby uses HTML attributes within templates to declare event handlers. jQuery and Backbone have popularized the CSS selector method of scoping event handlers. Knockout and Angular use HTML attributes.

We prefer the HTML markup approach, because it is more work to figure out what CSS selector matches the correct elements, and it is reasonably likely that HTML structure will change and silently break CSS selectors. With the markup approach, it is less likely that function names will be changed inadvertently, and an error will be thrown when the function can no longer be found.

Derby also has an innovative approach to event bubbling that is more efficient and intuitive compared to typical event bubbling. Instead of bubbling up the entire DOM to the root documentElement for every event, Derby will stop bubbling once it finds an element that is bound to the event. Like with routes, bubbling can be continued by calling a next() method passed to the event handler.

Fibers

Meteor uses the Fibers extension for Node.js that makes it possible to write synchronous looking code on the server instead of the more common callback style used by most Node.js projects. Some prefer this way of writing asynchronous code, others strongly disagree with it. There are a number of arguments on both sides.

This is very contentious issue at the moment, and we prefer to stay out of it. A very small percentage of modules in npm are written this way, so we have stuck to the more traditional Node.js style of callbacks.

Wrapping it up

You can achieve many of the same things with both frameworks. Both are powerful tools for implementing features that are very difficult to write in more traditional web frameworks like Rails and PHP.

Derby has been more focused on making sure it can support advanced features like conflict detection, offline clients, and extremely fast rendering. We are also more focused on compatibility with other modules in the existing Node.js ecosystem, and we have a permissive open source license.

Ultimately, we are all trying to create the best tools for developers, and competition can help to breed innovation. We are happy to see other approaches like Meteor, SpaceMagic, and Firebase, because we will all learn from each other. We will end up with better tools, a better web, and better experiences for users.

If you’re interested in what we’re working on, follow us on Twitter and GitHub.

Derby v0.3.0

This is a big release for us—Derby finally has support for automatic persistence! Initially we have support for MongoDB, and it will be possible to swap in anything else through use of a different adapter.

In addition, we have added support for subscriptions to queries and made big upgrades to the power of the HTML templates. Templates now support reactive helper functions, and we have replaced partials with more flexible HTML components.

By the way, you may have noticed that we skipped v0.2. We are following the common convention major.minor.patchlevel where odd minor versions are unstable. Node.js, the Linux kernel, Redis, and lots of other projects follow this convention. Since odd minor versions are unstable, patch level versions will likely break APIs. Patch level updates to even numbered versions will only include bug fixes. The first such stable version will likely be v0.4.0.

Want some database with that realtime syncing?

1
2
3
4
// No persistence
app.createStore({
  listen: server
})
1
2
3
4
5
6
7
// Persistence-a-go-go
derby.use(require('racer-db-mongo'))

app.createStore({
  listen:  server
, db:      {type: 'Mongo', uri: 'mongodb://localhost/database'}
})

No, seriously, it is two lines of code.

All of the db adapters will be released as separate npm modules. As with all npm modules, you need to add a dependency to your package.json file and then re-run $ npm install in order to require it.

Behind the scenes

Racer (the realtime engine powering Derby’s models) describes all data mutations in terms of a method, path, and some arguments. These methods, e.g. set, push, etc., can be thought of like the HTTP verbs, such as POST, PUT, and DELETE. Model paths are like URLs to a specific piece of data, and the method arguments are like the content of a request. Just as a REST API can be defined in terms of HTTP verbs and URLs, Racer has a routing system for mutation methods and paths.

By default, Racer uses a basic in-memory datastore that defines these store routes for each mutation method and path pattern. Database adapters like the Mongo adapter do the same thing, implementing each method in a database-specifc way. Because these routes can handle any method on any path, no additional server code needs to be written in order to persist data.

It is possible to add custom store routes for specifc paths if an application connects to an external API. Such routes could be provided as a generic plugin for Racer that makes it dead simple for other developers to use an API with Racer’s automatically synced models. More documentaion on store routes will be coming soon.

So where is the data saved?

Racer paths are translated into database collections and documents using a natural mapping:

1
collection.documentId.document

All synced paths (anything that doesn’t start with an underscore) must follow this convention. In other words, all model data stored at the first two path segments should be an object and not a string, number, or other primitive type.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// Examples:
model.set('todos.id_0.completed', true)
model.set('rooms.lobby.messages.5.text', 'Call me')
model.set('meta', {
  app: {
    title: 'Hi there'
  , author: 'Erik Mathers'
  }
})

// The first and second segments in root paths must be objects
model.set('title', 'Hi there')      // WARNING INVALID
model.set('app.title', 'Hi there')  // WARNING INVALID

// However, any type may be stored at any private path, which
// starts with an underscore and is not synced back to the server
model.set('_title', 'Hi there')     // OK

The document’s id (the second path segment) is automatically added as the id property of the document, so that it can be retrieved from the datastore.

1
2
3
4
5
6
7
8
9
10
11
12
model.set('meta', {
  app: {
    title: 'Hi there'
  , author: 'Erik Mathers'
  }
})
console.log(model.get('meta.app'))
// Logs: {
//   id: 'app'
// , title: 'Hi there'
// , author: 'Erik Mathers'
// }

Queries — we do ‘em live!

Racer has always had granular PubSub based on path patterns. In addition, it is now possible to specify subscriptions in terms of a more expressive query syntax. These queries are similar to most database queries, except that they can automatically detect when new data matches and then sync that data to all appropriate clients.

1
2
3
4
5
6
7
8
get('/:groupName', function(page, model, params) {
  var groupPath = 'groups.' + params.groupName
    , todosQuery = model.query('todos').where('group').equals(params.groupName)

  model.subscribe(groupPath, todosQuery, function() {
    ...
  })
})

In this example, a query is used to limit a client’s subscription to only todos that have a property group with the same name as the current group. The query syntax supports most common query operations, such as key lookups, numerical comparisons, sorts, and limits. See the documentation for more info.

A quick note on syntax changes

This release changes a number of items in Derby’s template syntax:

  • Bound template tags used to be wrapped in double or triple parentheses instead of curly braces. Instead, they are now wrapped in single curly braces. Now a bound variable looks like <input value="{.text}">. All of the parentheses started to get confusing with the addition of view helper functions, and the new syntax is much easier on the eyes. To output a left curly brace in HTML, use the entity &#123;. Conveniently, it’s an easy one to remember.
  • Instead of triple curlies, there is now an unescaped keyword. For example, what was previously {{{rawHtml}}} is now {{unescaped rawHtml}} and what was previously (((rawHtml))) is now {unescaped rawHtml}. This is more readable and keeps the braces consistent.
  • Also for readability, path aliases in templates must now be proceeded by an as keyword. For example, {{#each items as :item}}.
  • Handlebars-style partials have been removed in favor of a more powerful component system that looks like custom HTML tags. More on this below, but what was previously {{> nav}} is now <app:nav>. Inside of HTML attribute values, developers should now use view helper functions instead of partials.
  • Triple curly braces have been re-purposed for the much more awesome function of macro template tags inside of components. More on this below.
  • The hacky disabled="!((active))" syntax that was previously supported only for boolean HTML attributes has been removed in favor of the built-in not helper function. This should now be disabled="{not(active)}".

We realize changes like this are annoying, and we don’t expect to be making any more major changes to the template syntax after this. We have some additional features planned for components, but they should be backwards compatible with this current release.

Sometimes views need a little help(er)

We follow the logic-less templating approach of Handlebars and Mustache, which helps to better organize controller code. However, we found that we kept having to add data to the model just for the purpose of binding computed values. This got tedious, and didn’t work well with computed properties of each item in a list.

Therefore, we added reactive view helper functions. These functions get evaluted when rendering as well as whenever their inputs change. In addition, they can work as both getters and setters. This is super useful when binding to form elements, such as selected options or radio buttons:

1
2
3
4
// Remove all whitespace from a string
view.fn('unspace', function(value) {
  return value && value.replace(/\s/g, '')
})
1
2
3
4
5
6
7
8
{{#with home}}
  <h1 style="color:{unspace(.title.color)}">Welcome in {.title.color}!</h1>
  <select>
    {#each .colors}
      <option selected="{equal(.name, home.title.color)}">{{.name}}</option>
    {/}
  </select>
{{/}}

There are two default view helper functions, equal and not, that are aways available. It is also possible to define custom view helper functions, such as unspace in the example above. The equal and not functions can act as both getters and setters. In this example, when the page renders, the option with a name equal to the value of home.title.color will have a selected attribute and the others will not. When the user selects a different option from the drop down, home.title.color will be set to the value of the option that is now selected.

Components are the new partials hotness

Previously, Derby templates only supported simple partials like Handlebars. We have replaced partials with a much more powerful concept of components that can take attribute arguments and even different HTML content. Components are defined in the same way that partials were previously defined, but they are included in other templates like custom HTML tags. When Derby parses the templates, it replaces these custom component tags with the components’ contents, and the custom tags do not appear in the HTML output.

1
2
3
4
5
<Body:>
  <h1><app:greeting></h1>

<greeting:>
  Hello!

For now, all component includes start with the namespace app. This prevents potential conflicts with normal HTML tags, and it makes it clear which HTML tags are components that should be replaced. In the future, it will be possible to define component libraries with their own namespaces.

Literal values or variable values can be passed to components. These component attributes are available through “macro” template tags, which have triple curly braces. Macro template tags only reference component attribute names, and regular template tags (with one or two curly braces) only reference names from the model or context object. It is possible to use macro template tags to conditionally render any HTML content or other template tags.

1
2
3
4
5
6
7
8
9
<Body:>
  <h1><app:greeting message="Hello" to="{_user.name}"></h1>

<greeting:>
  {{{#if to}}}
    {{{message}}}, {{{to}}}!
  {{{else}}}
    {{{message}}}!
  {{{/}}}

produces the same output as:

1
2
3
4
5
6
7
8
<Body:>
  <h1>
    {#if _user.name}
      Hello, {_user.name}!
    {else}
      Hello!
    {/}
  </h1>

By default, all components are void HTML elements. This means that they must only have an opening tag and no closing tag, just like the <img> and <br> elements. A component can be defined as nonvoid, which means that it must have both a starting and a closing tag. Nonvoid components have access to a special content macro that makes it possible to pass HTML content to the component. For example:

1
2
3
4
5
6
7
8
9
10
<Body:>
  Welcome!
  <app:fancyButton>
    <b>Click me {{#if isUrgent}}now!{{/}}</b>
  </app:fancyButton>

<fancyButton: nonvoid>
  <button class="fancy">
    {{{content}}}
  </button>

produces the same output as:

1
2
3
4
5
<Body:>
  Welcome!
  <button class="fancy">
    <b>Click me {{#if isUrgent}}now!{{/}}</b>
  </button>

Components make it possible to write more expressive templates and reduce repeated boilerplate. We will continue to extend components and add support for creating reusable component libraries.

Derby v0.1.10

This release significantly improves templates and event bindings. It also fixes a ton of bugs, so get on it!

Namespaces to the rescue!

Creating apps with many pages is now simple thanks to the addition of namespaced templates. The new template import syntax makes it easy to break up templates into multiple pages. Then, a namespace can be passed to view.render() to specify which page to render. Namespaces could also represent different application states or be used to break up really complex pages into multiple independent sets of templates.

home.html

1
2
<Body:>
  Welcome to the home page

contact.html

1
2
<Body:>
  Give us a call sometime

index.html

1
2
3
4
5
<import: src="home">
<import: src="contact">

<Header:>
  <h1>Awesome town</h1>

index.js

1
2
3
4
5
6
7
get('/', function(page) {
  page.render('home')
})

get('/contact-us', function(page) {
  page.render('contact')
})

Growing out the Mustache

Derby templates have moved from Mustache style conditional blocks to Handlebars style. This is a relatively minor change, but we think it makes templates more readable. This is helpful for those less familiar with Mustache syntax and for reading templates that others have written.

What was previously:

1
2
3
4
5
6
7
8
9
<Body:>
  {{#users}}
    <h3>{{name}}</h3>
    {{#active}}
      Thanks for your active support!
    {{^}}
      We'd love to see you more often.
    {{/}}
  {{/}}

Now is written:

1
2
3
4
5
6
7
8
9
<Body:>
  {{#each users}}
    <h3>{{name}}</h3>
    {{#if active}}
      Thanks for your active support!
    {{else}}
      We'd love to see you more often.
    {{/}}
  {{/}}

For more detail, see Template Syntax in the docs.

Event bubbling done right

Derby makes binding to DOM events fast and easy via the x-bind attribute in templates. Previously, events only were detected when they were the target. Now events bubble up to parent elements, and they emit the event object with a reference to the target as well as the element that was bound via x-bind.

Unlike the DOM Event Model, the event is only called on the first handler that matches. It will only continue bubbling if a next() function is called, similar to how routes can pass control to the next route. This is a more efficient and less error-prone way of bubbling than requiring listeners to call a method like e.stopPropogation().

1
2
3
4
5
6
7
8
9
10
11
<!-- The root element will handle a click anywhere in the window. -->
<Root:>
  <html x-bind="click: deselect">

<Body:>
  {{#each items}}
    <div x-bind="click: select">
      <h3>{{name}}</h3>
      <img src="{{image}}">
    </div>
  {{/}}
1
2
3
4
5
6
7
8
9
10
exports.select = function(e, el, next) {
  var id = model.at(el).get('id')
  selectedId.set(id)
}

// There is no need to check the target of the click event, since it
// will only bubble up if the click was outside of one of the items.
exports.deselect = function() {
  selectedId.del()
}

Even the error messages are fast

For faster development, we added live-updating as files are edited in the last release. However, any errors would still kill the server and require a manual restart. Template and CSS error handling is now much improved, and errors with either of these are displayed in the browser as files are saved. Even better, the page updates and the error message disappears as soon as the error is corrected.

Derby v0.1.9

The latest release of Derby includes easier installation without Redis; automatic reloading of scripts, templates, and styles; better subscription functionality; and lots of bug fixes.

Modularity FTW

In preparation for adding persistent storage, we have finished a major refactor of Racer, the realtime model engine for Derby. The latest release includes the ability to plugin various functionality, and there will be separate plugins for different database, journal, and PubSub adapters. We already have preliminary plugins for a MongoDB database, Redis journal, and Redis PubSub, though we are still working on some bugs.

Another awesome benefit of the refactor is that Redis is no longer required by default, and you can now start playing around with Derby after a simple npm install -g derby.

Command-S and kick back

Thanks to Up and Node.js v0.6’s fs.watchFile, Derby now supports automatic page updates in development. Derby’s new derby.run method starts a server via Up and reloads the server whenever any script files in the project are saved.

1
require('derby').run(__dirname + '/lib/server')

In development, the browser will automatically reload when it reconnects and notices that the server has a new version. Simply generate a new project via

1
$ derby new project-name

and it will be set up to use this feature.

In addition, saving style files will automatically recompile CSS and update connected browsers. Saving template files will re-render the page in connected browsers. Get a big monitor and tweak those styles like it’s nobody’s business!

Subscribe – now with less whack!

The model.subscribe() method has been simplified to take advantage of scoped models. Previously, the format for subscribe was the somewhat wacky:

1
2
3
4
// WARNING: Old API
model.subscribe({_room: 'rooms.' + roomName}, function() {
  console.log(model.get('_room.stuff'))
})

The new format is more clear:

1
2
3
4
model.subscribe('rooms.' + roomName, function(err, room) {
  model.ref('_room', room)
  console.log(room.get('stuff'))
})

There is also a new method model.fetch(), which has the same format as subscribe. However, it only gets data from the store and sets it in the model. It does not create any subscriptions to ongoing updates, and it does not require the use of PubSub.

Subscribe and fetch will also support queries, though the implementation is still being finalized.

Seriously, we are going support MongoDB soon

I know this has been a long time coming—persistence is just around the corner. We have it mostly working with the Todos example, but we are still working through some bugs. Note that adding persistence will simply require configuring the store in the server file. It will look something like:

1
2
// No persistence
app.createStore({ listen: server })
1
2
3
4
5
6
7
8
9
10
11
12
13
// Persistence-a-go-go

derby
  .use(require('racer-journal-redis'))
  .use(require('racer-pubsub-redis'))
  .use(require('racer-db-mongo'))

app.createStore({
  listen:  server
, journal: {type: 'Redis'}
, pubSub:  {type: 'Redis'}
, db:      {type: 'Mongo', uri: 'mongodb://localhost/database'}
})

It shouldn’t require any changes to your application code, and it won’t affect the model API, so go ahead and prototype with the current version of Derby.

Note that if you try out this out now, bad things will happen. But hey, the code is on the Internet, so someone is obviously going to do it. Just know that it isn’t supposed to work yet.