How to be a Freelance Programmer, Part 1

I've attended lots of events in my day, and have had to talk to lots of strangers. The conversations usually start off well enough; some delightful repartee about cocktails or an incisive personal anecdote. We laugh. Once the chuckles fade, however, there is a pause, after which comes that well-worn question:

So what do you do? 

For a long time I wasn't sure how to answer. RestlessDev wasn't really a startup in the traditional sense anymore, since it wasn't trying to fulfill some grand vision or look for funding. After we stopped building our Facebook apps a few years ago, I used to tell people that I was semi-retired, my tongue-in-cheek way of saying I was still figuring it out. Back then I would work random gigs whenever I needed money and just have fun the rest of the time. San Francisco was a lot cheaper then, particularly when you live in the Mission in a rent-controlled apartment with 3 roommates.

Eventually I got more ambitious and decided to take my business more seriously. I started looking for bigger projects and began to understand what parts of my work I enjoyed most. Most importantly, I realized that freelancing was a viable alternative to having a real job, if managed correctly.

1. What is Freelancing?

Wikipedia has a great definition:

A freelancer is a person who is self-employed and is not committed to a particular employer long-term.

Freelancing can take on several different forms:

  1. A freelancer can be a "contract employee," indistinguishable from other employees except for pay rate and benefits. Sometimes they are only around for a particular project, and other times they are around for years.
  2. A freelancer can act as a "consultant," where they are hired through a third party staffing company, often to act as a contract employee for some other organization. While working through the staffing company, the freelancer can be a 1099 employee or W2 employee of the company, with each paying a different rate. Once the assignment is done, the freelancer is free to work for other companies.
  3. A freelancer can be hired to build something or perform some function independently, working outside the company's offices. Often the company is just looking to get something completed without wanting to handle or worry about the day-to-day process.

The important part is that a freelancer is not really beholden into any one company, but moves between them to jump on new projects and opportunities as they pop up.

There is a very subtle difference between a "freelancer" and a "contractor," at least in my understanding. I tend to think of a contractor as someone who is compensated on a similar scale as regular employees, where a freelancer is someone who is paid a premium over regular employees. For instance, if a regular employee at a company is paid $125k per year, a contractor might get paid the hourly equivalent, something like $65-$75 per hour. A freelancer would get paid more like $100-$125.

The difference isn't necessarily that the freelancer is that much more awesome than a regular employee. It is a reflection that the freelancer is only being brought in for a month or two at a time, whereas the contractor is brought in for longer engagements. The higher rate is due to the fact that the freelancer is available right now to fix the client's problem, and that they have the ability to slot in without much ramp-up time and no ongoing commitment past the short project term.

2. Is Freelancing Right for Me?

Freelancing is not the right choice for everyone. Here are some questions that might help you decide if it's right for you.

  1. Are you an independent worker? Are you productive even if there isn't anyone telling you what to do or looking over your shoulder? What have you done on your own in the last year?
  2. Are you good at saving money? Would you be ok if you didn't work for 3 months? How about 6?
  3. Do you have a customer service orientation? If a longstanding client calls on Friday at 6PM and says there is an emergency and they need some work done over the weekend, what would you say?
  4. Can you minimize your fixed costs if need be? With an expensive mortgage or school loans, it can be hard to negotiate breathing room.
  5. Is your independence more important to you than financial security? Would you turn down a full-time job that paid 30% more than you made the year before just because you wanted to stay your own course and roll the dice on making even more?

If you didn't answer each of these affirmatively, freelancing might not be for you.

Freelancing is an exciting and ever-evolving career. In the course of a year you can work for many different companies on many interesting projects. You can have ample free time to take trips and vacations if you are so inclined since there are gaps with work. If you aren't working onsite, you can set your daily schedule however you'd like.

It creates a different sort of lifestyle where (at least for me) it is easier to balance all of the competing priorities in life. For instance, you can just take random days off to hang out with your significant other around the city and visit all of the cafes that are too packed to even enter on Saturday and Sunday. If you have a family emergency, you can just fly back home to take care of it, even it you'll be gone for weeks. Your interactions with your clients are a lot different than they are with a regular employer, and I find that they are more honest. In San Francisco's startup culture, the balance between work and life can be blurred and many employers take advantage of that to get extra concessions from employees. It's much harder to do that with a freelancer, where the terms of the engagement are explicitly negotiated.

Like anything worth doing, though, it is a high-risk, high-reward endeavor. You'll have lots of days when you'll wake up and have nothing going on, and no idea where the next check is coming from. One needs to be comfortable with uncertainty to get through it, at least at the beginning.

3. Sounds Great. How Do I Get Started?

Before you get started with freelancing, there is some groundwork to be done. 

Start Saving

Your bank account is extremely important when freelancing. It is the only thing that prevents you from having to work with crappy clients or accept crappy rates. Ideally, you should have started building your savings years ago, but if not, you should start today.

Put Up a Website/Portfolio 

If you're going to start looking for freelance gigs, particularly for programming work, it's helpful to have a portfolio or some other site up to point people to. Ideally, it would be on your own domain with decent-looking design. The quality of your portfolio/site sends signals to prospects about the quality of your work, so make sure it looks good.

Programmers aren't necessarily designers, a pain I know all too well. It's totally acceptable to hire a designer to build your website, or (if resources are limited) to use a site like ThemeForest to get a decent starter theme. If anyone asks you can always say that you worked with a designer on your site; if you put up something that looks bad, there is a good chance you'll never get to have that conversation at all.

Take Care of Some Legal Stuff

I am not a lawyer, and this should not be construed as legal advice. It is just my personal viewpoint and your mileage may vary.

If you have assets that you'd like to protect and a couple bucks to spend, you may want to consider registering a corporation or an LLC. If executed correctly, these entities can protect your assets (such as your house) in the event that something goes wrong and you get sued. There are companies like BizFilings that can take care of the paperwork for you for less than $400. Similarly, you can get liability insurance (including professional liability insurance, that covers clients suing you for bad advice) at fairly reasonable prices. This is particularly important if you work through certain staffing firms, who require their consultants who wish to work on a 1099 basis to carry a certain level of insurance.

Figure Out Your Pipeline

When you're starting out you need to figure out where your first gigs are going to come from. This requires a couple things.

Firstly, you should become much more talkative. When recruiters message you make sure to respond, and tell them exactly what you are looking for. I know the client is looking for full-time, but might they work with me on a contract basis? Ask around with your friends, particularly if they work in an industry where there tends to be a lot of project-based work, like advertising or IT consulting. Join some local groups active in your area of expertise, or business associations that might introduce you to clients. Meetup.com is a good place to start as they have lots of language-specific groups that meet regularly. Once there, ask around about other events and pass out a lot of your new business cards. Send follow up emails, arrange happy hours.

Referals and networking are the best way to find clients.

If you are currently employed, you can do some projects on nights and weekends to jump start the process while still having a steady income. Ideally, you could work with your employer to try to change your schedule so that you have at least one day a week working from home. This will make it easier to take client calls and/or meet with prospects during the week.

As a very last resort, you can look at freelancer sites like Elance or the Help Wanted section of Craigslist. These can be helpful when you are first building your portfolio and/or skillset, but they tend to put you in touch with clients who are extremely price sensitive and often don't understand what goes into a successful software product. These sites are also full of offshore developers who have a much different cost structure than you do. 

Wrapping Up

So far we've covered what freelancing is, and how to get started in the industry. In my next post I will talk about managing your freelancing business.

WebSockets in Java

WebSockets have been around for a little while now, but haven't quite reached the point of being mainstream; to some of us old-timers, something like Socket.io seems like the fevered dream of a madman, real-time communication between a web server and a client? What!?!

Fear not, gentle readers. It will all be clear soon.

A Little Background

Java has something of a stodgy reputation nowadays, a language used by large organizations to build applications for their accounting departments. That isn't entirely fair. Sure, a lot of Java tools can commit sins like SimpleBeanFactoryAwareAspectInstanceFactory, and can make your eyes glaze over while reading esoteric XML config files. But there are also the good parts, like the incredible library support, the highly-tuned virtual machine that runs on almost any operating system, the ability to build real self-contained apps. 

When working in Java one of the components I have ended up using often is Jetty, a wonderful embeddable web server that supports pretty much every feature you need, including SSL, WebSockets, custom request handlers, etc. This tutorial shows how to use Jetty to build a WebSockets-based application.

Intro to WebSockets

WebSockets provide a mechanism for full-duplex communication between a web server and client. A client (web browser) can open a socket with a web server. After this, the two can both write data to this socket and respond to messages coming from each other through event handlers registered on each side. Additionally, the server can broadcast messages to all of its connected clients. According to Wikipedia, WebSockets started trickling into browsers early in 2010 and are now available in all of the current major browsers, except for the naggingly resilient Android browser. Chrome on Android works well, however.

A Simple Chatroom

To demonstrate how WebSockets work with Java I've put together a simple chatroom application and tossed it up on GitHub. In addition to the source code, I've included a build of the application that can be run immediately. Once cloned, just go into the root directory of the checkout and type:

java -jar dist/JavaWebSockets.jar -p [some free port number]

After this you can point your browser (and perhaps even several browsers simultaneously to see the chatroom in action) to:

http://localhost:[some free port number]

Try sending a few messages to see how it works. Notice how the messages appear in each browser window instantaneously without the need to poll. This is one of the many powers of WebSockets.

How It Works: Server

JavaChatroom.java

JavaChatroom is the application class. It is a pretty standard Java application, and most of the work happens in the startServer() method. The code below initializes the ContextHandler for the WebSocket, and sets it to listen for requests coming in to the /chat route.

A little further down, a second handler (a ResourceHandler) is added to the HandlerList which handles requests for the web files stored under /www in the application home directory. 

ChatroomSocket.java

ChatroomSocket is the WebSocket itself. It is a subclass of WebSocketAdapter and overrides the various event handlers needed to make the WebSocket function. The interesting bits are below:

The onWebSocketConnect() method saves the Session for this socket to a local variable for later use, and also registers it as a participant with the Chatroom object so that messages from other users can be broadcast over this socket.

The onWebSocketText() method handles incoming text messages from the client, and adds them to the chatroom conversation.

Chatroom.java

The Chatroom class represents the chatroom itself, maintaining the history of the conversation and as well as the list of participants. In this simple implementation, there is a single instance of the Chatroom class, and a reference to it can be obtained from a static method on the JavaChatroom object.

The constructor starts up a ScheduledExecutorService that messages each participant a keep-alive message every 15 seconds. This ensures that the clients stay connected and continue to receive conversation messages.

The addMessage() method first adds the new message to the conversation history, then iterates over the participant sessions and sends the new message to each in turn. Finally, it removes any participants who are no longer connected.

How It Works: Client

chat.js

The web application first connects to the server through a new instance of the WebSocket class and registers the event handlers. The onmessage() handler listens for incoming messages from the server and appends them to the conversation, while filtering out the "keep-alive" messages.

Finally, event handles are registered for clicking the Send button or hitting the Enter key. In either case, the message that is typed in the chat text box is sent via the socket to the server and the text box is cleared.

Wrapping Up

There you have it, a simple application demonstrating how to use WebSockets in Java.

There are many other ways in which these same concepts can be extended to create amazing applications. Imagine a client application that is notified in real-time whenever an object on the server side is updated, or a cooperative browser-based game in which each players' position is relayed to their teammates instantaneously.

WebSockets are an enabling technology, giving us developers capabilities that just didn't exist before.

Use them wisely! 

Do It Yourself

Let me alone cause no one wants to be
Hanging around with someone messing up like me
I guess my way isn't good enough
When I try I just keep on wrecking stuff

Bouncing Souls, "Argyle"

Everyone carries different baggage with them into their chosen field. I don't mean "baggage" in the strictly negative sense; some baggage can be pretty helpful, like Batman's utility belt or Felix's Bag of Tricks. 

For me that baggage is punk rock.

I don't just mean the music. I mean the lifestyle, the feelings it evokes, the outsiderness and otherness. All of it. As an introvert, I wasn't the most social kid while growing up. Late in my high school career I discovered punk rock. Seeing people who were able to be cool without fitting in to mainstream culture, who weren't afraid to stand out. There was a lot of conformity in the semi-rural area where I went to high school, and these kids with the faster-louder music blaring from their car stereos seemed to offer a glimpse of something more exciting. I was hooked.

By the time I was at university, my whole social experience (what there was of it; I was still an introvert) was dictated by this music. I went to all of the parties, watched all my friends' bands, hung out at their radio shows. I eventually started playing bass myself, and listened mostly to relatively obscure early 80's English bands who often couldn't play and definitely couldn't sing.

Eventually I started to grow up. My taste in music changed, and I came to internalize my otherness rather than feel a need to be part of the scene or dress a particular way. I never forgot the feelings punk rock gave me, and keep many of the values to this day.

Chief amongst those values is the "Do It Yourself" aesthetic, often shortened to DIY. The idea behind this is that the barriers between amateurs and professionals in a given field are often artificial, and if you work hard enough, you can create something of value even if you lack the formal training. Sure, it might not be as pretty, but it's yours, damn it. And people better pay attention.


Recently I've been doing lots of work with different PHP frameworks and content management systems. Naturally there were some things I liked about each; for instance, the Yii Framework (a pretty standard MVC) has a cool routing system based on regular expressions, and WordPress (the venerable CMS) has themes, which are pretty useful for when you want to share a common backend across several client sites. There were also some things I disliked. Yii's database layer for example; it isn't bad as far as ORMs go (they all have problems) but it certainly gives developers enough rope to hang themselves, and I had to spend a lot of time stripping it out of certain parts of a client app when the queries it was generating were taking 10 times longer than they should have.

There was definitely room for improvement here, so I decided to give it a shot. The result is Sodapop, my first (complete) PHP MVC framework

It's still under active development, but it's pretty useful already. It follows the standard MVC pattern of having controller objects with action methods, model objects that represent rows in the database, layouts and view templates named based on their controller and action. It also has some unique features, such as themes and model classes that can be declared implicitly the first time they are invoked. 

There is still a lot I'd like to add to it, particularly around the database layer. Relations, migrations, support for other servers, stuff like that. But you know what? That's fine. It's something I made for my own projects and I can add to it as I need to while maintaining an internal consistency. If other people like what it offers, I'd love to have them along for the ride with me, offering suggestions and telling me where it needs improvement. Truthfully though, I'd make it even if nobody else looked at it. It's just fun to learn how these things work.

One could justifiably ask what the point is in creating something like a new web framework when there are already so many; aren't there some things where doing it yourself is just an exercise in ego gratification? Sure, there is that. But it's worth looking at the bigger picture.

Careers in our industry are frighteningly long, especially if you want to stay technical and not move into management. At any given time there are lots of sexy tools; today (at least in the PHP world) we have Laravel, Symfony, Yii, and myriad others. If you looked at it a few years ago, though, you would have seen a slightly different list, with Zend, CakePHP, and others at the top of the heap. 

The tools you use will always change, and you'll have to be learning all the time to survive.

While you do need to have a general competence and familiarity with popular tools, as well as the ability to get up to speed quickly, I would rather know concepts than dedicate my life to learning every intricacy of a specific tool. And there is no better way to learn concepts than doing it yourself.

When it comes to building websites for clients, I tend to use whichever framework they are comfortable with, including the popular ones. For my own projects, I want to have more control, and the ability to build in features that I'll still be able to leverage in 5 years after the frameworks du jour have splintered and been replaced by some new hotness. And I'll continue to refine my own tools to incorporate the good parts of the new frameworks that come out.


One of the magical things about being a programmer is that it affords you the ability to create the world that you live in. Programs have many components, and you can use components created by others, or use your own. You choose your own level of engagement. I can't think of any other field that offers that.

There is an art in having the self-awareness to know if what you are adding is creating value, or is just different for the sake of different. It can be fun to reinvent the wheel, but you won't necessarily improve on the original. 

Doing it yourself under the right parameters can help push your boundaries and grow as a developer. It can teach you how to design good systems and help you understand the tradeoffs that go into building such systems. It can also get you out of the cycle of reapplying the same patterns to solve new problems by giving you new tools to work with.

When I lose, every time I win, cause no one will ever be
Messing up stuff and doing things wrong quite like me
No one will ever be like me
No one will ever be like me

SASS and Transitional Technologies

I was first introduced to SASS early last year. The agency I was working with at the time (Evolution Bureau) began having a series of tech talks amongst the developers where we would introduce each other to different technologies we were excited about; my friend Josh gave a little presentation about his favorite CSS preprocessor.

For those who aren't familiar, SASS is a CSS extension language. The syntax is immediately familar to anyone used to writing CSS, but it includes many handy capabilities that naked CSS doesn't have, such as variables, nesting, functions and mixins. There is a compiler that can be set to monitor your SASS directories which will compile your SASS documents into CSS every time you save them, making them easy to work into your normal development flow. The result is much more maintainable stylesheets.

Needless to say, it looked impressive. But something about it just didn't feel right to me.

It seemed weird to have a critical part of an application (the stylesheets) entirely generated by a tool running on a developer's machine. I couldn't quite put my finger on why, exactly; it's just weird. Like, say there is some old website quietly humming along somewhere for a couple years. All the original developers have left, and you get tasked with updating some of the pages. Mostly just content updates, but a lot stuff needs to be rearranged and formatted. You download the site locally, pop open the CSS file to start your work and what do you see? Thousands of lines of machine-generated CSS rules. Maybe you can find the original SASS file on there somewhere (it isn't technically used by the production site, so there is a chance it wasn't pushed) but even then you have to set up your local machine to compile it and drop it in the same directory as the old files.

It's sort of like a code smell. Your code is no longer self-contained, and if SASS were to get hit by a bus tomorrow, your site would be very hard to update.

Fast forward a few months. I started work on another project with a different company, building a responsive site with Foundation. I had to start using SASS every day. And you know what?

It really is awesome. I'm not sure I would go back to using naked CSS if given the choice.

An article I read last week reminded me of my original thoughts about SASS, though. Matt Greer wrote Coffeescript's Time is Waning to explain the reasons why he won't likely use Coffeescript going forward; roughly speaking, Coffeescript is to Javascript what SASS is to CSS. 

The core of his argument is that as Javascript/ECMAScript is evolving, it is incorporating many of the features that made Coffeescript compelling; as browsers (and Node.js) start to support the newer standards, he is finding the friction of using the intermediate language to be harder to justify.

I could easily see something similar happening to CSS. Coding with SASS and other preprocessors is just so much more efficient that it seems inevitable that standards bodies and browser vendors will start bringing their best features into the core product, ultimately making them redundant. And this is as it should be.

Some technologies are transitional by nature, a bridge that must exist from the "bad old days" to something new and better. I think "transitionalness" is what I couldn't put my finger on originally; that as great as the tool is, the fact that it isn't a core technology made it feel like a mile marker rather than an final destination.

Of course, we don't know what the future looks like. In the meantime, I'll continue to pick the best tool for the job.

Welcome to the new RestlessDev Blog!

Hi everyone!

This is the new blog for my company, RestlessDev. In it you'll find random musings about business and technology, Silicon Valley culture and the life inside a small software company,

To start off I've imported a few older articles from my previous blog (which has since been taken offline) about a project I did called CodaServer. Obviously it hasn't quite taken off as I originally envisioned, but I still think it's interesting from a technology perspective. 

In rereading these old articles I'm impressed by two things:

  1. How little development has changed since then, and how a lot of the things CodaServer was meant to address are still problems.
  2. How little these problems seem to matter. I may have misjudged others' frustration levels just a little bit.

It was still a fun project to do as it taught me a lot. It was the first time I got to design a language and an interpretter for that language, and I had to put together a server platform for it as well. It even had a little client application not unlike the MySQL client!

Anyway, it's good to be back.

Flexible Business Processes with CodaServer

This is a repost of an article from my old blog about CodaServer. It's being published here for posterity.

Most software developers work in the trenches, the corporate IT departments, writing code to help automate a company's various processes.  While the Googles get the glory and Facebooks find the fame, our intrepid warriors make life better for one knowledge worker at a time.

One problem corporate developers have is that business processes, once in code, are difficult to change.  This is compounded when multiple languages or development platforms are in use, as the changes need to happen at the same time in multiple places.

The Old Way

Let's start with an example taken from a simple warehousing operation.

  • Pallets are delivered by UPS or Federal Express to the receiving department.
  • They are then inspected for damage by the staff.
  • If there is damage, they are rejected and sent back to the supplier.
  • If they are good, they are "broken down" and the boxes on them are stocked in the warehouse.

Using a traditional SQL database, there may be a table for "pallets" with a column called "status" having each of the highlighted words above as possible values.  (There would be other stuff in there too, but I'm keeping it simple.)  There would also be a (say) PHP class called "Pallet" with methods corresponding to each of these actions, as well as business logic handling the validation of various fields and making sure that the specified pallet was in the right state to have the action performed.  Then there is a web tool that let users record this stuff online, and it would also do some form validation as the user tried to do various things.  There is also a Java-based reporting engine that runs against the database, and it needs to know about all of the statuses for some of its ad hoc querying capabilities.

Now, say management decided to change this.  They add a staging status between delivery and inspection, since the warehouse is receiving so many shipments and the staff can't inspect them quickly enough.  The staging area will act as a short-term storage area, and the status will give management greater visibility into where their boxes are.   What needs to change?

Well, the database is simple.  "Staging" becomes a new status field value.  Just gotta make sure to add it to all of the environments.  The Pallet class gets a new method, and most of the other methods may need to change too to make sure they are checking for the new status.  The front end has some new conditional logic to take care of, and it needs a new button and status made available.  And the Java reporting system needs to know about staging.

The CodaServer Way

CodaServer handles "tables with status" using its FORM entity.  Instead of making status a data value in a table, it lets developers define the state diagram for the entity.  This is done by defining a series of "leads to" relationships:

  • Delivered leads to Inspected.
  • Inspected leads to Rejected or Broken.
  • Rejected leads to Nothing.
  • Broken leads to Nothing.

To add a new status to the list involves simply declaring the new status through an ALTER FORM command and rewiring Delivered to point to it via another ALTER FORM.

CodaServer lets you specify both an adjective and a verb for each status.

  • Delivered/Deliver
  • Inspected/Inspect
  • Rejected/Reject
  • Broken/Break

These verbs are what happens to the object to make it enter the adjective state.  Incidently, these verbs replace the traditional INSERT, UPDATE and DELETE operations in SQL databases, although you can still use UPDATE to modify an object without changing its state.

UPDATE pallets SET status = 'Inspected', 
inspection_date = CURRENT_TIMESTAMP WHERE id = 4;

is replaced by

INSPECT pallets SET delivery_date = CURRENT_TIMESTAMP WHERE id = '4';

CodaServer handles all of the busy work of determining if Pallet 4 is in the Delivered status, without your having to do anything.  It also fires off the BEFORE INSPECT and AFTER INSPECT triggers if you have them specified to handle any validation or cleanup that needs to be done.

What's more:

All of this is discoverable through metadata commands.

Applications can be written in such a way that they know about your business entities, their statuses, actions, and workflow without you needing to write any additional application code. Think about this, because it's a pretty big deal.  All application metadata is available through a cross-platform web services API, a business rules engine enforces your workflow and data integrity, and your workflow can change without you having to write more application code.

That's developing smarter.

Til next time.

User Permissions in CodaServer, Part 1

This is a repost of an article from my old blog about CodaServer. It's being published here for posterity.

Most business systems implement the concept of "users," those crazy scamps who login and actually use the software.  They're pretty important; each has different capabilities within the system and their access levels need to be made available to application code so that it can render a suitable interface.

The Old Way

Traditional database systems have very solid user management systems with one caveat: Nobody uses them.

This is one of the greatest antipatterns in enterprise development.  Due to the impedance mismatch between object-oriented application code and relational database tables, most application frameworks find it expedient to use a single database user with global permissions on the schema and implement a separate user management piece using database tables.  This is bad on several fronts.

  • Database permissions are pretty bulletproof.  If you don't have INSERT on a table, you can't stick anything into it.
  • Custom user management code is error-prone.  All of the thinking needs to be done by developers on a timeline.
  • Custom user management code reinvents the wheel, poorly.  I have seen a number of half-baked permission schemes in my day, and I'm not really that old.  There is no consistancy from one company to the next and the inner workings/caveats of these systems tend to live in the head of one or two developers.

That said, developers have no choice but to do it this way.  The CRUD model used in SQL simply isn't dynamic enough for a good permissions architecture.

Everything is an UPDATE.

Once you grant UPDATE on a table, you pretty much give a user access to do anything they want with it.

The CodaServer Way

CodaServer has a much different approach.  It is role-based, meaning that permissions are always assigned to roles rather than assigned to users directly.  Being the Business Rules Engine, it isn't limited to having CRUD as a model.  You can create a process map that goes something this:

  • Enter leads to Place or Cancel.
  • Place leads to Ship or Cancel.
  • Cancel doesn't lead to anything.

Developers can then assign roles permissions on Enter, Place, Ship, and Cancel.  It also has an ACL-type permission scheme where developers can create adhoc permissions and grant them to roles, SQL-style permissions for table objects, and permissions for stored procedures.

This plurality of permission schemes is meant to obviate the need for roll-it-yourself user management code and remove a major headache from developement.

Another way that CodaServer differs from traditional SQL databases is that the role-based permissions system is actually part of the schema for each application.  It is deployed with the application as it moves between different environments and the roles themselves are managed by developers rather than database administrators.  (The administrators have their own domain of responsibility covered in an upcoming article)

The logic behind this is that saying which role has what access to the system is very much a part of the business rules.  Is there any reason why a shipping clerk should not be able receive a shipment in the development instance but should in production?  Probably not.  You may want to say that Charlie should be stripped of his shipping clerk role because things he signs for tend to disappear, but the role itself is still a valid entity and shouldn't change.

A user in CodaServer can have any number of roles in an application environment, and their permissions in the system are a union of all the permissions of the roles they have been granted.  Metadata about a user's roles and permissions can be gleaned through the SHOW ROLES and SHOW PERMISSIONS commands.  No more trolling around in the information_schema.

Finally, a user management system that works with you rather than against you.

Til next time.

Schema Versioning in CodaServer

This is a repost of an article from my old blog about CodaServer. It's being published here for posterity.

Databases have always been a little awkward in the versioning department.

A database server is an environment all its own.  Different databases can have identical schemas but much different data in them.  Some of this data is very much part of the "structure" of the schema:  Tables that are used to populate important application drop down boxes, for instance, or metadata tables for various reporting functions.

This data is programmer data instead of user data, and the application can't work without it.

How does one keep both schema and programmer data in sync across different environments?

The Old Way

SQL databases are specialists.  They do their thing handling queries and update statements flying in from all directions.  They maintain data integrity by following the rules of the schema.  They handle enormous loads.  But that's about it.

Database servers are often known as "instances," and for good reason:  They are entirely self-contained.  They generally don't know about each other and don't involve themselves in the rest of the software ecosystem.  If you want to set up a testing environment, you "fire up a new instance" and load your schema onto it.  As you roll out new versions of application code, you run the diffs of the schema against the new environment and backfill the data.

The developer (or system administrator) does the thinking while the database server just runs its queries and twiddles its hard drive platters.

We should be so lucky.

The CodaServer Way

CodaServer has a much different architecture.  One CodaServer manages your development, testing, and production environments.  If you need more capacity, you can fire up multiple CodaServer instances, point them at the same system datasource, and put a load balancer in front of them.  Since all the communication with CodaServer is over HTTP-based Web Services, it has an architecture very similar to that of the web, and can leverage all the same techniques used to scale general purpose web sites.

(For more detail on CodaServer's architecture, you can look at the manual's architecture page)

Having one server (or load balanced group of servers) manage all of your environments has another benefit:  Simple version control.

As you build up your CodaServer application in the DEV environment, CodaServer keeps track of all Data Definition Language commands (like CREATE TABLE) in its transaction log.  When you want to move your application to TEST, simply run the PROMOTE command.  When it's ready for PROD, PROMOTE from test.  Your application is always constant in each environment because deployment always follows the same pattern.  There cannot physically be a schema in production that was not in development first.

But wait, there's more.

CodaServer also has the concept of REF TABLES, tables that remain constant at all times between environments.  All schema changes and INSERTs, UPDATEs, and DELETEs to REF TABLES occur immediately in DEV, TEST, and PROD.  This is ideal for most types of programmer data.

Some might think it's limiting that CodaServer doesn't allow developers to create tables directly on different application instances.  This isn't really as big a problem as it seems.

CodaServer's only concern is protecting the data model of your applications.  Since the applications use standard SQL databases, you are free to add any additional tables you want for reporting, ad hoc denormalization, data transformations, etc.  The only caveat is that CodaServer based applications won't know they are there, which is perfectly fine since they don't affect the datamodel.

(It is highly recommended that you adopt a naming convention for these tables that is different than that of your CodaServer tables to prevent collisions.)

So there you have it:  A database that does the heavy lifting for you.

Til next time.

Creating Crons in CodaServer, Comfortably

This is a repost of an article from my old blog about CodaServer. It's being published here for posterity.

Benjamin Franklin once said "Lost time is never found again."  That is doubly true if that time is when your company's batch processing was supposed to run, and now none of your suppliers are going to get their checks.  And they cancel your accounts, and you can't get more supplies.  Your customers rebel and go to your competitors, your dog leaves you to find a master who can remember not to screw up the mundane details.

That time is the most precious of all.

In programming, we constantly deal with events.  These can be user-generated (mouse clicks, form submissions) or machine-generated (a web service call) and we can easily build systems that capture the events and respond to them.  Less easily captured, however, are events generated by time.

How important are these events for a data model?  Pretty darned important.  There are many tasks which have a heavy data component that simply cannot be run on real-time production systems; rebuilding search indexes over millions of rows data, for instance, or moving millions of transactions from the denormalized transaction schema into the general ledger.  There are also operations that are done at a particular time of day as part of a business rule.  Electronic voting systems need to stop accepting votes at a particular time, and student registration systems need to open their course catalogs to certain students at certain times.

It's a big deal.

The Old Way

Traditional, time-based events are the domain of the venerable cron daemon.  This little guy reads cron tab files which specify times for which certain scripts are to be run.  Scripts can be set to run every month, on the 10th, at 3:34AM, or every Wednesday for each minute of the day.  It's a remarkably flexible system with a few drawbacks.

  • The scripts are not necessarily part of the application's main codebase.  If you have a PHP web app based on CakePHP, it's difficult to reuse that system's configuration to drive your cron script, which needs to run via the command line.  It also makes class libraries more difficult to share.
  • The cron daemon can die, or the server it is on can die, or the script can die or time out.  All of these could leave the database in a inconsistent state and can be difficult to troubleshoot after the fact.
  • Cron is generally handled as an outlier to the rest of the system.  It is difficult to version control, often has an owner who is not a developer (usually a sysadmin), and requires special consideration during code pushes.

The CodaServer Way

CodaServer's solution is to bring cron inside the database.

Instead of scripts, CodaServer's CRON runs stored procedures which have been previously defined by developers.  The syntax should be familiar.

CRON check_cron ON accounting.dev IS 0 0 1 * * process_checks;

This will create a new cron called "check_cron" in the development instance of the accounting application that runs the "process_checks" procedure on midnight on the first of each month.  By placing cron inside the business rules engine, CodaServer addresses each of the problems above.

  • The cron isn't separate from the rest of the code.  In fact it doesn't require anything in application code at all.  They become part of the application's business rules.
  • The cron is guaranteed to run as long as the business rules engine is up.  There are fewer moving parts.
  • The crons can be created and maintained by developers with the MANAGE_CRONS permission on the server.

Crons are unique amongst CodaServer objects in that they are not part of the application's transaction log.  They need to be created explicitly for each instance under which they should run.  This was meant to ease system administration and prevent the situation in which unintended events are pushed from environment to environment during code promotion with no way to turn them off.  The procedures they call are part of the transaction log, however, so these will be moved between environments automatically.

Cron has been a great friend for many years, and I'm really happy we can take him along and give him a new home.

Til next time.

Format Validation in CodaServer

This is a repost of an article from my old blog about CodaServer. It's being published here for posterity.

Validation means many things to many people.  It's that feeling you get when you are unconditionally accepted by the one you love, when all the toil and tears finally...

Uh...

For most of us programmers, validation comes in two forms:

  • Format-level Validation: Does this piece of data look the way it should?
  • Data-level Validation:  Does this properly-formatted piece of data represent an appropriate value for where I'm seeing it?

This article will focus on the former.

The Old Way

Using conventional technologies, format validation has been something of an annoyance for us.  Every layer of the application stack makes a token effort at addressing it, but none of them actually complete the job, and we are left negotiating between them.

  •     The database has database types, such as VARCHAR, NUMERIC, and BLOB.  As the final resting place for data, these place the most basic constraints on what it can look like and how much space it can occupy.  Of course, each database has slightly different constraints here.  A CLOB in Oracle is a TEXT in MySQL, VARCHARS can be 4000 characters in Oracle (I think, it's been a while) but only 255 in MySQL.
  • Application languages have their own types, like ints and Strings, which kind of map to SQL types.  Except when they don't, when they are arrays, or classes.  Adaption code must be written to handle this conversion and ensure the application values fit into the database.
  • Application frameworks often include their own validation layers for processing form input and making sure it conforms to some higher level structure.  Is that string an email address?  Does that purchase order number look like one of ours?  How about the ZIP code?  What about a Canadian postal code?  Usually the frameworks include limited number of these validators and leave it to the developers to round out the set.
  • Sometimes Javascript is employed to stop bad data from ever leaving the web browser.  Just in case.

And this is only one application language on one database.  Perl libraries for validation can't be used in Ruby.  Let's rewrite 'em!  All this format validation hassle adds virtually no value to the final application, but it needs to be done anyway.

The CodaServer Way

Naturally, CodaServer has an answer for this problem:

An Extensible Type System!

Natively, CodaServer has 8 datatypes as well as an Array specifier. These correspond closely to the SQL types and are pretty generic containers for data. The real power comes from the ability to define your own types using the CREATE TYPE command.

CodaServer utilizes regular expressions to give you virtually unlimited ability to describe what your data should look like.  (If you are unfamiliar with regular expressions, there are many great resources to get you started online.  I use this site a lot to get ideas, and this site to test them out.  CodaServer uses the Perl type of regular expression.)

The CREATE TYPE command has the following syntax:

CREATE TYPE email_address AS (
  validation_mask = '/^(.+)@([^();:,<>]+.[a-zA-Z]{2,4})/',
  save_mask = '/^(.+)@([^();:,<>]+.[a-zA-Z]{2,4})/'
);

The validation_mask value determines whether or not CodaServer throws an error, and the save_mask determines how the validated result is saved to the database.  This uses a facility called regular expression substitution which is a beyond the scope of this article, although there are resources online that can give more details.

That's all there is to it!  Now you have a new type that is available server-wide.  You can use it in procedure argument lists, table definitions, procedural code blocks... where ever you would like.

On the application code side, you can simply pass in whatever the user submits in a form and CodaServer will tell you whether or not it validates.  And (there's more!) CodaServer's ability to return multiple error messages with each statement means that you can submit your whole form, and you will get back all of the errors at once.  No more of the "submit and see what else is wrong" that other database systems make you do.

Doesn't that feel better?

Til next time.