The Gradual Steep

Dynamic Pivot Tables with JSON and PostgreSQL

Sun, 18 Sep 2016 00:00:00 +0000

Pivoting data is a useful technique in reporting, allowing you to present data in columns that is stored as rows. Assuming you’re using a relational database, you can construct such queries using the SQL Server PIVOT operator or Postgres crosstab function. However, these queries are limited in that all pivot columns must be explicitly defined in the query.

In the relational model, rows give you flexibility while columns are static. Ideally, we could use the data-driven flexibility of rows to define our pivot table dynamically. The static typing of SQL generally makes this challenging, but the JSON datatype and related functions available natively in PostgreSQL provide a compelling solution. We can use the strong declarative syntax of SQL to dynamically create JSON objects with properties derived from the rows in our database.

Example Schema

Consider the following schema for storing book sales:

CREATE TABLE book (
  book_id INTEGER PRIMARY KEY,
  title TEXT NOT NULL,
  author TEXT NOT NULL,
  publication_year INTEGER NOT NULL,
  genre TEXT NOT NULL,
  price MONEY NOT NULL
);

CREATE TABLE customer (
  customer_id INTEGER PRIMARY KEY,
  firstname TEXT NOT NULL,
  lastname TEXT NOT NULL,
  state CHAR(2) NOT NULL
);

CREATE TABLE sale (
  sale_id INTEGER PRIMARY KEY,
  book_id INTEGER NOT NULL,
  customer_id INTEGER NOT NULL,
  sale_date DATE NOT NULL
);

Populate it with data for our queries, and we’re ready to play around with pivots.

Crosstab Solution

First, let’s look at the traditional crosstab method from PostgreSQL’s tablefunc module.

CREATE EXTENSION IF NOT EXISTS tablefunc;

SELECT * FROM crosstab(
  $$
    SELECT
      date_part('year', sale_date) AS year,
      date_part('month', sale_date) AS month,
      COUNT(*)
    FROM sale
    GROUP BY sale_date
    ORDER BY 1
  $$,
  $$ SELECT m FROM generate_series(1,12) m $$
) AS (
  year int,
  "Jan" int,
  "Feb" int,
  "Mar" int,
  "Apr" int,
  "May" int,
  "Jun" int,
  "Jul" int,
  "Aug" int,
  "Sep" int,
  "Oct" int,
  "Nov" int,
  "Dec" int
);

This query will return data that looks something like this:

year | Jan | Feb | Mar | Apr | May | Jun | Jul | Aug | Sep | Oct | Nov | Dec
-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----
2010 |   6 |   2 |   5 |   3 |   4 |   7 |   6 |   5 |   2 |   6 |   2 |   1
2011 |   2 |   7 |   1 |   7 |   5 |   4 |   1 |   5 |   1 |  10 |   1 |   1
2012 |   4 |   4 |   7 |   4 |   5 |   3 |   3 |   4 |   2 |   6 |   3 |   3
2013 |   7 |   4 |   7 |   3 |   2 |   6 |   5 |   4 |   2 |   2 |   2 |   6
2014 |   4 |   3 |   6 |   3 |   1 |   4 |   3 |   4 |   4 |   7 |   5 |   3
2015 |   2 |   3 |   3 |   4 |   2 |   3 |   3 |   5 |   3 |   4 |   1 |   4
2016 |   5 |   5 |   6 |   8 |   2 |   5 |   4 |   4 |   6 |   1 |   3 |   6

The details of exactly what’s going on here are not intuitive. Essentially the crosstab function is matching the 2nd column of the first query to the 1st column of the second and aggregating values into the appropriate ‘year’ and ‘month’ bins behind the scenes. It has the advantage of being fairly compact, but the logic is extremely opaque and it feels a bit like magic. What’s possible in a crosstab query is limited to input queries formatted in a very specific way, and all output columns have to be expressly defined in advance. As the Postgres docs say: The crosstab function is declared to return setof record, so the actual names and types of the output columns must be defined in the FROM clause of the calling SELECT statement.

JSON Solution

Let’s see if we can improve this. PostgreSQL added a native JSON datatype back in version 9.2 and as of 9.5 has extended the support with enhanced functions, operators, and better performance. Using this functionality provides a great solution that is a bit more readable and also completely dynamic:

WITH month_total AS (
  SELECT
    date_part('year', sale_date) AS year,
    to_char(sale_date, 'Mon') AS month,
    COUNT(*) AS total
  FROM sale
  GROUP BY date_part('year', sale_date), to_char(sale_date, 'Mon')
)

SELECT
  year,
  jsonb_object_agg(month,total) AS months
FROM month_total
GROUP BY year
ORDER BY year;

Let’s unpack this a bit. First, our month_total CTE looks very similar to the first query in the crosstab example. Here we are aggregating the total sales for each month. This CTE returns a set with one row per month (e.g. 2014 January, 2014 February, etc) and the total sales for that month. We then use the jsonb_object_agg function to dynamically build JSON objects for each year where the keys are months and the values are sales totals. The output should look like this:

year |                                  months
-----+-------------------------------------------------------------------------------
2014 | {"Jan": 132, "Feb": 109, "Mar": 118, "Apr": 126, "May": 131, "Jun": 111, ... }
2015 | {"Jan": 103, "Feb": 115, "Mar": 113, "Apr": 137, "May": 113, "Jun": 128, ... }
2012 | {"Jan": 110, "Feb": 109, "Mar": 115, "Apr": 109, "May": 126, "Jun": 96, ... }
2013 | {"Jan": 122, "Feb": 121, "Mar": 120, "Apr": 120, "May": 106, "Jun": 128, ... }
2011 | {"Jan": 128, "Feb": 121, "Mar": 109, "Apr": 116, "May": 120, "Jun": 104, ... }
2010 | {"Jan": 124, "Feb": 111, "Mar": 120, "Apr": 111, "May": 115, "Jun": 112, ... }
2016 | {"Jan": 118, "Feb": 122, "Mar": 116, "Apr": 111, "May": 112, "Jun": 111, ... }

Using a library like node-postgres to interface with PostgreSQL will return the data to our javascript code almost identically as the earlier table. In both cases, the results are returned as an array where each row is its own object with the column names as properties. Using PostgreSQL’s native JSON functionality, however, allows us to write a more elegant query where the columns (or object properties) are determined dynamically at execution time.

Extending this Technique

Let’s explore how simple this makes it to adjust which slice of our little OLAP cube we’re viewing:

WITH state_total AS (
  SELECT
    c.state,
    b.genre,
    COUNT(*) AS total
  FROM sale s
  INNER JOIN book b USING(book_id)
  INNER JOIN customer c USING(customer_id)
  GROUP BY c.state, b.genre
)

SELECT
  state,
  jsonb_object_agg(genre, total) AS months
FROM state_total
GROUP BY state
ORDER BY state;

This query will return sales data grouped by state on the vertical axis and genre on the horizontal. From here you can imagine slicing the data any way you want with relatively minimal changes.

This same technique can also be used when properties of a particular entity are not explicitly defined as columns, but the data is stored as rows (imagine a key value store for item properties or a tagging system, etc). In these cases, the dynamic properties can define the fields returned in the output just as the reporting dimensions do in these examples.

Tabletop Telecast: Live Streaming Sentinel Tactics

Tue, 30 Dec 2014 00:00:00 +0000

This past fall I was brought in to produce live video coverage of a competitive board game tournament by Greater Than Games in St. Louis, MO. The goal was to create a quality viewing experience comparable to high level eSports tournaments (think StarCraft II, League of Legends, and Hearthstone) except with a tabletop game on a fraction of the budget. It’s been a particularly interesting project because of the assortment of disciplines involved. Everything from computer programming and audio mixing to photography and stage lighting were involved in one way or another. We definitely didn’t quite approach the million dollar production values of The International or Blizzard’s WCS, but I believe we were able to create an enjoyable and compelling viewing experience that will continue to improve over time.

Sentinel Tactics is a hex grid tactical combat miniatures game. Players control characters from the “Sentinel Comics” superhero universe and duke it out through various scenarios and skirmishes. Dice rolls are used to determine attack resolution and other aspects of the game. In the tournament setting, three person teams face off in skirmish play, drafting characters before the match similar to competitive MOBAs. The first team to score three incapacitations, or knockouts, wins the match.

eSports broadcasts benefit largely from the extensive design work of the game developers themselves to present relevant information to the viewer. Video games are intended to be viewed on a screen by the player, so translating that experience into a digital broadcast is fairly straightforward. Board games, on the other hand, are optimized for play in the physical world and exposing all the relevant game state information on a single, digital screen takes careful planning and additional design work. While we always intended the matches to be accompanied by high level audio commentary, we wanted viewers to easily understand what was going on during a game without leaning heavily on information from the casters. This led us to a strategy involving a set of stationary cameras along with roaming cameras and visual overlays. These goals demanded at least five angles to consistently capture everything we needed from the game board as well as the announcers (1 overhead camera for the full map, 1 overhead camera for a dice closeup, 2 roaming cameras, and 1 camera for the casters). The physical game utilizes cards, panels, and tokens to track information for each character, all of which are not as easily parsable in a video format. Hence, a set of custom graphical overlays were developed to present the score, current turn, character health, and other key information on screen.

Hardware Setup

Fortunately for this project, creating high quality digital video has continued to become more affordable in the past decade. What once cost tens of thousands of dollars can now be produced at an order of magnitude lower cost with largely consumer equipment. For Sentinel Tactics, we used a combination of consumer, prosumer, and professional hardware to get the job done. An ATEM 1 M/E Production Studio 4K provided the video backbone and mixing capabilities, and a Decklink 4k Extreme capture card allowed us to ingest the video into our computer on the fly and simultaneously send overlay information back to the mixer. All aspects of the video mix were controlled directly from the computer through custom macros and an X-Keys USB control board. We used a combination of Canon XHA1 and Panasonic HC-V750 cameras with component video/HDMI to SDI converters to get everything into HD-SDI format for the video mixer. Principle lighting and all stationary cameras were mounted to a 10-foot truss above the players and game board.

On the audio end, a Yamaha MGP24X mixer allowed us to manage three completely independent mixes for running the production. Of course, there was the live mix for the stream and recording, but behind the scenes the casters had a modified mix that allowed the production booth to communicate with them directly through their headphones. Additionally, the roaming camera operators had a separate mix and were continually being fed information and instructions from the booth. A pair of Shure VP82 shotgun mics hanging from the lighting truss provided game audio from the table to compliment the announcer’s commentary.

Software and Overlays

The most complex elements of the broadcast were the custom overlays and templates that we needed to update in real time as the game progressed. In searching for methods to enable this interactivity, we came across an open source software package called CasparCG which has been developed by a broadcasting company in Sweden. CasparCG is designed for outputting visual templates for use in broadcast applications, and was able to function seamlessly with our Decklink capture card. In essence, Caspar creates an independent video feed of the overlay with alpha transparency, which is then sent from the computer to the video mixer and can be used just like any other video source.

The CasparCG rendering engine is built to utilize Adobe Flash for custom templates. However, the most common use case employs basic templates designed in Flash Professional with simple text that can be changed manually in the CasparCG interface during a broadcast. Given the complexity of the information and visuals we needed to display and update for Sentinel Tactics, this basic functionality was insufficient. We would be simply unable to manually update team information, scores, health, and other data through text inputs while simultaneously mixing the video and audio and managing the stream. What’s more, such restrictions place severe limitations on the design of the overlays and the way information must be displayed. Fortunately, CasparCG contains a fully functional Flash engine, enabling us to build custom templates using ActionScript. Anything Flash could display, we were able to create, animate, and modify programmatically in real time. This opened up tremendous potential for the visuals and allowed us to capture much of our vision.

However, this solution still left the issue of getting data to the ActionScript templates and updating it on demand. For this, we built a Node.js web service and application on top of the hapi framework to track live game state and statistics. On initial load, the ActionScript templates issue an HTTP request for the current game state and render themselves accordingly. Subsequent updates are pushed to any running templates through web sockets via the Pusher messaging service. On the interface end, we built an iPad-optimized web application in Clojurescript/Om to handle all state input. This enabled the person tracking game state to be right at the table with the players so they would have the most immediate access to any information they needed. It also freed the production staff from micromanaging game state, so the booth could focus on shot selection, mixing, and the overall production. Any time the broadcast required a particular visual template, we could bring it up and the information would be accurate. While there were a couple minor hiccups in our initial outing with this technology, overall it worked very well. You can see it in action here on the YouTube clips from our first broadcast.

Real-Time Video Compositing with Javascript in a Web Browser

While our CasparCG/ActionScript templating system worked well for the majority of cases, one of the design ideas in our original concept couldn’t be realized with that technology stack. We wanted to highlight the painted miniatures when the players were choosing their characters during the draft. We were inspired by the loading and matchmaking screens of games like Heroes of the Storm, where all characters entering a match would be lined up in formation. Our idea was to have rotating versions of the miniatures appear as each player chose their character for the upcoming match. Since there are currently 15 playable characters, doing so in a way that would cover all possible team configurations required compositing keyed videos of each miniature into their arrangement on the fly to make up the final shot. For the videos themselves, we used a turntable, lightbox, and DSLR to record an individual clip for each mini on a white background. However, the composite was a particular challenge because virtually no video delivery formats support an alpha channel, and none do that are currently supported by Flash.

HTML5 video came to the rescue with the VP8 codec. VP8 supports an alpha channel, and Chrome is able to display it. We created a static web page that used Javascript to pull in the current draft and handle any web socket updates as characters were chosen. Chrome was able to run all six rotating videos together in a composite that was managed in real time via Javascript. In fact, all 15 videos were loaded initially and looped concurrently. Any not in use were simply positioned off screen to be moved into place on the fly as characters were chosen or changed. This web page was run as a secondary monitor on our streaming computer, the output of which was sent directly to the ATEM video mixer as a source. The only major issue we encountered was an auto-update to Chrome on the day of the broadcast which severely degraded the smoothness of video playback. This caused our cleverly-engineered drafting screen to be extremely choppy and unusable early on. We ended up manually forcing Chrome to revert to an earlier version between matches, which provided a temporary fix to the issue. You can see a sample of this page in action here.

Conclusions

Overall our inaugural broadcast went very well and was received with great enthusiasm by the fans. As with any live production there were a few hiccups, but for the first time producing anything of this complexity everyone was quite pleased. You can watch any of the matches over on YouTube if you like. Some of them were pretty exciting!

YouTube: Sentinel Tactics Inaugural Tournament 2014

Special thanks to the other members of our production team at HyperCatalyst, Trevor Casterline and Jordan Arentsen, for their invaluable work making everything come together from template design to coding.

There are a few additions we would like to make in the near term including a second dice camera (we ended up having two dice boxes due to the size of the table), and some audio fixes and improvements to the mixing. We also have ideas for enhancements and additions to the overlays. Our initial concept for all the templates included much more than we were able to build for the first outing. Given the opportunity to continue producing tournaments for Sentinel Tactics, these enhancements should come in time. Some interesting updates on that front are already in the pipeline. We’ll be live from San Antonio with the next tournament at PAX South on January 24-25, 2015 as well as PAX East on March 7-8 in Boston! Stay tuned for more information, and keep an eye on the Greater Than Games Twitch channel!

Paying Off Our Technical Debt

Mon, 01 Sep 2014 00:00:00 +0000

All new businesses incur some amount of debt whether in the form of loans, investment notes, owners equity, or any number of other things. Many software companies also incur technical debt. Technical Debt refers to employing software practices that provide expedited solutions at the expense of long-term development efficiency and maintainability. As with any kind of debt, the short-term benefits provided can sometimes be essential to delivering an actual product in time, given the nature of software projects to run behind schedule. However, if the quality of code is sacrificed for speed in the short term, it must be paid back with interest later. Paying interest on technical debt amounts to extra work that would not have to be done if no corners were cut originally.

For us this interest rate was high.

More serious than any issues with quick solutions or corner-cutting, however, were dramatically shifted requirements from our original product design. Simple Campus Housing provides a web-based software package to colleges and universities for managing their on-campus housing and residence halls. When first prototyping our product for prospective universities, there was a strong aversion to using any software that was not hosted on campus servers behind their firewalls. In an effort to mitigate these concerns and support the wide array of possible IT infrastructures, we chose as ubiquitous and cross-platform a technology we could at the time: PHP.

Original Architecture

Our original architecture was a single PHP application per client using the Lithium MVC framework and Doctrine ORM. This allowed us not only to host the code on different operating systems and web servers, but also use whatever relational database technology each client had available. At one point we had production systems running on Linux, Apache, and PostgreSQL; Linux, Apache, and MySQL; Windows, IIS, and SQL Server; plus dev environments on OSX, Apache, and SQLite all using the same code base. This stack worked well for a while and was great to get us in the door, but we ran into maintainability and scalability problems quickly. Even from the start, we had to break out our roommate matching system into a separate web service, since we needed more control over the server resources and availability. As the complexity of the app rose, we quickly realized we needed complete control over all aspects of the environment to ensure the best user experience (and to support features we wanted to build).

Fortunately, the overall climate of IT in higher education has warmed significantly to the benefits of using cloud services since our first prototypes in 2009. We were able to move all code into our own cloud a couple years ago, which was a great first step. However, running an entirely separate server instance for each client was far from ideal.

We decided in June of last year that the time had come to begin a serious effort paying off our technical debt to really more forward. Our original architecture decisions were causing more and more issues with scaling, bugs, and increased development time. I’ve heard it said that “technical debt is often paid for by operations.” But when you are a team of three, development is operations. Something definitely had to be done.

New Architecture

Even though reading and particularly refactoring code is often much slower than writing new code, we wanted to avoid falling into the old Netscape trap of rewriting from scratch. Since our ideal new architecture was going to look significantly different from the monolithic PHP app we had built, however, much of it would have to be rewritten over time. This ideal architecture would be much more microservices-oriented, and allow us to chose the best programming tool for each task without affecting other parts of the application. Additionally, it would allow us to consolidate clients instead of requiring completely separate servers for each one. We also felt very strongly that we needed to support isomorphic code on the frontend. The real-time room selection feature we built relied a litany of complex business logic that had to be coded and debugged in the frontend javascript as well as the backend PHP. Given its complexity, maintaining and updating that system alone was proving to be a significant investment. Finally, we had to move away from an ORM. As convenient as it was in supporting disparate databases early on, the inherent inefficiencies were causing significant performance problems. Additionally, ORMs by their nature tightly couple code throughout an application, and we needed to break this coupling to improve maintainability.

We knew the business would not be served with new development out-of-commission for a significant period, so we decided to take a slow and incremental approach. We focused first on two key areas that would allow us to serve primary business objectives while moving us toward an updated technology stack. The first was decoupling our code from the ORM where it was least performant and most intertwined. In particular, this meant rewriting our “property system” which provides a way to generically address any piece of data in the system. This system is used by many other components from roommate matching to reports, and rewriting it would alleviate the worst performance problems and allow us to communicate with the database without relying on our PHP ORM. We built a new system using direct PostgreSQL queries and a node.js backend that improved performance by about 2 orders of magnitude. This node backend would also provide the basis for our eventual service API that would route requests for all components in the future. The second area we focused on was authentication. We needed to support single-sign-on (SSO) for a number of clients, and rewriting our authentication system also removed the primary barrier we had to consolidating our servers. We replaced native PHP with CAS Authentication. This enabled us to use the same method whether users needed to be authenticated through our internal system or the single-sign-on server at their university. Combined, these changes allowed the business to reduce its growing server costs while delivering required feature updates and improved performance for clients.

The above diagram shows the current state of our system. A request is directed to the correct service by haproxy, which also adds a custom http header to identify the originating client. Much of our code still exists in the old php app, with authentication and a few other components stripped out. Over time we will rebuild portions of the php in our new system as we need to upgrade them and add functionality. Jasig CAS now provides the basis for our internal authentication, and communicates with a dedicated node.js service to determine client credentials. Some ajax requests are transmitted directly to the new node API over http. Other backend requests to the API can occur over RabbitMQ for increased speed. Client configuration settings and account credentials are now stored in a Redis database, while each client has an independent PostgreSQL database for main application data.

The primary transition to this architecture happened in February, and overall went quite well considering the scope of the changes. Since then we’ve added the RabbitMQ updates, fleshed out our SSO support, improved our frontend build process, and transitioned some smaller components. Most importantly, we have been able to continue making updates to the old application throughout this entire transition in service to our clients.

The Future

We are now working on new features and one of the most exciting goals enabled by our architecture transition: Isomorphic Clojure[Script]. While this project is still in its early stages, we are very enthusiastic about what we’ve seen so far. Our current plan is to use Om with Facebook’s React to build pages, rendering them on the backend using Clojure and Nashorn, and on the frontend with ClojureScript in the browser. I hope to post more about the details as we progress.

Embedding a Twitter feed using API v1.1

Sat, 05 Oct 2013 00:00:00 +0000

Back in June of this year, Twitter discontinued their public REST API version 1.0, which made it impossible to anonymously pull twitter feeds directly into other websites. As of API version 1.1, all access to tweets must take place through authenticated applications over the OAuth protocol. This had the effect of breaking numerous sites that relied on a twitter feed for news updates and embedded contact information. Our company website was one such site affected by this change, and it rendered a popular jQuery plugin entirely useless. I still find quite a few sites around the web with embedded twitter feeds that are perpetually “loading” due to the old API closing down. The only simple solution provided by Twitter is to use their pre-styled embedded timeline widget, which is distinctively twitter branded. Fortunately, there is another way around this problem.

Twitter’s API version 1.1 allows registered applications to query for tweets, and anyone can register an application. All you have to do is create an “application” for your website on Twitter’s developer site (dev.twitter.com), and set up a little boilerplate server-side code to handle the OAuth authentication and pull in the feed. You can then pass this information along to the frontend javascript or render it on the page directly.

I wrote a simple php script for this very purpose, which you can grab over on github:

http://github.com/adamanthil/php-twitter-proxy

This program takes a twitter handle and tweet count as query parameters, and returns the resulting posts in JSON. With some basic javascript you should be able to replace the jQuery “tweet” plugin, and have a custom styled twitter feed on your site. It makes use of the codebird-php library to handle the OAuth details. Status updates from of a user’s timeline are returned as plain text & html (with working links) along with the relative timestamp.

For example, the following query:

get-tweets.php?screen_name=adamanthil&count=2

Will return this:

[{
  "tweet_text": "Tracking email opens with an image is just like the Halting Problem.
  	You might know the msg was opened, or you might never know either way",
  "tweet_html": "Tracking email opens with an image is just like the Halting Problem.
  	You might know the msg was opened, or you might never know either way",
  "time": "2 days ago"
},
{
  "tweet_text": "Incredible talk by @richhickey on simplifying software development.
  	Cant recommend it enough http://t.co/B9TMdNjau1 /via @rtfeldman",
  "tweet_html": "Incredible talk by @richhickey on simplifying software development.
  	Cant recommend it enough <a href='http://t.co/B9TMdNjau1' target='_blank'>
  	infoq.com/presentations/…</a> /via @rtfeldman",
  "time": "3 days ago"
}]

I’m using this code in my footer here, and we’ve got it running over at simplecampushousing.com as well. Feel free to fork the project and make any changes if you like. It’s pretty basic right now, but it works dandy.

Returning Hierarchical Data in a Single SQL Query

Sun, 22 Sep 2013 00:00:00 +0000

I am a big proponent of using the right tool for the job. And as far as databases are concerned, I believe that in most cases a relational database is still the correct tool. However, one historical drawback of relational databases has been a difficulty in retrieving hierarchical data in a way that is easily compatible with most programming languages. Fortunately, support for modern data types such as JSON and advanced features including recursive common table expressions (CTEs) provide some excellent ways to tackle this problem.

My relational database of choice is PostgreSQL, which is open source and feature rich. It is fantastic, and you should definitely be using it. Postgres version 9.3 was recently released with even better support for the JSON data type among its new features. Here are some methods I routinely use to retrieve structured, hierarchical data using raw SQL in a single query.

Nested Objects

When working with data in your backend code, you often need quick access to the relationships within your data model. Including nested JSON objects in your result set is a great way to accomplish this naturally. This approach works particularly well if you’re interfacing with the database using node.js or another platform with fast JSON decoding.

Imagine the following hypothetical database of projects managed by employees:

CREATE TABLE employee (
  employee_id INT PRIMARY KEY,
  name TEXT NOT NULL
);

CREATE TABLE project (
  project_id INT PRIMARY KEY,
  employee_id INT NOT NULL REFERENCES employee(employee_id),
  name text NOT NULL
);

With some data:

 employee
--------------------------------
 employee_id |       name
-------------+------------------
           1 | Jon Snow
           2 | Thoren Smallwood
           3 | Samwell Tarley

 project
-----------------------------------------------------------
 project_id | employee_id |              name
------------+-------------+--------------------------------
          1 |           1 | Infiltrate Mance Rayder's Camp
          2 |           3 | Research the Wights

Suppose we wanted to return a list of projects and the employees responsible for them as an array of nested objects. PostgreSQL provides some useful JSON functions for manipulating data, and we will use the row_to_json function to nest objects directly in the query results. row_to_json provides the ability to turn a database row into a json object, which is the key. Consider the following query:

SELECT
  p.*,
  row_to_json(e.*) as employee
FROM project p
INNER JOIN employee e USING(employee_id)

If we’re using the pg node.js library to interface with Postgres, this query will return the following directly to our Javascript:

[
  {
    "project_id": 1,
    "employee_id": 1,
    "name": "Infiltrate Mance Rayder's Camp",
    "employee": {
      "employee_id": 1,
      "name": "Jon Snow"
    }
  },
  {
    "project_id": 2,
    "employee_id": 3,
    "name": "Research the Wights",
    "employee": {
      "employee_id": 3,
      "name": "Samwell Tarley"
    }
  }
]

Exactly what we’re looking for. It is very natural to work with this kind of data structure in Javascript or any other imperative language.

Augmenting Nested Objects

Sometimes it is necessary to return additional fields along with a given object that may not be directly included in the database table (due to normalization or other factors). For example, perhaps we are storing the date a project was assigned but would also like to return the age of that project in days. Let’s add to our previous data model:

ALTER TABLE project ADD COLUMN dateassigned DATE;

UPDATE project SET dateassigned = '2013/09/10' WHERE project_id = 1;
UPDATE project SET dateassigned = '2013/09/16' WHERE project_id = 2;

INSERT INTO project (project_id, employee_id, name, dateassigned)
VALUES (3, 3, 'Send a raven to Kings Landing', '2013/09/21');
INSERT INTO project (project_id, employee_id, name, dateassigned)
VALUES (4, 2, 'Scout wildling movement', '2013/09/01');

This time we are going to reverse the desired results and query for a list of employees and their respective projects. But let’s add the age of each project in addition to the date assigned. How can we accomplish this?

One approach would be to use a subquery to create a new virtual table, and then use row_to_json to transform that virtual table row into JSON. We will do essentially the same thing, but use a common table expression to redefine the project table with an additional “age” column. Also note that since employee->project is a one-to-many relationship, we are going to use the json_agg aggregate function to return a JSON array of objects instead of a single object as we did previously with row_to_json.

WITH project AS (
  SELECT
    p.*,
    date_part('day', age(now(), dateassigned::timestamp)) as age
  FROM project p
)

SELECT
  e.employee_id,
  e.name,
  json_agg(p.*) as projects
FROM employee e
INNER JOIN project p USING (employee_id)
WHERE employee_id = 3
GROUP BY e.employee_id, e.name

This query returns the following JSON to our Javascript backend. Note it is filtered to a single employee to reduce the size of the result set:

[
  {
    "employee_id": 3,
    "name": "Samwell Tarley",
    "projects": [{
      "project_id": 2,
      "employee_id": 3,
      "name": "Research the Wights",
      "dateassigned": "2013-09-16",
      "age": 6
    },
    {
      "project_id": 3,
      "employee_id": 2,
      "name": "Send a raven to Kings Landing",
      "dateassigned": "2013-09-21",
      "age": 1
    }]
  }
]

Recursive Common Table Expressions

Your dataset may include more complicated relationships in the form of tree-structured, hierarchical data as well. For example, tracking the management structure of our sample organization, where each “employee” reports to an immediate superior. This forms a tree of employees with the boss as the root node followed by subordinates. Probably the most common way to model such a tree is using the Adjacency List Model. In this representation, each employee record contains a reference to their immediate superior. If they are the head of the organization, this reference is null.

Let’s modify our existing schema once more to reflect this:

ALTER TABLE employee ADD COLUMN superior_id INT REFERENCES employee(employee_id);

INSERT INTO employee (employee_id, name, superior_id)
VALUES (4, 'Jeor Mormont', null);
UPDATE employee SET superior_id = 4 WHERE employee_id <> 4;

INSERT INTO employee (employee_id, name, superior_id)
VALUES (5, 'Ghost', 1);
INSERT INTO employee (employee_id, name, superior_id)
VALUES (6, 'Iron Emmett', 1);
INSERT INTO employee (employee_id, name, superior_id)
VALUES (7, 'Hareth', 6);

We can now use a recursive CTE (common table expression) to return this tree of data in a single query along with the depth of each node. Recursive CTEs allow you to reference the virtual table within its own definition. They take the form of two queries joined by a union, where one query acts as the terminating condition of the recursion and the other joins to it. Technically they are implemented iteratively in the underlying engine, but it can be useful to think recursively when composing the queries.

WITH RECURSIVE employeetree AS (
  SELECT e.*, 0 as depth
  FROM employee e
  WHERE e.employee_id = 1

  UNION ALL

  SELECT e.*, t.depth + 1 as depth
  FROM employee e
  INNER JOIN employeetree t
    ON t.employee_id = e.superior_id
)

SELECT * FROM employeetree

The above query will return Jon Snow, employee_id = 1, and all employees subordinate to him either directly or through the tree. In our small dataset, the result looks like this:

 employee_id |    name     | superior_id | depth
-------------+-------------+-------------+-------
           1 | Jon Snow    |           4 |     0
           5 | Ghost       |           1 |     1
           6 | Iron Emmett |           1 |     1
           7 | Hareth      |           6 |     2

This is fairly straightforward in our example, but it can be a powerful tool on larger trees especially when extended and combined with other techniques.

Combining Everything

We can use recursive CTEs in conjunction with the JSON functions to produce some really useful results. The following query will return the record for “Hareth”, employee_id = 7, and a nested list of his superiors up to and including the root node. Each superior will be a complete employee object including their projects.

WITH RECURSIVE employeetree AS (
  WITH employeeprojects AS (
    SELECT
      p.employee_id,
      json_agg(p.*) as projects
    FROM (
      SELECT
        p.*,
        date_part('day', age(now(), dateassigned::timestamp)) as age
      FROM project p
    ) AS p
    GROUP BY p.employee_id
  )

  SELECT
    e.*,
    null::json as superior,
    COALESCE(ep.projects, '[]') as projects
  FROM employee e
  LEFT JOIN employeeprojects ep
    USING(employee_id)
  WHERE superior_id IS NULL

  UNION ALL

  SELECT
    e.*,
    row_to_json(sup.*) as superior,
    COALESCE(ep.projects, '[]') as projects
  FROM employee e
  INNER JOIN employeetree sup
    ON sup.employee_id = e.superior_id
  LEFT JOIN employeeprojects ep
    ON ep.employee_id = e.employee_id
)

SELECT *
FROM employeetree
WHERE employee_id = 7

This query returns the following nested objects:

{
  "employee_id": 7,
  "name": "Hareth",
  "superior_id": 6,
  "superior": {
     "employee_id": 6,
     "name": "Iron Emmett",
     "superior_id": 1,
     "superior": {
        "employee_id": 1,
        "name": "Jon Snow",
        "superior_id": 4,
        "superior": {
           "employee_id": 4,
           "name": "Jeor Mormont",
           "superior_id": null,
           "superior": null,
           "projects": []
        },
        "projects":[
           {
              "project_id":1,
              "employee_id":1,
              "name":"Infiltrate Mance Rayder's Camp",
              "dateassigned":"2013-09-10",
              "age":12
           }
        ]
     },
     "projects":[]
  },
  "projects": []
}

This is really just the tip of the iceberg, but as you can see there is considerable power available natively from PostgreSQL. It is great to be able to work with data directly from a database query without having to reformat it, use a mapping layer, or run any additional queries.

The Elusive Productivity Zone

Thu, 05 Sep 2013 00:00:00 +0000

When working in software development, and in a startup in particular, making the most efficient use of your time is a critical challenge. It can feel like you are perpetually behind schedule, and the next deadline might just be the one you can’t quite pull off no matter how many cases of Club Maté you consume, as requirements are constantly shifting and priorities reorganizing themselves. This problem is exacerbated by the nature of software development itself.

It is no secret that interruption is terrible for programming, and it’s not hard to figure out why. Imagine trying to solve a long division problem in your head, but before you come up with the answer the phone rings and you get pulled away. How long would it take you to get the right answer after returning to the problem? Surely you could just pick up where you left off? Surely not. This is the primary reason so many programmers get their best work done in the middle of the night. At night there are virtually no interruptions, and you are able to concentrate for extended periods of time.

I sometimes like to say that I have “consciousness inertia,” which is probably just a pretentious way of saying I don’t like to get up early. When I am awake it is easy to make myself stay awake, and when I am asleep my body absolutely does not want to get up. Besides being a lame excuse for not being a morning person, however, there is some real truth to that idea for me on the other end of the equation when working late. If I am able to dig into a problem and really get in “the zone,” code flies off the fingers and it is truly sublime development bliss. Achieving such a level of concentration and engagement with the problem at hand is perpetually elusive, but maintaining that highly productive state is comparatively easy. Hence, if I get “in the zone” late in the day, I really want to maintain that state for as long as possible. This is the other big reason I find myself working late. Fortunately there is actually some evidence that the occasional all-nighter is great for the quality and creativity of your work as well. And I have definitely found this to be true for myself, at least in moderation. In general, there is a tremendous amount of inertia when designing software, and tapping into the positive side of that is critical to facilitating optimal productivity.

So, how does one find the “The Zone”? That is a great question, and I certainly don’t have the definitive answer. But here are a some tips that I’ve found help me.

Caffeine

Seriously- it’s amazing. It consistently surprises me how much a little natural caffeination can really clear your mind and enable concentration. Obviously this advice should be taken within reason, however there is no reputable evidence of any downside of moderate caffeine usage in healthy adults. Did you know that there is a positive correlation between caffeine consumption and the period of rapid technological development we’ve seen over the last few hundred years? Prior to The Enlightenment, the primary beverages consumed were depressants (beer in particular), but during the 1600s people switched to consuming primarily stimulants (coffee and tea). True story. Check out this fascinating video by CGPGrey for more interesting information about caffeine. By far my favorite method of caffeination is loose leaf tea, which is wonderful and full of all sorts of other great health benefits as well. I find the relatively moderate levels of caffeine in tea also help guard against the “crash” that is commonly associated with coffee. Tea is so great it even inspired the name of this blog.

Facebook exists to rob your creativity

This point isn’t exclusively about Facebook, but online distraction in general. However, I think Facebook is probably the most annoying offender for me. I don’t even really like Facebook, and yet for some reason I seem to have this muscle memory where in moments of pause I find myself almost without conscious control typing it into the address bar. It is such a frictionless way to interrupt yourself, and it’s awful for productivity. The latest meme from George Takei can wait, and so can the late breaking news on The Verge about the kind of candy Google will use to name its latest mobile operating system, or whatever important online information is vying for your attention. If you need to add “127.0.0.1 facebook.com” to your /etc/hosts file, so be it. (Special thanks to blissdev for this suggestion). Get offline. You will be so much more productive.

Find some great music

The right music is a huge boon to productivity. It can really help you drown out external noise and distraction (literally) and concentrate on the task at hand. On any given week, I typically find myself listening to an eclectic variety of things. In the last 7 days for example, I have listened to Handel’s Messiah, Michael Buble, Coldplay, Ingrid Michaelson, Christina Aguilera, Enya, the soundtrack from Wicked, the a cappella hip hop group Naturally 7, and a variety of electronic dance music, to name just a few things. However, for me there are a couple music genres that seem to be particularly adept at inducing high productivity. The first is instrumental baroque music, particularly the organ fugues and violin concerti of J.S. Bach. Perhaps it is the interweaving polyphony that helps me tie together threads of thought, I’m not sure. But that kind of music can really help induce some deep and meaningful concentration. The second, and probably that which I listen to most while working, is trance music (I know, how stereotypical of me). Trance is a subgenre of electronic dance music characterized by melodic phrases that build up and down throughout long, uninterrupted tracks. Perhaps it is the “fast tempos and uplifting melodies” (to steal a phrase from the world’s number one DJ, Armin van Buuren) that really help me get in the zone and keep up the mental energy for long sessions of work. I even made a mix of some of my favorite tracks, so I could listen uninterrupted for three hours during times of necessary productivity. These specific generes might not be your cup of tea, but find some that are! The right music can really help isolate your thoughts and remove distractions.

Concentrate

This may seem obvious, but really forcing yourself to concentrate on the problem at hand can be immensely useful. Honestly, all of the above suggestions are just ways to help you concentrate. I find that getting distracted by my own, unrelated thoughts is the most common reason I can’t get “in the zone,” and really making yourself think about the problem at hand initially can often spur much deeper productivity once you get going. Of course, it is so easy to distract yourself by other things, especially those important things you really enjoy, which brings us to my final tip:

Love what you do

There is such a deep connection between happiness and productivity. I recently watched a fascinating TED Talk by Shawn Achor, talking about the effects of positivity on work. There is a tendency to view happiness only as the result of achieving a goal, which inevitably means happiness is never achieved since your goals continually change as you move on to the next goal. However, being happy actually makes it easier to achieve those goals and remain productive. If you really enjoy the work you do, it is unbelievably easier to work efficiently and not get distracted. This is really the reason I’m an entrepreneur. I certainly do not achieve it all the time (don’t ask me about configuring applications for new clients), but finding problems that interest you and creating ways to work on them for your job is an incredible thing.

When I was a kid, I attended a summer art class at the University of Wisconsin-Milwaukee as part of their College for Kids program. I don’t remember much about the content of the course, however, I vividly remember something that the teacher told us about life. He said, “Your worst day at school will be better than your best day at work.” Seriously. A teacher said that! Of all people. That is, of course, complete and utter nonsense. And if it’s true for you, I think you should probably change something. Being the obedient square I was, I believed him at the time. But that really should not be true at all, and I have been fortunate enough that it never really has been for me. Certainly you can have some awful days at work, especially because the responsibility and potential levels of stress are so much more heightened than in school, but it shouldn’t be the norm. In Steve Jobs’ inspirational commencement address at Stanford in 2005 he posed the question, “If today were the last day of my life, would I want to do what I am about to do today? And whenever the answer has been ‘no’ for too many days in a row I know I need to change something.” Loving the work that you do on a day to day basis is the single best way, in my experience, to ensure that you can be consistently productive and “in the zone.” So, find what you love and figure out how to make it your job.