What if Developing a Simple Data Management Application Just Took 1-2 Days?

What if developing an application just took 1-2 days?
What if I can create it myself with only 10 clicks?
What if I don’t need you developers anymore?

Said every manager since the beginning of history. This is what all managers dream of. Click click click, next next next, and you’re done! Easy, right? Time-to-market: Zero.

Of course, we engineers tend to sincerely doubt that this is possible

Data transformation and navigation

Let’s have a look at some tech stuff.

ergonAs a personal passion, I have always loved the idea of non-procedural approaches to manipulating data (e.g. SQL or XSLT). One of the best pieces of software I’ve ever seen to manipulate data was used by Ergon, a previous employer of mine and a customer of Data Geekery who has created a tool called JTT – Java Table Tool, a dinosaur written around 15 years ago. It was essentially a live RDBMS schema and data navigation tool written as a Swing desktop application. With only little metadata, this application was then capable of providing overviews of:

  • All the tables that you as a user had access to
  • When clicking on a table, you got an editable list of all the records in that table with standard filtering and grouping options
  • When double-clicking on a record, you got an editable popup with details
  • When clicking on a record, you got a “children” view with tabs for all foreign keys that link to this table. Obviuosly, the tabs were again filled with records, which could be navigated through the same way as the “parent” records
  • Foreign key values were not displayed using technical IDs, but using relevant data from the linked record

… and much much more. All business logic and complex update rules were implemented using triggers and grants and just a little meta data to decide what information is primary and what information is secondary (or hidden). Most of the views obviously also were exportable to CSV, XLS, or PDF.

Ergon used this wonderful JTT for internal purposes only, e.g. for accounting, invoice management, as a CRM, as an HR tool. It pretty much ran the company and it did its job very very well. It was one of the technically most awesome products that I’ve ever seen. So lean, so simple, and so powerful (albeit, the UI… Oh well, Swing).

I pressed the product manager and the sales managers to consider revitalising this gem and to make a webapp from it that can be sold to other software companies as a product. At the time, something like Eclipse RAP might have been a good choice to allow for a hybrid desktop and web UI.

Unfortunately, they never wanted to make a proper product from this tool. So, I have always thought that at some point, I’ll create my own JTT, I’ll sell it and I’ll get rich. A browser-based database schema and data navigation tool that allows you to set up a basic data management software product in virtually 2-3 days, even when running on large schemas.

Too late, it already exists!

So these were our plans. And then I read this InfoQ article (and later on, also this DZone article by Val Huber, the same author).

“Unfortunately,” for me and for Data Geekery, I have come to discover Espresso Logic’s Live Browser, which does exactly that. Ironically, I have already blogged about an Espresso Logic product before, when I had spotted their pretty cool reactive REST API (where here, reactive means that with a simple rule engine, you could model all sorts of Excel-spreadsheet-like data updates).

But this Live Browser indeed tops what I had in mind from my JTT experience. It is actually built on top of the aforementioned reactive REST API, so it inherits all the nice features, such as the “role-based, row/column level read and update permissions”, the reactive programming features, etc. Here’s an example view from their product announcement website:

Another example from the InfoQ article:

As you can see, pretty much all of the JTT features that I’ve mentioned before are available out of the box:

  • Table selection
  • Filtering
  • Detail views
  • Foreign key navigation
  • Child navigation
  • Data manipulation

(Ergon, if you’re reading this: You see? I told you :-) )

Consider having this as a general-purpose database inspection tool in your company. As a developer, you can quickly navigate the schema (and the data!) in a way that you will never find in Toad or SQL Developer. Obviuosly, the tools don’t compete, as SQL Developer is a database development tool, whereas Live Browser is more of an actual… well, a live data browser.

This browser could also be used as a prototyping engine to assess whether your database schema really models the business case of your customer – a quick display to verify the requirements. Well, as I said, it is a general purpose data browser, that can be used for virtually any simple use-case.

espresso-companyNow, I have again signed up for a free trial at Espresso Logic, to try Live Browser myself. I could log into Logic Designer immediately (as described in my previous article). Do note that there is no single-sign-on in place between the Logic Designer and the Live Browser, so I needed to reuse my credentials to log in also to the browser.

Once I’ve logged in, I could really play around with the data in an easy and straightforward way. All the server-side rules that calculate totals are working as well. E.g. I tried changing the price for a product inside a PurchaseOrder (i.e. inside a LineItem), and it updated the PurchaseOrder’s “Amount Total” value automatically.

I wish I had created this product three years ago when SaaS started getting big. Now, I guess, it’s too late… Congrats, Espresso Logic!

More information

For more info, read Val’s article on DZone

The Power of Spreadsheets in a Reactive, RESTful API

Being mostly a techie, I’ve recently and admittedly been deceived by my own Dilbertesque attitude when I stumbled upon this buzzword-filled TechCrunch article about Espresso Logic. Ever concerned about my social media reputation (e.g. reddit and hackernews karma), I thought it would be witty to put a link on those platforms titled:

Just found this article on TechCrunch. Reads like a markov-chain-generated series of buzzwords.

With such a catchy headline, the post quickly skyrocketed – and like many other redditors, my thoughts were with Geek and Poke:

But like a few other redditors, I couldn’t resist clicking through to the actual product that claims to implement “reactive programming” through a REST and JSON API. And I’m frankly impressed by the ideas behind this product. For once, the buzzwords are backed by software implementing them very nicely! Let’s first delve into…

Reactive Programming

Reactive programming is a term that has gained quite some traction recently around Typesafe, the company behind Akka. It has also gained additional traction since Erik Meijer (creator of LINQ) has left Microsoft to fully dedicate his time to his new company Applied Duality. With those brilliant minds sharply on the topic, we’ll certainly hear more about the Reactive Manifesto in the near future.

excelBut in fact, every manager knows the merits of “reactive programming” already as they’re working with the most reactive and probably the most awesome software on the planet: Microsoft Excel, a device whose mystery is only exceeded by its power. Think about how awesome Excel is. You have hundreds of rules, formulas, cell-interdependencies. And any time you change a value, the whole spreadsheet magically updates itself. That’s Reactive Programming.

The power of reactive programming lies in its expressiveness. With only very little expressive logic, you can express what otherwise needs dozens of lines of SQL, or hundreds of lines of Java.

Espresso Logic

With this in mind, I started to delve into Espresso Logic’s free trial. Note, that I’m the kind of impatient person who wants quick results without reading the docs. In case you work the other way round, there are some interesting resources to get you started:

Anyway, the demo ships with a pre-installed MySQL database containing what looks like a typical E-Commerce schema containing customer, employee, lineitem, product, purchaseorder, and purchaseorder_audit tables:

The schema browsing view in Espresso Logic

The schema browsing view in Espresso Logic

So I get schema navigation information (such as parent / child relationships) and an overview of rules. These rules look like triggers calculating sums or validating things. We’ll get to these rules later on.

Live API

So far, things are as expected. The UI is maybe a bit edgy, as the product only exists since late 2013. But what struck me as quite interesting is what Espresso Logic calls the Live API. With a couple of clicks, I can assemble a REST Resource tree structure from various types of resources, such as database tables. The Espresso Designer will then almost automatically join tables to produce trees like this one:

The Resource Tree view of Espresso Logic

The Resource Tree view of Espresso Logic

Notice how I can connect child entities to their parents quite easily. Now, this API is still a bit limited. For instance, I couldn’t figure out how to drag-and-drop a reporting relationship where I calculate the order amount per customer and product. However, I can switch the Resource Type from “Normal” to “SQL” to achieve just that with a plain old GROUP BY and aggregate function.

I started to grasp that I’m actually managing and developing a RESTful API based on the available database resources! A little further down the menu, I then found the “Quick Ref” item, which helped me understand how to call this API:

A quick API reference

A quick API reference

So, each of the previously defined resources is exposed through a URL as I’d expect from any RESTful API. What looks really nice is that I have built-in API versioning and an API key. Note, it is strongly discouraged from an OWASP point of view to pass API keys around in GET requests. This is just a use-case for a quick-start demo and for the odd developer test. Do not use this in production!

Anyway, I called the URL in my browser with the API key as parameter (going against my own rules):

https://eval.espressologic.com/rest/[my-user]/demo/v1/AllCustomers?auth=[my-key]:1

And I got a JSON document like this:

[
  {
    "@metadata": {
      "href": "https://eval.espressologic.com/rest/[my-user]/demo/v1/Customers/Alpha%20and%20Sons",
      "checksum": "A:cf1f4fb79e8e7142"
    },
    "Name": "Alpha and Sons",
    "Balance": 105,
    "CreditLimit": 900,
    "links": [
    ],
    "Orders": [
      {
        "@metadata": {
          "href": "https://eval.espressologic.com/rest/[my-user]/demo/v1/Customers.Orders/6",
          "checksum": "A:0bf14e2d58cc97b5"
        },
        "OrderNumber": 6,
        "TotalAmount": 70,
        "Paid": false,
        "Notes": "Pack with care - fragile merchandise",
        "links": [
        ], ...

Notice how each resource has a link and a checksum. The checksum is needed for optimistic locking, which is built-in, should you choose to concurrently update any of the above resources. Notice also, how the nested resource Orders is referenced as Customers.Orders. I can also access it directly by calling the above URL.

Live Logic / Reactive Programming

So far so good. Similar things have been implemented in a variety of software. For instance, Adobe Experience Manager / Apache Sling intuitively exposes the JCR repository through REST as well. But where the idea behind Espresso Logic really started fascinating me is when I clicked on “Live Logic”, and I was exposed to a preconfigured set of rules that are applied to the data:

The rules view

The rules view

I’ve quickly skimmed through the manual to see if I understood correctly. These rules actually resemble the kind of rules that I can enter in any spreadsheet software. For instance, it appears as though the customer.balance column is calculated as the sum of all purchaseorder.amount_total having a paid value of false, and so on.

So, if I continue through this rule-chain I’ll wind up with lineitem.product_price being the shared dependency of all other calculated values. When changing that value, a whole set of updates should run through my rule set to finally change the customer.balance:

changing lineitem.product_price
-> changes lineitem.amount
  -> changes purchaseorder.amount_total
    -> changes customer.balance

Depending how much of a console hacker you are, you might want to write your own PUT call using curl, or you can leverage the REST Lab from the Espresso Designer, which helps you get all the parameters right. So, assuming we want to change a line item from the previous call:

{
  "@metadata": {
    "href": "https://eval.espressologic.com/rest/[my_user]/demo/v1/Customers.Orders.LineItems/11",
    "checksum": "A:2e3d8cb0bff42763"
  },
  "lineitem_id": 11,
  "ProductNumber": 2,
  "OrderNumber": 6,
  "Quantity": 2,
  "Price": 25,
  ...

Let’s just try to update that to have a price of 30:

Using the REST lab to execute PUT requests

Using the REST lab to execute PUT requests

And you can see in the response, there is a transaction summary, which shows that the Customers.Orders.TotalAmount has changed from 50 to 60, the Customers.Balance has changed from 105 to 95, and an audit record has been written. The audit record itself is also defined by a rule like any other rule. But there’s also an ordinary log file that shows what really happened when I ran this PUT request:

The log view showing all the INSERTs and UPDATEs executed

The log view showing all the INSERTs and UPDATEs executed

Imagine having to put all those INSERT and UPDATE statements into a correct order yourself, and correctly manage caching, and transactions! Instead, all we have done is define some rules. For a complete overview of what rule types are available, consider this page of the Live Logic manual

Out of scope features for this post

… So far, we’ve had a look at the most obvious features of Espresso Logic. There are more, though. A couple of examples:

Server-side JavaScript

If rules cannot express it, JavaScript can. There are various points of the application where you can inject your JavaScript snippets, e.g. for validation, more complex rule expressions, request and response transformation, etc. Although we haven’t tried it, it reads like row-based triggers written in JavaScript.

Stored procedure support

The people behind Espresso Logic are “legacy-embracing” people, just like us at Data Geekery. Their target audience might already have thousands of complex stored procedures with lots of business logic in them. Those should not be rewritten in JavaScript. But just like tables, views, and REST resources, they are exposed through the REST API, taking GET parameters for IN parameters and returning JSON for OUT parameters and cursors.

From a jOOQ perspective, it’s pretty awesome to see that someone else is taking stored procedures as seriously as we do.

Row / column level security

There is a built-in user and role management module that allows you to provide centrally-managed, fine-grained access control to your data. Not many databases support row-level security like the Oracle database, for instance. So having this kind of feature in your platform really adds value to many RDBMS integrations. Some further resources on that topic:

Conclusion: Querying vs. updating vs. rule-based persistence

On our jOOQ blog and our marketing websites (e.g. hibernate-alternative.com), we always advocate two main use-cases when operating on databases:

  • Querying: You have very complex queries to calculate things like reports. For this, SQL (e.g. through jOOQ) is perfect
  • Updating: You have a very complex domain model with lots of items and deltas that you want to persist in one go. For this, Hibernate / ORMs are perfect

But today, Espresso Logic has shown to us that there is yet another use-case. One that is covered by reactive programming (or “spreadsheet-programming“) techniques. And that’s:

  • Rule-based persistence: You have a very complex domain model with lots of items and lots of rules which you want to validate, calculate, and keep in sync all the time. For this, both SQL and ORMs are solutions at the wrong level of abstraction.

This “new” use-case is actually quite common in a lot of enterprise applications where complex business rules are currently spelled out in millions of lines of imperative code that is very hard to decipher and even harder to validate / modify. How can you reverse-engineer your business rules from millions of lines of legacy code, written in COBOL?

At Data Geekery, we’re always looking out for brand new tech. Espresso Logic is a young startup with a new product. Yet, as originally mentioned, they’re a startup with seed funding, a very compelling and innovative idea, and a huge market of legacy COBOL applications that wants to start delving into “sexy” new technologies, such as RESTful APIs, JSON, reactive programming. It might just work! If you haven’t seen enough, go work through this tutorial, which covers advanced examples such as a “bill of materials price rollup”, “bill of materials kit explosion”, “budget rollup”, “audit salary chagnes” and more.

We’ll certainly keep an eye out for future improvements to the Espresso Logic platform!