The Caveats of Dual-Licensing

We’ve been in business for more than one year now with our dual-licensing strategy for jOOQ. While this strategy has worked very well for us, it has also been a bit of a challenge for some of our customers. Today, we’re going to show you what caveats of dual-licensing we’ve run into.

Our dual-licensing strategy

For those of you not acquainted with our license model, just a brief reminder to get you into the subject. We mainly consider ourselves as a vendor of Open Source software. However, contrary to a variety of other companies like Gradleware or Red Hat, we don’t want to build our business model on support, which is a much tougher business than licensing. Why?

  • Support contracts need a lot more long-term trust by customers, and more outbound sales. There’s only little inbound interest for such contracts, as people don’t acquire support until they need it.
  • Vendor-supplied support competes with third-party support (as provided by UWS, for instance), which we want to actively encourage. We’d love to generate business for an entirely new market, not compete with our allies.

So we were looking for a solution involving commercial licensing. We wanted to keep an Open Source version of our product because:

  • We’ll get traction with Open Source licensing much much more quickly than with commercial licensing
  • While Open Source is a very tough competitor for vendors, it is also a great enabler for consumers. For instance, we’re using Java, Eclipse, H2, and much more. Great software for free!

It wouldn’t be honest to say that we truly believe in “free as in freedom” (libre), but we certainly believe in “free as in beer” (gratis) – because, who doesn’t. So, one very simple solution to meet the above goals was to offer jOOQ as Open Source with Open Source databases, and to offer jOOQ under a commercial license with commercial databases.

The Caveat

This was generally well received with our user base as a credible and viable dual-licensing model. But there were also caveats. All of a sudden, we didn’t have access to these distribution channels any more:

  • Maven Central
  • GitHub

… and our paying customers didn’t have access to these very useful OSS rights any more:

  • Source Code
  • Modifications

Solution 1 – Ship Source Code

Well, we actually ship our source code with the commercial binaries. At first, this was done merely for documentation purposes.

Regardless of the actual license constraints, when you’re in trouble, e.g. when your productive system is down and you have to urgently fix a bug, doesn’t it just suck if you don’t have access to the source code of third-party dependencies? You will just have to guess what it does. Or illegally de-compile it.

We don’t want to be that company. We trust our customers to deal responsibly with our source code.

Solution 2 – Allow Modifications

Our commercial licenses come in two flavours: Yearly and Perpetual. We quickly realised that some of our customers do not want to be dependent on us as a vendor. Perpetual licenses obviously help making customers more independent. But the disadvantage of perpetual licenses is the fact that vendors will not support old versions forever, and customers won’t have the right to upgrade to the next major release. While they are probably fine with not having access to new features, they would still like to receive an occasional bugfix.

The solution we’ve come to adopt is a very pragmatic one: Customers already have the source code (see above), so why not allow customers to also apply urgent fixes themselves? Obviously, such modifications will void the warranty offered by us, but if you buy jOOQ today and 5 years down the line, you discover a very subtle bug in what will then be an unsupported version of jOOQ… don’t you just want to fix it?


Dual-licensing is a tricky business. You partition your user-base into two:

  • The paying / premium customers
  • The “freemium” customers

By all means, you must prevent your premium customers from being at a disadvantage compared to your “freemium” customers. There are certain rights that are probably OK to remove (such as the right of free distribution). But there are rights that are just annoying not to have. And those rights are the rights that matter the most to the every day work of an engineer:

To fix that bloody bug :-)

We’re very curious: What are your opinions towards dual-licensing?

Using Your RDBMS for Messaging is Totally OK

Controversial database topics are a guaranteed success on reddit, because everyone has an opinion on those topics. More importantly, many people have a dogmatic opinion, which always triggers more debate than pragmatism.

So, recently, I posted a link to an older article titled The Database As Queue Anti-Pattern by Mike Hadlow, and it got decent results on /r/programming:


Mike’s post was pretty much against all sorts of queueing in the database, which matches the opinions I have heard from a couple of JavaZone speakers in a recent discussion, who all agreed that messaging in the database is “evil.”

… and I’m saying: No. Messaging in the database is not an anti-pattern, it is (can be) totally OK. Let’s consider why:


First off, if you don’t plan on deploying thousands of message types and millions of messages per hour, then you might have a very simple messaging problem. Since you’re already using an RDBMS, using that RDBMS also for messaging is definitely an option.

Obviously, many of you will now think:

If all you have is a hammer, everything looks like a nail

… and you’re right! But guess what, the reasons for only having a hammer can be any of:

  • You don’t have the time to equip with a sophisticated tool box
  • You don’t have the money to equip with a sophisticated tool box
  • You actually don’t really need the sophisticated tool box

Think about these arguments. Yes, the solution might not be perfect, or even ugly…

The PHP Hammer

But we’re engineers and as such, we’re not here to debate perfection. We’re here to deliver value to our customers and if we can get the job done with the hammer, why not just forget our vanity and simply use that hammer to get the job done.

Transactional queues

In a perfect world, queues are transactional and guarantee (to the extent allowed by underlying theory) atomic message delivery or failure – something that we’ve been taking for granted with JMS forever.

At GeeCON Krakow, I had a very interesting discussion with Konrad Malawski regarding his talk about Akka Persistence. If we remove all the hype and buzzwords (e.g. reactive programming, etc.) from Akka, we can see that Akka is just a proprietary alternative to JMS, which looks more modern but is lacking tons of features that we’re used to having in JMS (e.g. like transactional queue persistence).

One of the interesting aspects of that discussion with Konrad Malawski is the fact that a 100% message delivery guarantee is a myth (details here). Which leads to the conclusion:

Messaging is really hard

It is, indeed! So if you really think you need to embed a sophisticated MQ system, beware of the fact that you will have to learn how it really works and how to correctly operate it.

If you’re using RDBMS-backed queues, you can get rid of this additional transactional complexity, because your queue operations participate in the transactions that you already have with your database. You get ACID for free!

No additional operations efforts

What developers very often underestimate (we can’t say this enough) are the costs incurring to your operations team when you add new external systems to yours.

Having just one simple RDBMS (and your own application) is a very very lean and simple architecture. Having an RDBMS, an MQ, and your application is already more complex.

There are a lot of excellent DBA out there who know what they’re doing when operating productive databases. Finding excellent “MQA” is much harder.

If you’re using Oracle: Use Oracle AQ

Oracle has a very sophisticated built-in queueing API called Oracle AQ, which can interoperate with JMS.

Queues in AQ are essentially just tables that contain a serialised version of your message type. If you’re using jOOQ, we’ve blogged about how to integrate Oracle AQ with jOOQ, recently.

RDBMS-centric applications can be much easier

We’ve blogged about that before as well: Why Your Boring Data Will Outlast Your Sexy New Technology.

Your data might just survive your application. Consider Paypal replacing Java with JavaScript (it could also have gone the other way round). In the end, however, do you think that Paypal also replaced all their databases? I don’t. Migrating from Oracle to DB2 (different vendor), or from Oracle to MongoDB (different DBMS type) is mostly motivated by political decisions rather than technical ones. Specifically, people don’t migrate from RDBMS to NoSQL databases entirely. They usually just implement a specific domain with NoSQL (e.g. document storage, or graph traversal)

Assuming that the above really applies to you (it may, of course, not apply): If your RDBMS is in the middle of your system, then running queues in your RDBMS to communicate between system components is quite an obvious choice, isn’t it? All system parts are already connected to the database. Why not keep it that way?


The arguments listed here are all pretty obvious and pragmatic. At some point, they no longer hold true, as your messaging demands are really big enough to justify the integration with a sophisticated MQ system.

But many people have strong opinions about the “hammer / nail” argument. Those opinions may be correct but premature. Very often in software engineering, it is entirely acceptable and sufficient to work with just one tool. The hammer of software: The RDBMS.

Asynchronous SQL Execution with jOOQ and Java 8’s CompletableFuture

Reactive programming is the new buzzword, which essentially just means asynchronous programming or messaging.

Fact is that functional syntax greatly helps with structuring asynchronous execution chains, and today, we’ll see how we can do this in Java 8 using jOOQ and the new CompletableFuture API.

In fact, things are quite simple:

// Initiate an asynchronous call chain

    // This lambda will supply an int value
    // indicating the number of inserted rows
    .supplyAsync(() -> DSL
        .values(3, "Hitchcock")
    // This will supply an AuthorRecord value
    // for the newly inserted author
    .handleAsync((rows, throwable) -> DSL
        .fetchOne(AUTHOR, AUTHOR.ID.eq(3))

    // This should supply an int value indicating
    // the number of rows, but in fact it'll throw
    // a constraint violation exception
    .handleAsync((record, throwable) -> {
        return record.insert();
    // This will supply an int value indicating
    // the number of deleted rows
    .handleAsync((rows, throwable) -> DSL

    // This tells the calling thread to wait for all
    // chained execution units to be executed

What did really happen here? Nothing out of the ordinary. There are 4 execution blocks:

  1. One that inserts a new AUTHOR
  2. One that fetches that same AUTHOR again
  3. One that re-inserts the newly fetched AUTHOR (throwing an exception)
  4. One that ignores the thrown exception and delets the AUTHOR again

Finally, when the execution chain is established, the calling thread will join the whole chain using the CompletableFuture.join() method, which is essentially the same as the Future.get() method, except that it doesn’t throw any checked exception.

Comparing this to other APIs

Other APIs like Scala’s Slick have implemented similar things via “standard API”, such as calls to flatMap(). We’re currently not going to mimick such APIs as we believe that the new Java 8 APIs will become much more idiomatic to native Java speakers. Specifically, when executing SQL, getting connection pooling and transactions right is of the essence. The semantics of asynchronously chained execution blocks and how they relate to transactions is very subtle. If you want a transaction to span more than one such block, you will have to encode this yourself via jOOQ’s Configuration and its contained ConnectionProvider.

Blocking JDBC

Obviously, there will always be one blocking barrier to such solutions, and that is JDBC itself – which is very hard to turn into an asynchronous API. In fact, few databases really support asynchronous query executions and cursors, as most often, a single database session can only be used by a single thread for a single query at a time.

We’d be very interested to learn about your asynchronous SQL querying requirements, so feel free to comment on this post!

Top 10 Most Popular Articles on the jOOQ Blog

What do people do when they run out of topics? They recycle previous topics and create top 10 lists. Here is a list of the top 10 most popular articles from the jOOQ blog:

  1. Top 10 Very Very VERY Important Topics to Discuss
    A fun, not so serious parody on what is being discussed on reddit’s /r/programming. Hint: Bikeshedding topics are the most popular. Like this one. That was so meta!
  2. 10 Subtle Best Practices when Coding Java
    This is a really interesting article about not-so-common advice that might be handy every once in a while.
  3. 10 Common Mistakes Java Developers Make when Writing SQL
    A classic and must-read for all SQL developers (not only those that usually write Java)
  4. SQL Trick: row_number() is to SELECT what dense_rank() is to SELECT DISTINCT
    We’re surprised ourselves that this is so popular. But it appears that we’re really well ranked on Google when people are looking for ROW_NUMBER() and DENSE_RANK(). And the trick is very useful, of course!
  5. Why You Should NOT Implement Layered Architectures
    What a silly rant! And how it went up in the ranking within only two days! This is not really very serious advice. Obviuosly, you should (as always) do what fits best to your problem domain. But we wanted to make people think about the status quo and how it is often applied too rigidly, without thinking about all the options. Looks like we’ve hit a sweet spot with developers frustrated with overengineered applications…
  6. MIT Prof. Michael Stonebraker: “The Traditional RDBMS Wisdom is All Wrong”
    Michael Stonebraker is a very controversial person per se. In this article, we’re linking to a talk by Stonebraker where he claims (again) that the RDBMS end is nigh. A year later, we can see that NoSQL is still on the rise, whereas NewSQL is still no where. See also the next article…
  7. The 10 Most Popular DB Engines (SQL and NoSQL)
    This is an interpretation of a popular ranking of (R)DBMS, showing that even if Oracle, MySQL, and SQL Server are the most wide-spread databases, something’s about to change.
  8. Does Java 8 Still Need LINQ? Or is it Better than LINQ?
    Again, controversy is king. Of course, LINQ is awesome and often we wish we had something like LINQ in Java. In this article, however, we’re claiming that with Java 8’s Streams API and lambda expressions, we might no longer need LINQ, as collections transformation is already sufficiently covered, and LINQ-to-SQL is not what made LINQ popular (which is where jOOQ is more useful)
  9. 10 More Common Mistakes Java Developers Make when Writing SQL
    A follow-up article to the previous, very popular article about common SQL mistakes. Yes, there’s a lot to learn in this area.
  10. The Java Fluent API Designer Crash Course
    From a jOOQ perspective, this is one of the most interesting articles explaining the very simple and easy-to-apply rules that we’re using to produce our API in the form of an internal domain-specific language. If you want to build jOOQ for your own query language (e.g. Cassandra’s CQL), just follow these simple rules

Thanks for reading our blog! We promise to keep you up to date with more interesting (and occasionally useless) content!

How Nashorn Impacts API Evolution on a New Level

Following our previous article about how to use jOOQ with Java 8 and Nashorn, one of our users discovered a flaw in using the jOOQ API as discussed here on the user group. In essence, the flaw can be summarised like so:

Java code

package org.jooq.nashorn.test;

public class API {
    public static void test(String string) {
        throw new RuntimeException("Don't call this");

    public static void test(Integer... args) {

JavaScript code

var API = Java.type("org.jooq.nashorn.test.API");
API.test(1); // This will fail with RuntimeException

After some investigation and the kind help of Attila Szegedi, as well as Jim Laskey (both Nashorn developers from Oracle), it became clear that Nashorn disambiguates overloaded methods and varargs differently than what an old Java developer might expect. Quoting Attila:

Nashorn’s overload method resolution mimics Java Language Specification (JLS) as much as possible, but allows for JavaScript-specific conversions too. JLS says that when selecting a method to invoke for an overloaded name, variable arity methods can be considered for invocation only when there is no applicable fixed arity method.

I agree that variable arity methods can be considered only when there is no applicable fixed arity method. But the whole notion of “applicable” itself is completely changed as type promotion (or coercion / conversion) using ToString, ToNumber, ToBoolean is preferred over what intuitively appear to be “exact” matches with varargs methods!

Let this sink in!

Given that we now know how Nashorn resolves overloading, we can see that any of the following are valid workarounds:

Explicitly calling the test(Integer[]) method using an array argument:

This is the simplest approach, where you ignore the fact that varargs exist and simply create an explicit array

var API = Java.type("org.jooq.nashorn.test.API");

Explicitly calling the test(Integer[]) method by saying so:

This is certainly the safest approach, as you’re removing all ambiguity from the method call

var API = Java.type("org.jooq.nashorn.test.API");

Removing the overload:

public class AlternativeAPI1 {
    public static void test(Integer... args) {

Removing the varargs:

public class AlternativeAPI3 {
    public static void test(String string) {
        throw new RuntimeException("Don't call this");

    public static void test(Integer args) {

Providing an exact option:

public class AlternativeAPI4 {
    public static void test(String string) {
        throw new RuntimeException("Don't call this");

    public static void test(Integer args) {
        test(new Integer[] { args });

    public static void test(Integer... args) {

Replacing String by CharSequence (or any other “similar type”):

Now, this is interesting:

public class AlternativeAPI5 {
    public static void test(CharSequence string) {
        throw new RuntimeException("Don't call this");

    public static void test(Integer args) {

Specifically, the distinction between CharSequence and String types appears to be very random from a Java perspective in my opinion.

Agreed, implementing overloaded method resolution in a dynamically typed language is very hard, if even possible. Any solution is a compromise that will introduce flaws at some ends. Or as Attila put it:

As you can see, no matter what we do, something else would suffer; overloaded method selection is in a tight spot between Java and JS type systems and very sensitive to even small changes in the logic.

True! But not only is overload method selection very sensitive to even small changes. Using Nashorn with Java interoperability is, too! As an API designer, over the years, I have grown used to semantic versioning, and the many subtle rules to follow when keeping an API source compatible, behavior compatible – and if ever possible – to a large degree also binary compatible.

Forget about that when your clients are using Nashorn. They’re on their own. A newly introduced overload in your Java API can break your Nashorn clients quite badly. But then again, that’s JavaScript, the language that tells you at runtime that:


… yields

[10, NaN, 2, 3]

… and where

++[[]][+[]]+[+[]] === "10"

yields true! (sources here)

For more information about JavaScript, please visit this introductory tutorial

This Just In!! Updated Silly Keyword Metrics

It’s this time of the year and we’re repeating the silly metrics keyword count across all of jOOQ‘s code base.

We have a couple of exciting news! public is still one of the favourite keywords (yeah, we’re an API), but return is closing in, as well as final (yeah, we love final).

A bit more surprising:

  • if has surpassed throws and void. Is jOOQ becoming less object oriented, and more imperative?
  • true is still more popular than false. Yeah, we’re thinking positively about life
  • continue has one more instance in our code base. Gotta love the occasional continue!
  • do experienced a 600% increase!
  • … we still try more than we catch
  • … does the rise of char mean that we’re parsing more SQL?
  • … and we’ve removed one instance of volatile
  • … and still no strictfp or native
2013                2014                   
Keyword      Count  Keyword      Count     
public       8127   public       9379      
return       6801   return       8079      
final        6608   final        7561      
import       5938   import       7232      
static       3903   static       5154      
new          3110   new          3915      
extends      2111   extends      2884      
int          1822   int          2132      
throws       1756   if           1985      
void         1707   throws       1898      
if           1661   void         1834      
this         1464   this         1803      
private      1347   private      1605      
class        1239   class        1437      
case         841    case         1225      
else         839    else         940       
package      711    package      842       
boolean      506    boolean      623       
throw        495    throw        553       
for          421    for          469       
long         404    long         456       
true         384    true         439       
byte         345    interface    407       
interface    337    byte         397       
false        332    false        396       
protected    293    break        357       
super        265    protected    328       
break        200    super        328       
try          149    switch       197       
switch       146    try          193       
implements   139    catch        167       
catch        127    implements   162       
default      112    default      156       
instanceof   107    instanceof   156       
char         96     char         122       
short        91     short        93        
abstract     54     finally      54        
double       43     abstract     50        
transient    42     transient    45        
finally      34     double       44        
float        34     float        35        
enum         25     while        35        
while        23     enum         31        
continue     12     continue     13        
synchronized 8      synchronized 10        
volatile     6      do           6         
do           1      volatile     5  

Download the free and ASL 2.0 licensed “silly-metrics” program and start counting keywords in your software.

This is the Final Discussion!

Pun intended… Let’s discuss Java final.

Recently, our popular blog post “10 Subtle Best Practices when Coding Java” had a significant revival and a new set of comments as it was summarised and linked from JavaWorld. In particular, the JavaWorld editors challenged our opinion about the Java keyword “final“:

More controversially, Eder takes on the question of whether it’s ever safe to make methods final by default:

“If you’re in full control of all source code, there’s absolutely nothing wrong with making methods final by default, because:”

  • “If you do need to override a method (do you really?), you can still remove the final keyword”
  • “You will never accidentally override any method anymore”

Yes, indeed. All classes, methods, fields and local variables should be final by default and mutable via keyword.

Here are fields and local variables:

    int finalInt   = 1;
val int finalInt   = 2;
var int mutableInt = 3;

Whether the Scala/C#-style val keyword is really necessary is debatable. But clearly, in order to modify a field / variable ever again, we should have a keyword explicitly allowing for it. The same for methods – and I’m using Java 8’s default keyword for improved consistency and regularity:

class FinalClass {
    void finalMethod() {}

default class ExtendableClass {
            void finalMethod      () {}
    default void overridableMethod() {}

That would be the perfect world in our opinion, but Java goes the other way round making default (overridable, mutable) the default and final (non-overridable, immutable) the explicit option.

Fair enough, we’ll live with that

… and as API designers (from the jOOQ API, of course), we’ll just happily put final all over the place to at least pretend that Java had the more sensible defaults mentioned above.

But many people disagree with this assessment, mostly for the same reason:

As someone who works mostly in osgi environments, I could not agree more, but can you guarantee that another api designer felt the same way? I think it’s better to preempt the mistakes of api designers rather than preempt the mistakes of users by putting limits on what they can extend by default. – eliasv on reddit


Strongly disagree. I would much rather ban final and private from public libraries. Such a pain when I really need to extend something and it cannot be done.

Intentionally locking the code can mean two things, it either sucks, or it is perfect. But if it is perfect, then nobody needs to extend it, so why do you care about that.

Of course there exists valid reasons to use final, but fear of breaking someone with a new version of a library is not one of them. – meotau on reddit

Or also…

I know we’ve had a very useful conversation about this already, but just to remind other folks on this thread: much of the debate around ‘final’ depends on the context: is this a public API, or is this internal code? In the former context, I agree there are some good arguments for final. In the latter case, final is almost always a BAD idea. – Charles Roth on our blog

All of these arguments tend to go into one direction: “We’re working on crappy code so we need at least some workaround to ease the pain.”

But why not think about it this way:

The API designers that all of the above people have in mind will create precisely that horrible API that you’d like to patch through extension. Coincidentally, the same API designer will not reflect on the usefulness and communicativeness of the keyword final, and thus will never use it, unless required by the Java language. Win-win (albeit crappy API, shaky workarounds and patches).

The API designers that want to use final for their API will reflect a lot on how to properly design APIs (and well-defined extension points / SPIs), such that you will never worry about something being final. Again, win-win (and an awesome API).

Plus, in the latter case, the odd hacker will be kept from hacking and breaking your API in a way that will only lead to pain and suffering, but that’s not really a loss.

Final interface methods

For the aforementioned reasons, I still deeply regret that final is not possible in Java 8 interfaces. Brian Goetz has given an excellent explanation why this has been decideed upon like that. In fact, the usual explanation. The one about this not being the main design goal for the change ;-)

But think about the consistency, the regularity of the language if we had:

default interface ImplementableInterface {
            void abstractMethod   () ;
            void finalMethod      () {}
    default void overridableMethod() {}

(Ducks and runs…)

Or, more realistically with our status quo of defaulting to default:

interface ImplementableInterface {
          void abstractMethod   () ;
    final void finalMethod      () {}
          void overridableMethod() {}


So again, what are your (final) thoughts on this discussion?

If you haven’t heard enough, consider also reading this excellent post by Dr. David Pearce, author of the whiley programming language

Why You Should NOT Implement Layered Architectures

Abstraction layers in software are what architecture astronauts tell you to do. Instead, however, half of all applications out there would be so easy, fun, and most importantly: productive to implement if you just got rid of all those layers.

Frankly, what do you really need? You need these two:

  • Some data access
  • Some UI

Because that’s the two things that you inevitably have in most systems. Users, and data. Here’s Kyle Boon’s opinion on possible choices that you may have

Very nice choice, Kyle. Ratpack and jOOQ. You could choose any other APIs, of course. You could even choose to write JDBC directly in JSP. Why not. As long as you don’t go pile up 13 layers of abstraction:

Geek and Poke's Footprints - Licensed CC-BY 2.0
Geek and Poke’s Footprints – Licensed CC-BY 2.0

That’s all bollocks, you’re saying? We need layers to abstract away the underlying implementation so we can change it? OK, let’s give this some serious thought. How often do you really change the implementation? Some examples:

  • SQL. You hardly change the implementation from Oracle to DB2
  • DBMS. You hardly change the model from relational to flat or XML or JSON
  • JPA. You hardly switch from Hibernate to EclipseLink
  • UI. You simply don’t replace HTML with Swing
  • Transport. You just don’t switch from HTTP to SOAP
  • Transaction layer. You just don’t substitute JavaEE with Spring, or JDBC transactions

Nope. Your architecture is probably set in stone. And if – by the incredible influence of entropy and fate – you happen to have made the wrong decision in one aspect, about 3 years ago, well you’re in for a major refactoring anyway. If SQL was the wrong choice, well good luck to you migrating everything to MongoDB (which is per se the wrong choice again, so prepare for migrating back). If HTML was the wrong choice, well even more tough luck to you. Likelihood of your layers not really helping you when a concrete incident happens: 95% (because you missed an important detail)

Layers = Insurance

If you’re still thinking about implementing an extremely nice layered architecture, ready to deal with pretty much every situation where you simply switch a complete stack with another, then what you’re really doing is filing a dozen insurance policies. Think about it this way. You can get:

  • Legal insurance
  • Third party insurance
  • Reinsurance
  • Business interruption insurance
  • Business overhead expense disability insurance
  • Key person insurance
  • Shipping insurance
  • War risk insurance
  • Payment protection insurance
  • pick a random category

You can pay and pay and pay in advance for things that probably won’t ever happen to you. Will they? Yeah, they might. But if you buy all that insurance, you pay heavily up front. And let me tell you a secret. IF any incident ever happens, chances are that you:

  • Didn’t buy that particular insurance
  • Aren’t covered appropriately
  • Didn’t read the policy
  • Got screwed

And you’re doing exactly that in every application that would otherwise already be finished and would already be adding value to your customer, while you’re still debating if on layer 37 between the business rules and transformation layers, you actually need another abstraction because the rule engine could be switched any time.

Stop doing that

You get the point. If you have infinite amounts of time and money, implement an awesome, huge architecture up front.

Your competitor’s time to market (and fun, on the way) is better than yours. But for a short period of time, you were that close to the perfect, layered architecture!

When the Java 8 Streams API is not Enough

Java 8 was – as always – a release of compromises and backwards-compatibility. A release where the JSR-335 expert group might not have agreed upon scope or feasibility of certain features with some of the audience. See some concrete explanations by Brian Goetz about why …

But today we’re going to focus on the Streams API’s “short-comings”, or as Brian Goetz would probably put it: things out of scope given the design goals.

Parallel Streams?

Parallel computing is hard, and it used to be a pain. People didn’t exactly love the new (now old) Fork / Join API, when it was first shipped with Java 7. Conversely, and clearly, the conciseness of calling Stream.parallel() is unbeatable.

But many people don’t actually need parallel computing (not to be confused with multi-threading!). In 95% of all cases, people would have probably preferred a more powerful Streams API, or perhaps a generally more powerful Collections API with lots of awesome methods on various Iterable subtypes.

Changing Iterable is dangerous, though. Even a no-brainer as transforming an Iterable into a Stream via a potential method seems to risk opening pandora’s box!.

Sequential Streams!

So if the JDK doesn’t ship it, we create it ourselves!

Streams are quite awesome per se. They’re potentially infinite, and that’s a cool feature. Mostly – and especially with functional programming – the size of a collection doesn’t really matter that much, as we transform element by element using functions.

If we admit Streams to be purely sequential, then we could have any of these pretty cool methods as well (some of which would also be possible with parallel Streams):

  • cycle() – a guaranteed way to make every stream infinite
  • duplicate() – duplicate a stream into two equivalent streams
  • foldLeft() – a sequential and non-associative alternative to reduce()
  • foldRight() – a sequential and non-associative alternative to reduce()
  • limitUntil() – limit the stream to those records before the first one to satisfy a predicate
  • limitWhile() – limit the stream to those records before the first one not to satisfy a predicate
  • maxBy() – reduce the stream to the maximum mapped value
  • minBy() – reduce the stream to the minimum mapped value
  • partition() – partition a stream into two streams, one satisfying a predicate and the other not satisfying the same predicate
  • reverse() – produce a new stream in inverse order
  • skipUntil() – skip records until a predicate is satisified
  • skipWhile() – skip records as long as a predicate is satisfied
  • slice() – take a slice of the stream, i.e. combine skip() and limit()
  • splitAt() – split a stream into two streams at a given position
  • unzip() – split a stream of pairs into two streams
  • zip() – merge two streams into a single stream of pairs
  • zipWithIndex() – merge a stream with its corresponding stream of indexes into a single stream of pairs

jOOλ’s new Seq type does all that

All of the above is part of jOOλ. jOOλ (pronounced “jewel”, or “dju-lambda”, also written jOOL in URLs and such) is an ASL 2.0 licensed library that emerged from our own development needs when implementing jOOQ integration tests with Java 8. Java 8 is exceptionally well-suited for writing tests that reason about sets, tuples, records, and all things SQL.

But the Streams API just slightly feels insufficient, so we have wrapped JDK’s Streams into our own Seq type (Seq for sequence / sequential Stream):

// Wrap a stream in a sequence
Seq<Integer> seq1 = seq(Stream.of(1, 2, 3));

// Or create a sequence directly from values
Seq<Integer> seq2 = Seq.of(1, 2, 3);

We’ve made Seq a new interface that extends the JDK Stream interface, so you can use Seq fully interoperably with other Java APIs – leaving the existing methods unchanged:

public interface Seq<T> extends Stream<T> {

     * The underlying {@link Stream} implementation.
    Stream<T> stream();
	// [...]

Now, functional programming is only half the fun if you don’t have tuples. Unfortunately, Java doesn’t have built-in tuples and while it is easy to create a tuple library using generics, tuples are still second-class syntactic citizens when comparing Java to Scala, for instance, or C# and even VB.NET.


jOOλ also has tuples

We’ve run a code-generator to produce tuples of degree 1-8 (we might add more in the future, e.g. to match Scala’s and jOOQ’s “magical” degree 22).

And if a library has such tuples, the library also needs corresponding functions. The essence of these TupleN and FunctionN types is summarised as follows:

public class Tuple3<T1, T2, T3>
	Comparable<Tuple3<T1, T2, T3>>, 
	Serializable, Cloneable {
    public final T1 v1;
    public final T2 v2;
    public final T3 v3;
	// [...]


public interface Function3<T1, T2, T3, R> {

    default R apply(Tuple3<T1, T2, T3> args) {
        return apply(args.v1, args.v2, args.v3);

    R apply(T1 v1, T2 v2, T3 v3);

There are many more features in Tuple types, but let’s leave them out for today.

On a side note, I’ve recently had an interesting discussion with Gavin King (the creator of Hibernate) on reddit. From an ORM perspective, Java classes seem like a suitable implementation for SQL / relational tuples, and they are indeed. From an ORM perspective.

But classes and tuples are fundamentally different, which is a very subtle issue with most ORMs – e.g. as explained here by Vlad Mihalcea.

Besides, SQL’s notion of row value expressions (i.e. tuples) is quite different from what can be modelled with Java classes. This topic will be covered in a subsequent blog post.

Some jOOλ examples

With the aforementioned goals in mind, let’s see how the above API can be put to work by example:


// (tuple(1, "a"), tuple(2, "b"), tuple(3, "c"))
Seq.of(1, 2, 3).zip(Seq.of("a", "b", "c"));

// ("1:a", "2:b", "3:c")
Seq.of(1, 2, 3).zip(
    Seq.of("a", "b", "c"), 
    (x, y) -> x + ":" + y

// (tuple("a", 0), tuple("b", 1), tuple("c", 2))
Seq.of("a", "b", "c").zipWithIndex();

// tuple((1, 2, 3), (a, b, c))
    tuple(1, "a"),
    tuple(2, "b"),
    tuple(3, "c")

This is already a case where tuples have become very handy. When we “zip” two streams into one, we want a wrapper value type that combines both values. Classically, people might’ve used Object[] for quick-and-dirty solutions, but an array doesn’t indicate attribute types or degree.

Unfortunately, the Java compiler cannot reason about the effective bound of the <T> type in Seq<T>. This is why we can only have a static unzip() method (instead of an instance one), whose signature looks like this:

// This works
static <T1, T2> Tuple2<Seq<T1>, Seq<T2>> 
    unzip(Stream<Tuple2<T1, T2>> stream) { ... }
// This doesn't work:
interface Seq<T> extends Stream<T> {
    Tuple2<Seq<???>, Seq<???>> unzip();

Skipping and limiting

// (3, 4, 5)
Seq.of(1, 2, 3, 4, 5).skipWhile(i -> i < 3);

// (3, 4, 5)
Seq.of(1, 2, 3, 4, 5).skipUntil(i -> i == 3);

// (1, 2)
Seq.of(1, 2, 3, 4, 5).limitWhile(i -> i < 3);

// (1, 2)
Seq.of(1, 2, 3, 4, 5).limitUntil(i -> i == 3);

Other functional libraries probably use different terms than skip (e.g. drop) and limit (e.g. take). It doesn’t really matter in the end. We opted for the terms that are already present in the existing Stream API: Stream.skip() and Stream.limit()


// "abc"
Seq.of("a", "b", "c").foldLeft("", (u, t) -> t + u);

// "cba"
Seq.of("a", "b", "c").foldRight("", (t, u) -> t + u);

The Stream.reduce() operations are designed for parallelisation. This means that the functions passed to it must have these important attributes:

But sometimes, you really want to “reduce” a stream with functions that do not have the above attributes, and consequently, you probably don’t care about the reduction being parallelisable. This is where “folding” comes in.

A nice explanation about the various differences between reducing and folding (in Scala) can be seen here.


// tuple((1, 2, 3), (1, 2, 3))
Seq.of(1, 2, 3).duplicate();

// tuple((1, 3, 5), (2, 4, 6))
Seq.of(1, 2, 3, 4, 5, 6).partition(i -> i % 2 != 0)

// tuple((1, 2), (3, 4, 5))
Seq.of(1, 2, 3, 4, 5).splitAt(2);

The above functions all have one thing in common: They operate on a single stream in order to produce two new streams, that can be consumed independently.

Obviously, this means that internally, some memory must be consumed to keep buffers of partially consumed streams. E.g.

  • duplication needs to keep track of all values that have been consumed in one stream, but not in the other
  • partitioning needs to fast forward to the next value that satisfies (or doesn’t satisfy) the predicate, without losing all the dropped values
  • splitting might need to fast forward to the split index

For some real functional fun, let’s have a look at a possible splitAt() implementation:

static <T> Tuple2<Seq<T>, Seq<T>> 
splitAt(Stream<T> stream, long position) {
    return seq(stream)
          .partition(t -> t.v2 < position)
          .map((v1, v2) -> tuple(
     -> t.v1),
     -> t.v1)

… or with comments:

static <T> Tuple2<Seq<T>, Seq<T>> 
splitAt(Stream<T> stream, long position) {
    // Add jOOλ functionality to the stream
    // -> local Type: Seq<T>
    return seq(stream)
    // Keep track of stream positions
    // with each element in the stream
    // -> local Type: Seq<Tuple2<T, Long>>
    // Split the streams at position
    // -> local Type: Tuple2<Seq<Tuple2<T, Long>>,
    //                       Seq<Tuple2<T, Long>>>
          .partition(t -> t.v2 < position)
    // Remove the indexes from zipWithIndex again
    // -> local Type: Tuple2<Seq<T>, Seq<T>>
          .map((v1, v2) -> tuple(
     -> t.v1),
     -> t.v1)

Nice, isn’t it? A possible implementation for partition(), on the other hand, is a bit more complex. Here trivially with Iterator instead of the new Spliterator:

static <T> Tuple2<Seq<T>, Seq<T>> partition(
        Stream<T> stream, 
        Predicate<? super T> predicate
) {
    final Iterator<T> it = stream.iterator();
    final LinkedList<T> buffer1 = new LinkedList<>();
    final LinkedList<T> buffer2 = new LinkedList<>();

    class Partition implements Iterator<T> {

        final boolean b;

        Partition(boolean b) {
            this.b = b;

        void fetch() {
            while (buffer(b).isEmpty() && it.hasNext()) {
                T next =;

        LinkedList<T> buffer(boolean test) {
            return test ? buffer1 : buffer2;

        public boolean hasNext() {
            return !buffer(b).isEmpty();

        public T next() {
            return buffer(b).poll();

    return tuple(
        seq(new Partition(true)), 
        seq(new Partition(false))

I’ll let you do the exercise and verify the above code.

Get and contribute to jOOλ, now!

All of the above is part of jOOλ, available for free from GitHub. There is already a partially Java-8-ready, full-blown library called functionaljava, which goes much further than jOOλ.

Yet, we believe that all what’s missing from Java 8’s Streams API is really just a couple of methods that are very useful for sequential streams.

In a previous post, we’ve shown how we can bring lambdas to String-based SQL using a simple wrapper for JDBC (of course, we still believe that you should use jOOQ instead).

Today, we’ve shown how we can write awesome functional and sequential Stream processing very easily, with jOOλ.

Stay tuned for even more jOOλ goodness in the near future (and pull requests are very welcome, of course!)

Look no Further! The Final Answer to “Where to Put Generated Code?”

This recent question on Stack Overflow made me think.

Why does jOOQ suggest to put generated code under “/target” and not under “/src”?

… and I’m about to give you the final answer to “Where to Put Generated Code?”

This isn’t only about jOOQ

Even if you’re not using jOOQ, or if you’re using jOOQ but without the code generator, there might be some generated source code in your project. There are many tools that generate source code from other data, such as:

  • The Java compiler (ok, byte code, not strictly source code. But still code generation)
  • XJC, from XSD files
  • Hibernate from .hbm.xml files, or from your schema
  • Xtend translates Xtend code to Java code
  • You could even consider data transformations, like XSLT
  • many more…

In this article, we’re going to look at how to deal with jOOQ-generated code, but the same thoughts apply also to any other type of code generated from other code or data.

Now, the very very interesting strategic question that we need to ask ourselves is: Where to put that code? Under version control, like the original data? Or should we consider generated code to be derived code that must be re-generated all the time?

The answer is nigh…

It depends!

Nope, unfortunately, as with many other flame-wary discussions, this one doesn’t have a completely correct or wrong answer, either. There are essentially two approaches:

Considering generated code as part of your code base

When you consider generated code as part of your code base, you will want to:

  • Check in generated sources in your version control system
  • Use manual source code generation
  • Possibly use even partial source code generation

This approach is particularly useful when your Java developers are not in full control of or do not have full access to your database schema (or your XSD or your Java code, etc.), or if you have many developers that work simultaneously on the same database schema, which changes all the time. It is also useful to be able to track side-effects of database changes, as your checked-in database schema can be considered when you want to analyse the history of your schema.

With this approach, you can also keep track of the change of behaviour in the jOOQ code generator, e.g. when upgrading jOOQ, or when modifying the code generation configuration.

When you use this approach, you will treat your generated code as an external library with its own lifecycle.

The drawback of this approach is that it is more error-prone and possibly a bit more work as the actual schema may go out of sync with the generated schema.

Considering generated code as derived artefacts

When you consider generated code to be derived artefacts, you will want to:

  • Check in only the actual DDL, i.e. the “original source of truth” (e.g. controlled via Flyway)
  • Regenerate jOOQ code every time the schema changes
  • Regenerate jOOQ code on every machine – including continuous integration machines, and possibly, if you’re crazy enough, on production

This approach is particularly useful when you have a smaller database schema that is under full control by your Java developers, who want to profit from the increased quality of being able to regenerate all derived artefacts in every step of your build.

This approach is fully supported by Maven, for instance, which foresees special directories (e.g. target/generated-sources), and phases (e.g. <phase>generate-sources</phase>) specifically for source code generation.

The drawback of this approach is that the build may break in perfectly “acceptable” situations, when parts of your database are temporarily unavailable.

Pragmatic approach

Some of you might not like that answer, but there is also a pragmatic approach, a combination of both. You can consider some code as part of your code base, and some code as derived. For instance, jOOQ-meta’s generated sources (used to query the dictionary views / INFORMATION_SCHEMA when generating jOOQ code) are put under version control as few jOOQ contributors will be able to run the jOOQ-meta code generator against all supported databases. But in many integration tests, we re-generate the sources every time to be sure the code generator works correctly.



I’m sorry to disappoint you. There is no final answer to whether one approach or the other is better. Pick the one that offers you more value in your specific situation.

In case you’re choosing your generated code to be part of the code base, read this interesting experience report on the jOOQ User Group by Witold Szczerba about how to best achieve this.