Java 8 Friday: More Functional Relational Transformation

In the past, we’ve been providing you with a new article every Friday about what’s new in Java 8. It has been a very exciting blog series, but we would like to focus again more on our core content, which is Java and SQL. We will still be occasionally blogging about Java 8, but no longer every Friday (as some of you have already notice).

In this last, short post of the Java 8 Friday series, we’d like to re-iterate the fact that we believe that the future belongs to functional relational data transformation (as opposed to ORM). We’ve spent about 20 years now using the object-oriented software development paradigm. Many of us have been very dogmatic about it. In the last 10 years, however, a “new” paradigm has started to get increasing traction in programming communities: Functional programming.

Functional programming is not that new, however. Lisp has been a very early functional programming language. XSLT and SQL are also somewhat functional (and declarative!). As we’re big fans of SQL’s functional (and declarative!) nature, we’re quite excited about the fact that we now have sophisticated tools in Java to transform tabular data that has been extracted from SQL databases. Streams!

SQL ResultSets are very similar to Streams

As we’ve pointed out before, JDBC ResultSets and Java 8 Streams are quite similar. This is even more true when you’re using jOOQ, which replaces the JDBC ResultSet by an org.jooq.Result, which extends java.util.List, and thus automatically inherits all Streams functionality. Consider the following query that allows fetching a one-to-many relationship between BOOK and AUTHOR records:

Map<Record2<String, String>, 
    List<Record2<Integer, String>>> booksByAuthor =

// This work is performed in the database
// --------------------------------------
ctx.select(
        BOOK.ID,
        BOOK.TITLE,
        AUTHOR.FIRST_NAME,
        AUTHOR.LAST_NAME
    )
   .from(BOOK)
   .join(AUTHOR)
   .on(BOOK.AUTHOR_ID.eq(AUTHOR.ID))
   .orderBy(BOOK.ID)
   .fetch()

// This work is performed in Java memory
// -------------------------------------
   .stream()

   // Group BOOKs by AUTHOR
   .collect(groupingBy(

        // This is the grouping key      
        r -> r.into(AUTHOR.FIRST_NAME, 
                    AUTHOR.LAST_NAME),

        // This is the target data structure
        LinkedHashMap::new,

        // This is the value to be produced for each
        // group: A list of BOOK
        mapping(
            r -> r.into(BOOK.ID, BOOK.TITLE),
            toList()
        )
    ));

The fluency of the Java 8 Streams API is very idiomatic to someone who has been used to writing SQL with jOOQ. Obviously, you can also use something other than jOOQ, e.g. Spring’s JdbcTemplate, or Apache Commons DbUtils, or just wrap the JDBC ResultSet in an Iterator…

What’s very nice about this approach, compared to ORM is the fact that there is no magic happening at all. Every piece of mapping logic is explicit and, thanks to Java generics, fully typesafe. The type of the booksByAuthor output is complex, and a bit hard to read / write, in this example, but it is also fully descriptive and useful.

The same functional transformation with POJOs

If you aren’t too happy with using jOOQ’s Record2 tuple types, no problem. You can specify your own data transfer objects like so:

class Book {
    public int id;
    public String title;

    @Override
    public String toString() { ... }

    @Override
    public int hashCode() { ... }

    @Override
    public boolean equals(Object obj) { ... }
}

static class Author {
    public String firstName;
    public String lastName;

    @Override
    public String toString() { ... }

    @Override
    public int hashCode() { ... }

    @Override
    public boolean equals(Object obj) { ... }
}

With the above DTO, you can now leverage jOOQ’s built-in POJO mapping to transform the jOOQ records into your own domain classes:

Map<Author, List<Book>> booksByAuthor =
ctx.select(
        BOOK.ID,
        BOOK.TITLE,
        AUTHOR.FIRST_NAME,
        AUTHOR.LAST_NAME
    )
   .from(BOOK)
   .join(AUTHOR)
   .on(BOOK.AUTHOR_ID.eq(AUTHOR.ID))
   .orderBy(BOOK.ID)
   .fetch()
   .stream()
   .collect(groupingBy(

        // This is the grouping key      
        r -> r.into(Author.class),
        LinkedHashMap::new,

        // This is the grouping value list
        mapping(
            r -> r.into(Book.class),
            toList()
        )
    ));

Explicitness vs. implicitness

At Data Geekery, we believe that a new time has started for Java developers. A time where Annotatiomania™ (finally!) ends and people stop assuming all that implicit behaviour through annotation magic. ORMs depend on a huge amount of specification to explain how each annotation works with each other annotation. It is hard to reverse-engineer (or debug!) this kind of not-so-well-understood annotation-language that JPA has brought to us.

On the flip side, SQL is pretty well understood. Tables are an easy-to-handle data structure, and if you need to transform those tables into something more object-oriented, or more hierarchically structured, you can simply apply functions to those tables and group values yourself! By grouping those values explicitly, you stay in full control of your mapping, just as with jOOQ, you stay in full control of your SQL.

This is why we believe that in the next 5 years, ORMs will lose relevance and people start embracing explicit, stateless and magicless data transformation techniques again, using Java 8 Streams.

Java 8 Friday: The Best Java 8 Resources – Your Weekend is Booked

At Data Geekery, we love Java. And as we’re really into jOOQ’s fluent API and query DSL, we’re absolutely thrilled about what Java 8 will bring to our ecosystem.

Every Friday, we’re showing you a couple of nice new tutorial-style Java 8 features, which take advantage of lambda expressions, method references, default methods, the Streams API, and other great stuff. You’ll find the source code on GitHub.

The Best Java 8 Resources – Your Weekend is Booked

We’re obviously not the only ones writing about Java 8. Ever since this great language update’s go live, there had been blogs all around the world appearing with great content and different perspectives on the subject. In this edition of the Java 8 Friday series, we’d like to summarise some of the best content that has been going on on that subject.

1. Brian Goetz’s Answers on Stack Overflow

Brian Goetz was the spec lead for JSR 335. Together with his Expert Group team, he has worked very hard to help Java 8 succeed. However, now that JSR 335 has shipped, his work is far from being over. Brian has had the courtesy of giving authoritative answers to questions from the Java community on Stack Overflow. Here are some of the most interesting questions:

Thumbs up to this great community effort. It cannot get any better than hearing authoritative answers from the spec lead himself.

2. Baeldung.com’s Collection of Java 8 Resources

This list of resources wouldn’t be complete without the very useful list of Java 8 resources (mostly authoritative links to specifications) from the guys over at Baeldung.com. Here is:

http://www.baeldung.com/java8

3. The jOOQ Blog’s Java 8 Friday Series

Yay, that’s us! :-)

Yes, we’ve worked hard to bring you the latest from our experience when integrating jOOQ with Java 8. Here are some of our most popular articles from the recent months:

4. ZeroTurnaround’s RebelLabs Blog

As part of the ZeroTurnaround content marketing strategy, ZeroTurnaround has launched RebelLabs quite a while ago where various writers publish interesting articles around the topic of Java, which aren’t necessarily related to JRebel and other ZT products. There is some great Java 8 related content having been published over there. Here are our favourite gems:

5. The Takipi Blog

Just like ZeroTurnaround and ourselves, our friends over at Takipi provide you with some awesome Java 8 content on their blog.

6. Benji Weber’s Fun Experiments with Java 8

This blog series we found particularly fun to read. Benji Weber really thinks outside of the box and does some crazy things with default methods, method references and all that. Things that Java developers could only dream of, so far. Here are:

7. The Geeks from Paradise Blog’s Java 8 Musings

Edwin Dalorzo from Informatech has been treating us with a variety of well-founded comparisons between Java 8 and .NET. This is particularly interesting when comparing Streams with LINQ. Here are some of his best writings:

Is this list complete?

No, it is missing many other, very interesting blog series. Do you have a series to share? We’re more than happy to update this post, just let us know (in the comments section)

Java 8 Friday: Most Internal DSLs are Outdated

At Data Geekery, we love Java. And as we’re really into jOOQ’s fluent API and query DSL, we’re absolutely thrilled about what Java 8 will bring to our ecosystem.

Java 8 Friday

Every Friday, we’re showing you a couple of nice new tutorial-style Java 8 features, which take advantage of lambda expressions, extension methods, and other great stuff. You’ll find the source code on GitHub.

Most Internal DSLs are Outdated

That’s quite a statement from a vendor of one of the most advanced internal DSLs currently on the market. Let me explain:

Languages are hard

Learning a new language (or API) is hard. You have to understand all the keywords, the constructs, the statement and expression types, etc. This is true both for external DSLs, internal DSLs and “regular” APIs, which are essentially internal DSLs with less fluency.

When using JUnit, people have grown used to using hamcrest matchers. The fact that they’re available in six languages (Java, Python, Ruby, Objective-C, PHP, Erlang) makes them somewhat of a sound choice. As a domain-specific language, they have established idioms that are easy to read, e.g.

assertThat(theBiscuit, equalTo(myBiscuit));
assertThat(theBiscuit, is(equalTo(myBiscuit)));
assertThat(theBiscuit, is(myBiscuit));

When you read this code, you will immediately “understand” what is being asserted, because the API reads like prosa. But learning to write code in this API is harder. You will have to understand:

  • Where all of these methods are coming from
  • What sorts of methods exist
  • Who might have extended hamcrest with custom Matchers
  • What are best practices when extending the DSL

For instance, in the above example, what exactly is the difference between the three? When should I use one and when the other? Is is() checking for object identity? Is equalTo() checking for object equality?

The hamcrest tutorial goes on with examples like these:

public void testSquareRootOfMinusOneIsNotANumber() {
    assertThat(Math.sqrt(-1), is(notANumber()));
}

You can see that notANumber() apparently is a custom matcher, implemented some place in a utility:

public class IsNotANumber
extends TypeSafeMatcher<Double> {

  @Override
  public boolean matchesSafely(Double number) {
    return number.isNaN();
  }

  public void describeTo(Description description) {
    description.appendText("not a number");
  }

  @Factory
  public static <T> Matcher<Double> notANumber() {
    return new IsNotANumber();
  }
}

While this sort of DSL is very easy to create, and probably also a bit fun, it is dangerous to start delving into writing and enhancing custom DSLs for a simple reason. They’re in no way better than their general-purpose, functional counterparts – but they’re harder to maintain. Consider the above examples in Java 8:

Replacing DSLs with Functions

Let’s assume we have a very simple testing API:

static <T> void assertThat(
    T actual, 
    Predicate<T> expected
) {
    assertThat(actual, expected, "Test failed");
}

static <T> void assertThat(
    T actual, 
    Predicate<T> expected, 
    String message
) {
    assertThat(() -> actual, expected, message);
}

static <T> void assertThat(
    Supplier<T> actual, 
    Predicate<T> expected
) {
    assertThat(actual, expected, "Test failed");
}

static <T> void assertThat(
    Supplier<T> actual, 
    Predicate<T> expected, 
    String message
) {
    if (!expected.test(actual.get()))
        throw new AssertionError(message);
}

Now, compare the hamcrest matcher expressions with their functional equivalents:

// BEFORE
// ---------------------------------------------
assertThat(theBiscuit, equalTo(myBiscuit));
assertThat(theBiscuit, is(equalTo(myBiscuit)));
assertThat(theBiscuit, is(myBiscuit));

assertThat(Math.sqrt(-1), is(notANumber()));

// AFTER
// ---------------------------------------------
assertThat(theBiscuit, b -> b == myBiscuit);
assertThat(Math.sqrt(-1), n -> Double.isNaN(n));

With lambda expressions, and a well-designed assertThat() API, I’m pretty sure that you won’t be looking for the right way to express your assertions with matchers any longer.

Note that unfortunately, we cannot use the Double::isNaN method reference, as that would not be compatible with Predicate<Double>. For that, we’d have to do some primitive type magic in the assertion API, e.g.

static void assertThat(
    double actual, 
    DoublePredicate expected
) { ... }

Which can then be used as such:

assertThat(Math.sqrt(-1), Double::isNaN);

Yeah, but…

… you may hear yourself saying, “but we can combine matchers with lambdas and streams”. Yes, of course we can. I’ve just done so now in the jOOQ integration tests. I want to skip the integration tests for all SQL dialects that are not in a list of dialects supplied as a system property:

String dialectString = 
    System.getProperty("org.jooq.test-dialects");

// The string must not be "empty"
assumeThat(dialectString, not(isOneOf("", null)));

// And we check if the current dialect() is
// contained in a comma or semi-colon separated
// list of allowed dialects, trimmed and lowercased
assumeThat(
    dialect().name().toLowerCase(),

    // Another matcher here
    isOneOf(stream(dialectString.split("[,;]"))
        .map(String::trim)
        .map(String::toLowerCase)
        .toArray(String[]::new))
);

… and that’s pretty neat, too, right?

But why don’t I just simply write:

// Using Apache Commons, here
assumeThat(dialectString, StringUtils::isNotEmpty);
assumeThat(
    dialect().name().toLowerCase(),
    d -> stream(dialectString.split("[,;]"))
        .map(String::trim)
        .map(String::toLowerCase())
        .anyMatch(d::equals)
);

No Hamcrest needed, just plain old lambdas and streams!

Now, readability is a matter of taste, of course. But the above example clearly shows that there is no longer any need for Hamcrest matchers and for the Hamcrest DSL. Given that within the next 2-3 years, the majority of all Java developers will be very used to using the Streams API in every day work, but not very used to using the Hamcrest API, I urge you, JUnit maintainers, to deprecate the use of Hamcrest in favour of Java 8 APIs.

Is Hamcrest now considered bad?

Well, it has served its purpose in the past, and people have grown somewhat used to it. But as we’ve already pointed out in a previous post about Java 8 and JUnit Exception matching, yes, we do believe that we Java folks have been barking up the wrong tree in the last 10 years.

The lack of lambda expressions has lead to a variety of completely bloated and now also slightly useless libraries. Many internal DSLs or annotation-magicians are also affected. Not because they’re no longer solving the problems they used to, but because they’re not Java-8-ready. Hamcrest’s Matcher type is not a functional interface, although it would be quite easy to transform it into one. In fact, Hamcrest’s CustomMatcher logic should be pulled up to the Matcher interface, into default methods.

Things dont’ get better with alternatives, like AssertJ, which create an alternative DSL that is now rendered obsolete (in terms of call-site code verbosity) through lambdas and the Streams API.

If you insist on using a DSL for testing, then probably Spock would be a far better choice anyway.

Other examples

Hamcrest is just one example of such a DSL. This article has shown how it can be almost completely removed from your stack by using standard JDK 8 constructs and a couple of utility methods, which you might have in JUnit some time soon, anyway.

Java 8 will bring a lot of new traction into last decade’s DSL debate, as also the Streams API will greatly improve the way we look at transforming or building data. But many current DSLs are not ready for Java 8, and have not been designed in a functional way. They have too many keywords for things and concepts that are hard to learn, and that would be better modelled using functions.

An exception to this rule are DSLs like jOOQ or jRTF, which are modelling actual pre-existing external DSLs in a 1:1 fashion, inheriting all the existing keywords and syntax elements, which makes them much easier to learn in the first place.

What’s your take?

What is your take on the above assumptions? What is your favourite internal DSL, that might vanish or that might be completely transformed in the next five years because it has been obsoleted by Java 8?

Stay tuned for more Java 8 Friday articles here on this blog.

Java 8 Friday: Language Design is Subtle

At Data Geekery, we love Java. And as we’re really into jOOQ’s fluent API and query DSL, we’re absolutely thrilled about what Java 8 will bring to our ecosystem.

Java 8 Friday

Every Friday, we’re showing you a couple of nice new tutorial-style Java 8 features, which take advantage of lambda expressions, extension methods, and other great stuff. You’ll find the source code on GitHub.

Language Design is Subtle

It’s been a busy week for us. We have just migrated the jOOQ integration tests to Java 8 for two reasons:

  • We want to be sure that client code compiles with Java 8
  • We started to get bored of writing the same old loops over and over again

The trigger was a loop where we needed to transform a SQLDialect[] into another SQLDialect[] calling .family() on each array element. Consider:

Java 7

SQLDialect[] families = 
    new SQLDialect[dialects.length];
for (int i = 0; i < families.length; i++)
    families[i] = dialects[i].family();

Java 8

SQLDialect[] families = 
Stream.of(dialects)
      .map(d -> d.family())
      .toArray(SQLDialect[]::new);

OK, it turns out that the two solutions are equally verbose, even if the latter feels a bit more elegant. :-)

And this gets us straight into the next topic:

Backwards-compatibility

For backwards-compatibility reasons, arrays and the pre-existing Collections API have not been retrofitted to accommodate all the useful methods that Streams now have. In other words, an array doesn’t have a map() method, just as much as List doesn’t have such a method. Streams and Collections/arrays are orthogonal worlds. We can transform them into each other, but they don’t have a unified API.

This is fine in everyday work. We’ll get used to the Streams API and we’ll love it, no doubt. But because of Java being extremely serious about backwards compatibility, we will have to think about one or two things more deeply.

Recently, we have published a post about The Dark Side of Java 8. It was a bit of a rant, although a mild one in our opinion (and it was about time to place some criticism, after all the praise we’ve been giving Java 8 in our series, before ;-) ). First off, that post triggered a reaction by Edwin Dalorzo from our friends at Informatech. (Edwin has written this awesome post comparing LINQ and Java 8 Streams, before). The criticism in our article evolved around three main aspects:

  • Overloading getting more complicated (see also this compiler bug)
  • Limited support for method modifiers on default methods
  • Primitive type “API overloads” for streams and functional interfaces

A response by Brian Goetz

I then got a personal mail from no one less than Brian Goetz himself (!), who pointed out a couple of things to me that I had not yet thought about in this way:

I still think you’re focusing on the wrong thing. Its not really the syntax you don’t like; its the model — you don’t want “default methods”, you want traits, and the syntax is merely a reminder that you didn’t get the feature you wanted. (But you’d be even more confused about “why can’t they be final” if we dropped the “default” keyword!) But that’s blaming the messenger (where here, the keyword is the messenger.)

Its fair to say “this isn’t the model I wanted”. There were many possible paths in the forest, and it may well be the road not taken was equally good or better.

This is also what Edwin had concluded. Default methods were a necessary means to tackle all the new API needed to make Java 8 useful. If Iterator, Iterable, List, Collection, and all the other pre-existing interfaces had to be adapted to accommodate lambdas and Streams API interaction, the expert group would have needed to break an incredible amount of API. Conversely, without adding these additional utility methods (see the awesome new Map methods, for instance!), Java 8 would have been only half as good.

And that’s it.

Even if maybe, some more class building tools might have been useful, they were not in the center of focus for the expert group who already had a lot to do to get things right. The center of focus was to provide a means for API evolution. Or in Brian Goetz’s own words:

Reaching out to the community

It’s great that Brian Goetz reaches out to the community to help us get the right picture about Java 8. Instead of explaining rationales about expert group decisions in private messages, he then asked me to publicly re-ask my questions again on Stack Overflow (or lambda-dev), such that he can then publicly answer them. For increased publicity and greater community benefit, I chose Stack Overflow. Here are:

The amount of traction these two questions got in no time shows how important these things are to the community, so don’t miss reading through them!

“Uncool”? Maybe. But very stable!

Java may not have the “cool” aura that node.js has. You may think about JavaScript-the-language whatever you want (as long as it contains swear words), but from a platform marketing perspective, Java is being challenged for the first time in a long time – and being “uncool” and backwards-compatible doesn’t help keeping developers interested.

But let’s think long-term, instead of going with trends. Having such a great professional platform like the Java language, the JVM, the JDK, JEE, and much more, is invaluable. Because at the end of the day, the “uncool” backwards-compatibility can also be awesome. As mentioned initially, we have upgraded our integration tests to Java 8. Not a single compilation error, not a single bug. Using Eclipse’s BETA support for Java 8, I could easily transform anonymous classes into lambdas and write awesome things like these upcoming jOOQ 3.4 nested transactions (API not final yet):

ctx.transaction(c1 -> {
    DSL.using(c1)
       .insertInto(AUTHOR, AUTHOR.ID, AUTHOR.LAST_NAME)
       .values(3, "Doe")
       .execute();

    // Implicit savepoint here
    try {
        DSL.using(c1).transaction(c2 -> {
            DSL.using(c2)
               .update(AUTHOR)
               .set(AUTHOR.FIRST_NAME, "John")
               .where(AUTHOR.ID.eq(3))
               .execute();

            // Rollback to savepoint
            throw new MyRuntimeException("No");
        });
    }

    catch (MyRuntimeException ignore) {}

    return 42;
});

So at the end of the day, Java is great. Java 8 is a tremendous improvement over previous versions, and with great people in the expert groups (and reaching out to the community on social media), I trust that Java 9 will be even better. In particular, I’m looking forward to learning about how these two projects evolve:

Although, again, I am really curious how they will pull these two improvements off from a backwards-compatibility perspective, and what caveats we’ll have to understand, afterwards. ;-)

Anyway, let’s hope the expert groups will continue to provide public feedback on Stack Overflow. Stay tuned for more awesome Java 8 content on this blog.

Java 8 Friday: Java 8 Will Revolutionize Database Access

At Data Geekery, we love Java. And as we’re really into jOOQ’s fluent API and query DSL, we’re absolutely thrilled about what Java 8 will bring to our ecosystem. For our Java 8 series, we’re honoured to host a very relevant guest post by Dr. Ming-Yee Iu.

Dr. Ming-Yee Iu completed a PhD on Database Queries in Java at EPFL. He has created the open source project Jinq to demonstrate some new techniques for supporting database queries in Java.

Our editorial note:

Ever since Erik Meijer has introduced LINQ to the .NET ecosystem, us Java folks have been wondering whether we could have the same. We’ve blogged about this topic before, a couple of times:

While most LINQesque APIs in the Java ecosystem operate as internal domain-specific languages like jOOQ, some try to tackle the integration on a bytecode level, like JaQu.

JINQ formalises runtime bytecode transformations through what Dr. Ming-Yee Iu calls symbolic execution. We find this very interesting to a point that we wonder if we should start building a JINQ-to-jOOQ JINQ provider, where the expressive power of the Java 8 Streams API could be combined with our great SQL standardisation and transformation features…?

Convince yourselves:

Java 8 Goodie: Java 8 Will Revolutionize Database Access

Java 8 is finally here! After years of waiting, Java programmers will finally get support for functional programming in Java. Functional programming support helps streamline existing code while providing powerful new capabilities to the Java language. One area that will be disrupted by these new features is how programmers work with databases in Java. Functional programming support opens up exciting new possibilities for simpler yet more powerful database APIs. Java 8 will enable new ways to access databases that are competitive with those of other programming languages such as C#’s LINQ.

The Functional Way of Working With Data

Java 8 not only adds functional-support to the Java language, but it extends the Java collection classes with new functional ways of working with data. Traditionally, working with large amounts of data in Java requires a lot of loops and iterators.

For example, suppose you have a collection of Customer objects:

Collection<Customer> customers;

If you were only interested in the customers from Belgium, you would have to iterate over all the customers and save the ones you wanted.

Collection<Customer> belgians = new ArrayList<>();
for (Customer c : customers) {
    if (c.getCountry().equals("Belgium"))
        belgians.add(c);
}

This takes five lines of code. It is also poorly abstracted. What happens if you have 10 million customers, and you want to speed up the code by filtering it in parallel using two threads? You would have to rewrite everything to use futures and a lot of hairy multi-threaded code.

With Java 8, you can write the same code in one line. With its support for functional programming, Java 8 lets you write a function saying which customers you are interested in (those from Belgium) and then to filter collections using that function. Java 8 has a new Streams API that lets you do this.

customers.stream().filter(
    c -> c.getCountry().equals("Belgium")
);

Not only is the Java 8 version of the code shorter, but the code is easier to understand as well. There is almost no boilerplate. The code calls the method filter(), so it’s clear that this code is used for filtering customers. You don’t have to spend your time trying to decipher the code in a loop to understand what it is doing with its data.

And what happens if you want to run the code in parallel? You just have to use a different type of stream.

customers.parallelStream().filter(
    c -> c.getCountry().equals("Belgium")
);

What’s even more exciting is that this functional-style of code works with databases as well!

The Functional Way of Working with Databases

Traditionally, programmers have needed to use special database query languages to access the data in databases. For example, below is some JDBC code for finding all the customers from Belgium:

PreparedStatement s = con.prepareStatement(
      "SELECT * "
    + "FROM Customer C "
    + "WHERE C.Country = ? ");
s.setString(1, "Belgium");
ResultSet rs = s.executeQuery();

Much of the code is in the form a string, which the compiler can’t check for errors and which can lead to security problems due to sloppy coding. There is also a lot of boilerplate code that makes writing database access code quite tedious. Tools such as jOOQ solve the problem of error-checking and security by providing a database query language that can be written using special Java libraries. Or you can use tools such as object-relational mappers to hide a lot of boring database code for common access patterns, but if you need to write non-trivial database queries, you will still need to use a special database query language again.

With Java 8, it’s possible to write database queries using the same functional-style used when working with the Streams API. For example, Jinq is an open source project that explores how future database APIs can make use of functional programming. Here is a database query written using Jinq:

customers.where(
    c -> c.getCountry().equals("Belgium")
);

This code is almost identical to the code using the Streams API. In fact, future versions of Jinq will let you write queries directly using the Streams API. When the code is run, Jinq will automatically translate the code into a database query like the JDBC query shown before.

So without having to learn a new database query language, you can write efficient database queries. You can use the same style of code you would use for Java collections. You also don’t need a special Java compiler or virtual machine. All of this code compiles and runs using the normal Java 8 JDK. If there are errors in your code, the compiler will find them and report them to you, just like normal Java code.

Jinq supports queries that can be as complicated as SQL92. Selection, projection, joins, and subqueries are all supported. The algorithm for translating Java code into database queries is also very flexible in what code it will accept and translate. For example, Jinq has no problem translating the code below into a database query, despite its complexity.

customers
    .where( c -> c.getCountry().equals("Belgium") )
    .where( c -> {
        if (c.getSalary() < 100000)
            return c.getSalary() < c.getDebt();
        else
            return c.getSalary() < 2 * c.getDebt();
        } );

As you can see, the functional programming support in Java 8 is well-suited for writing database queries. The queries are compact, and complex queries are supported.

Inner Workings

But how does this all work? How can a normal Java compiler translate Java code into database queries? Is there something special about Java 8 that makes this possible?

The key to supporting these new functional-style database APIs is a type of bytecode analysis called symbolic execution. Although your code is compiled by a normal Java compiler and run in a normal Java virtual machine, Jinq is able to analyze your compiled Java code when it is run and construct database queries from them. Symbolic execution works best when analyzing small functions, which are common when using the Java 8 Streams API.

The easiest way to understand how this symbolic execution works is with an example. Let’s examine how the following query is converted by Jinq into the SQL query language:

customers
    .where( c -> c.getCountry().equals("Belgium") )

Initially, the customers variable is a collection that represents this database query

SELECT *
  FROM Customers C

Then, the where() method is called, and a function is passed to it. In this where() method, Jinq opens the .class file of the function and gets the compiled bytecode for the function to analyze. In this example, instead of using real bytecode, let’s just use some simple instructions to represent the bytecode of the function:

  1. d = c.getCountry()
  2. e = “Belgium”
  3. e = d.equals(e)
  4. return e

Here, we pretend that the function has been compiled by the Java compiler into four instructions. This is what Jinq sees when the where() method is called. How can Jinq make sense of this code?

Jinq analyzes the code by executing it. Jinq doesn’t run the code directly though. It runs the code ‘abstractly’. Instead of using real variables and real values, Jinq uses symbols to represent all values when executing the code. This is why the analysis is called symbolic execution.

Jinq executes each instruction and keeps track of all the side-effects or all the things that the code changes in the state of the program. Below is a diagram showing all the side-effects that Jinq finds when it executes the four lines of code using symbolic execution.

Symbolic execution example

Symbolic execution example

In the diagram, you can see how after the first instruction runs, Jinq finds two side-effects: the variable d has changed and the method Customer.getCountry() has been called. With symbolic execution, the variable d is not given a real value like “USA” or “Denmark”. It is assigned the symbolic value of c.getCountry().

After all the instructions have been executed symbolically, Jinq prunes the side-effects. Since the variables d and e are local variables, any changes to them are discarded after the function exits, so those side-effects can be ignored. Jinq also knows that the methods Customer.getCountry() and String.equals() do not modify any variables or show any output, so those method calls can also be ignored. From this, Jinq can conclude that executing the function produces only one effect: it returns c.getCountry().equals("Belgium").

Once Jinq has understood what the function passed to it in the where() method does, it can then merge this knowledge with the database query underlying the customers collection to create a new database query.

Generating a database query

Generating a database query

And that’s how Jinq generates database queries from your code. The use of symbolic execution means that this approach is quite robust to the different code patterns outputted by different Java compilers. If Jinq ever encounters code with side-effects that can’t be emulated using a database query, Jinq will leave your code untouched. Since everything is written using normal Java code, Jinq can just run that code directly instead, and your code will produce the expected results.

This simple translation example should have given you an idea of how the query translation works. You should feel confident that these algorithms can correctly generate database queries from your code.

An Exciting Future

I hope I have given you a taste for how Java 8 enables new ways of working with databases in Java. The functional programming support in Java 8 allows you write database code in a similar way to writing code for working with Java collections. Hopefully, existing database APIs will soon be extended to support these styles of queries.

To play with a prototype for these new types of queries, you can visit http://www.jinq.org

Java 8 Friday Goodies: Lean Concurrency

At Data Geekery, we love Java. And as we’re really into jOOQ’s fluent API and query DSL, we’re absolutely thrilled about what Java 8 will bring to our ecosystem. We have blogged a couple of times about some nice Java 8 goodies, and now we feel it’s time to start a new blog series, the…

Java 8 Friday

Every Friday, we’re showing you a couple of nice new tutorial-style Java 8 features, which take advantage of lambda expressions, extension methods, and other great stuff. You’ll find the source code on GitHub.

Java 8 Goodie: Lean Concurrency

Someone once said that (unfortunately, we don’t have the source anymore):

Junior programmers think concurrency is hard.
Experienced programmers think concurrency is easy.
tweet thisSenior programmers think concurrency is hard.

That is quite true. But on the bright side, Java 8 will at least improve things by making it easier to write concurrent code with lambdas and the many improved APIs. Let’s have a closer look:

Java 8 improving on JDK 1.0 API

java.lang.Thread has been around from the very beginning in JDK 1.0. So has java.lang.Runnable, which is going to be annotated with FunctionalInterface in Java 8.

It is almost a no-brainer how we can finally submit Runnables to a Thread from now on. Let’s assume we have a long-running operation:

public static int longOperation() {
    System.out.println("Running on thread #"
       + Thread.currentThread().getId());

    // [...]
    return 42;
}

We can then pass this operation to Threads in various ways, e.g.

Thread[] threads = {

    // Pass a lambda to a thread
    new Thread(() -> {
        longOperation();
    }),

    // Pass a method reference to a thread
    new Thread(ThreadGoodies::longOperation)
};

// Start all threads
Arrays.stream(threads).forEach(Thread::start);

// Join all threads
Arrays.stream(threads).forEach(t -> {
    try { t.join(); }
    catch (InterruptedException ignore) {}
});

As we’ve mentioned in our previous blog post, it’s a shame that lambda expressions did not find a lean way to work around checked exceptions. None of the newly added functional interfaces in the java.util.function package allow for throwing checked exceptions, leaving the work up to the call-site.

jool-logo-blackIn our last post, we’ve thus published jOOλ (also jOOL, jOO-Lambda), which wraps each one of the JDK’s functional interfaces in an equivalent functional interface that allows for throwing checked exceptions. This is particularly useful with old JDK APIs, such as JDBC, or the above Thread API. With jOOλ, we can then write:

// Join all threads
Arrays.stream(threads).forEach(Unchecked.consumer(
    t -> t.join()
));

Java 8 improving on Java 5 API

Java’s multi-threading APIs had been pretty dormant up until the release of Java 5’s awesome ExecutorService. Managing threads had been a burden, and people needed external libraries or a J2EE / JEE container to manage thread pools. This has gotten a lot easier with Java 5. We can now submit a Runnable or a Callable to an ExecutorService, which manages its own thread-pool.

Here’s an example how we can leverage these Java 5 concurrency APIs in Java 8:

ExecutorService service = Executors
    .newFixedThreadPool(5);

Future[] answers = {
    service.submit(() -> longOperation()),
    service.submit(ThreadGoodies::longOperation)
};

Arrays.stream(answers).forEach(Unchecked.consumer(
    f -> System.out.println(f.get())
));

Note, how we again use an UncheckedConsumer from jOOλ to wrap the checked exception thrown from the get() call in a RuntimeException.

Parallelism and ForkJoinPool in Java 8

Now, the Java 8 Streams API changes a lot of things in terms of concurrency and parallelism. In Java 8, you can write the following, for instance:

Arrays.stream(new int[]{ 1, 2, 3, 4, 5, 6 })
      .parallel()
      .max()
      .ifPresent(System.out::println);

While it isn’t necessary in this particular case, it’s still interesting to see that the mere calling of parallel() will run the IntStream.max() reduce operation on all available threads of your JDK’s internal ForkJoinPool without you having to worry about the involved ForkJoinTasks. This can be really useful, as not everybody welcomed the JDK 7 ForkJoin API the complexity it has introduced.

Read more about Java 8’s parallel streams in this interesting InfoQ article.

More on Java 8

Parallelism was one of the main driving forces behind the new Streams API. Being able to just set a flag called parallel() on a Stream is marvellous in many situations.

In the last example, we’ve seen the OptionalInt.ifPresent() method that takes an IntConsumer argument to be executed if the previous reduce operation succeeded.

Other languages such as Scala have known an “Option” type to improve NULL handling. We’ve blogged about Optional before, and we’ll reiterate the Java 8 Optional type in the context of Java 8 Streams, so stay tuned!

In the mean time, have a look at Eugen Paraschiv’s awesome Java 8 resources page

Java 8 Friday Goodies: Map Enhancements

At Data Geekery, we love Java. And as we’re really into jOOQ’s fluent API and query DSL, we’re absolutely thrilled about what Java 8 will bring to our ecosystem. We have blogged a couple of times about some nice Java 8 goodies, and now we feel it’s time to start a new blog series, the…

Java 8 Friday

Every Friday, we’re showing you a couple of nice new tutorial-style Java 8 features, which take advantage of lambda expressions, extension methods, and other great stuff. You’ll find the source code on GitHub. tweet this

Java 8 Goodie: Map Enhancements

In previous posts, we’ve already dealt with a couple of new Streams features, for instance when sorting. Most API improvements are indeed part of the new Streams API. But a few nice methods were also added to java.util.List and most importantly, to java.util.Map. If you want a quick overview of feature additions, go to the JDK8 Javadoc and click on the new “Default Methods” tab:

java.util.Map default methods

java.util.Map default methods

For backwards-compatibility reasons, all new methods added to Java interfaces are in fact default methods. So we have a couple of exciting new additions!

compute() methods

Often, we fetch a value from a map, make some calculations on it and put it back into the map. This can be verbose and hard to get right if concurrency is involved. With Java 8, we can pass a BiFunction to the new compute(), computeIfAbsent(), or computeIfPresent() methods and have the Map implementation handle the semantics of replacing a value.

The following example shows how this works:

// We'll be using this simple map
// Unfortunately, still no map literals in Java 8..
Map<String, Integer> map = new HashMap<>();
map.put("A", 1);
map.put("B", 2);
map.put("C", 3);

// Compute a new value for the existing key
System.out.println(map.compute("A", 
    (k, v) -> v == null ? 42 : v + 41));
System.out.println(map);

// This will add a new (key, value) pair
System.out.println(map.compute("X", 
    (k, v) -> v == null ? 42 : v + 41));
System.out.println(map);

The output of the above program is this:

42
{A=42, B=2, C=3}
42
{A=42, B=2, C=3, X=42}

This is really useful for ConcurrentHashMap, which ships with the following guarantee:

The entire method invocation is performed atomically. Some attempted update operations on this map by other threads may be blocked while computation is in progress, so the computation should be short and simple, and must not attempt to update any other mappings of this Map.

forEach() method

This is a really nice goodie which lets you pass a method reference or a lambda to receive (key, value) pairs one by one. A trivial example would be this:

map.forEach((k, v) -> 
    System.out.println(k + "=" + v));

Its output being:

A=1
B=2
C=3

merge() method

Now this one is really not so easy to understand. The Javadoc uses this example here:

map.merge(key, msg, String::concat)

Given the following contract:

If the specified key is not already associated with a value or is associated with null, associates it with the given value. Otherwise, replaces the value with the results of the given remapping function, or removes if the result is null.

So, the above code translates to the following atomic operation:

String value = map.get(key);
if (value == null)
    map.put(key, msg);
else
    map.put(key, value.concat(msg));

This is certainly not an everyday functionality and might just have leaked from an implementation to the top-level API. Additionally, if the map already contains null (so, null values are OK), and your remappingFunction returns null, then the entry is removed. That’s quite unexpected. Consider the following program:

map.put("X", null);
System.out.println(map.merge(
    "X", null, (v1, v2) -> null));
System.out.println(map);

Its output is:

null
{A=1, B=2, C=3}

Update: I first wrote the above code first with JDK 8 build 116. With build 129, things have changed completely again. First off, the value passed to merge() is not allowed to be null. Secondly. nullvalues are treated by merge() just like absent values. To produce the same output, we’ll write:

map.put("X", 1);
System.out.println(map.merge(
    "X", 1, (v1, v2) -> null));
System.out.println(map);

This merge() operation has thus removed a value from the map. That’s probably OK because the semantics of “merge” is often a combination of INSERT, UPDATE, and DELETE if we’re using SQL-speak. And a somewhat reasonable way to indicate that a value should be removed is to return null from such a function.

But the map is allowed to contain null values, which can never be inserted into the map using merge(). tweet this

getOrDefault()

This is a no-brainer. Right? Right! Wrong!

Unfortunately, there are two types of Maps. Those supporting null keys and/or values and those who don’t support nulls. While the previous merge() method didn’t distinguish between a map not containing a key and a map containing a key with a null value, this new getOrDefault() only returns the default when the key is not contained. It won’t protect you from a NullPointerException:

map.put("X", null);
try {
  System.out.println(map.getOrDefault("X", 21) + 21);
}
catch (NullPointerException nope) {
  nope.printStackTrace();
}

That’s quite a bummer. In general, it can be said the Map API has become even more complex with respect to nulls. tweet this

Trivial additions

There are a few more methods, like putIfAbsent() (pulled up from ConcurrentHashMap, remove() (with key and value arguments), replace().

Conclusion

All in all, it can be said that a lot of atomic operations have made it to the top-level Map API, which is good. But then again, the pre-existing confusion related to the semantics of null in maps has deepened. The terminologies “present” vs. “absent”, “contains”, “default” don’t necessarily help clarifying these things, which is surprisingly against the rules of keeping an API consistent and most importantly, regular. Thus as a consumer of this API, ideally, you should keep null out of maps, both as keys and as values!

Next week in this blog series, we’re going to look at how Java 8 will allow you to define local transactional scope very easily, so stay tuned!

More on Java 8

In the mean time, have a look at Eugen Paraschiv’s awesome Java 8 resources page