Java 8 Friday: 10 Subtle Mistakes When Using the Streams API

At Data Geekery, we love Java. And as we’re really into jOOQ’s fluent API and query DSL, we’re absolutely thrilled about what Java 8 will bring to our ecosystem.

Java 8 Friday

Every Friday, we’re showing you a couple of nice new tutorial-style Java 8 features, which take advantage of lambda expressions, extension methods, and other great stuff. You’ll find the source code on GitHub.

10 Subtle Mistakes When Using the Streams API

We’ve done all the SQL mistakes lists:

But we haven’t done a top 10 mistakes list with Java 8 yet! For today’s occasion (it’s Friday the 13th), we’ll catch up with what will go wrong in YOUR application when you’re working with Java 8. (it won’t happen to us, as we’re stuck with Java 6 for another while)

1. Accidentally reusing streams

Wanna bet, this will happen to everyone at least once. Like the existing “streams” (e.g. InputStream), you can consume streams only once. The following code won’t work:

IntStream stream = IntStream.of(1, 2);
stream.forEach(System.out::println);

// That was fun! Let's do it again!
stream.forEach(System.out::println);

You’ll get a

java.lang.IllegalStateException: 
  stream has already been operated upon or closed

So be careful when consuming your stream. It can be done only once

2. Accidentally creating “infinite” streams

You can create infinite streams quite easily without noticing. Take the following example:

// Will run indefinitely
IntStream.iterate(0, i -> i + 1)
         .forEach(System.out::println);

The whole point of streams is the fact that they can be infinite, if you design them to be. The only problem is, that you might not have wanted that. So, be sure to always put proper limits:

// That's better
IntStream.iterate(0, i -> i + 1)
         .limit(10)
         .forEach(System.out::println);

3. Accidentally creating “subtle” infinite streams

We can’t say this enough. You WILL eventually create an infinite stream, accidentally. Take the following stream, for instance:

IntStream.iterate(0, i -> ( i + 1 ) % 2)
         .distinct()
         .limit(10)
         .forEach(System.out::println);

So…

  • we generate alternating 0’s and 1’s
  • then we keep only distinct values, i.e. a single 0 and a single 1
  • then we limit the stream to a size of 10
  • then we consume it

Well… the distinct() operation doesn’t know that the function supplied to the iterate() method will produce only two distinct values. It might expect more than that. So it’ll forever consume new values from the stream, and the limit(10) will never be reached. Tough luck, your application stalls.

4. Accidentally creating “subtle” parallel infinite streams

We really need to insist that you might accidentally try to consume an infinite stream. Let’s assume you believe that the distinct() operation should be performed in parallel. You might be writing this:

IntStream.iterate(0, i -> ( i + 1 ) % 2)
         .parallel()
         .distinct()
         .limit(10)
         .forEach(System.out::println);

Now, we’ve already seen that this will turn forever. But previously, at least, you only consumed one CPU on your machine. Now, you’ll probably consume four of them, potentially occupying pretty much all of your system with an accidental infinite stream consumption. That’s pretty bad. You can probably hard-reboot your server / development machine after that. Have a last look at what my laptop looked like prior to exploding:

If I were a laptop, this is how I'd like to go.
If I were a laptop, this is how I’d like to go.

5. Mixing up the order of operations

So, why did we insist on your definitely accidentally creating infinite streams? It’s simple. Because you may just accidentally do it. The above stream can be perfectly consumed if you switch the order of limit() and distinct():

IntStream.iterate(0, i -> ( i + 1 ) % 2)
         .limit(10)
         .distinct()
         .forEach(System.out::println);

This now yields:

0
1

Why? Because we first limit the infinite stream to 10 values (0 1 0 1 0 1 0 1 0 1), before we reduce the limited stream to the distinct values contained in it (0 1).

Of course, this may no longer be semantically correct, because you really wanted the first 10 distinct values from a set of data (you just happened to have “forgotten” that the data is infinite). No one really wants 10 random values, and only then reduce them to be distinct.

If you’re coming from a SQL background, you might not expect such differences. Take SQL Server 2012, for instance. The following two SQL statements are the same:

-- Using TOP
SELECT DISTINCT TOP 10 *
FROM i
ORDER BY ..

-- Using FETCH
SELECT *
FROM i
ORDER BY ..
OFFSET 0 ROWS
FETCH NEXT 10 ROWS ONLY

So, as a SQL person, you might not be as aware of the importance of the order of streams operations.

jOOQ, the best way to write SQL in Java

6. Mixing up the order of operations (again)

Speaking of SQL, if you’re a MySQL or PostgreSQL person, you might be used to the LIMIT .. OFFSET clause. SQL is full of subtle quirks, and this is one of them. The OFFSET clause is applied FIRST, as suggested in SQL Server 2012’s (i.e. the SQL:2008 standard’s) syntax.

If you translate MySQL / PostgreSQL’s dialect directly to streams, you’ll probably get it wrong:

IntStream.iterate(0, i -> i + 1)
         .limit(10) // LIMIT
         .skip(5)   // OFFSET
         .forEach(System.out::println);

The above yields

5
6
7
8
9

Yes. It doesn’t continue after 9, because the limit() is now applied first, producing (0 1 2 3 4 5 6 7 8 9). skip() is applied after, reducing the stream to (5 6 7 8 9). Not what you may have intended.

BEWARE of the LIMIT .. OFFSET vs. "OFFSET .. LIMIT" trap!

7. Walking the file system with filters

We’ve blogged about this before. What appears to be a good idea is to walk the file system using filters:

Files.walk(Paths.get("."))
     .filter(p -> !p.toFile().getName().startsWith("."))
     .forEach(System.out::println);

The above stream appears to be walking only through non-hidden directories, i.e. directories that do not start with a dot. Unfortunately, you’ve again made mistake #5 and #6. walk() has already produced the whole stream of subdirectories of the current directory. Lazily, though, but logically containing all sub-paths. Now, the filter will correctly filter out paths whose names start with a dot “.”. E.g. .git or .idea will not be part of the resulting stream. But these paths will be: .\.git\refs, or .\.idea\libraries. Not what you intended.

Now, don’t fix this by writing the following:

Files.walk(Paths.get("."))
     .filter(p -> !p.toString().contains(File.separator + "."))
     .forEach(System.out::println);

While that will produce the correct output, it will still do so by traversing the complete directory subtree, recursing into all subdirectories of “hidden” directories.

I guess you’ll have to resort to good old JDK 1.0 File.list() again. The good news is, FilenameFilter and FileFilter are both functional interfaces.

8. Modifying the backing collection of a stream

While you’re iterating a List, you must not modify that same list in the iteration body. That was true before Java 8, but it might become more tricky with Java 8 streams. Consider the following list from 0..9:

// Of course, we create this list using streams:
List<Integer> list = 
IntStream.range(0, 10)
         .boxed()
         .collect(toCollection(ArrayList::new));

Now, let’s assume that we want to remove each element while consuming it:

list.stream()
    // remove(Object), not remove(int)!
    .peek(list::remove)
    .forEach(System.out::println);

Interestingly enough, this will work for some of the elements! The output you might get is this one:

0
2
4
6
8
null
null
null
null
null
java.util.ConcurrentModificationException

If we introspect the list after catching that exception, there’s a funny finding. We’ll get:

[1, 3, 5, 7, 9]

Heh, it “worked” for all the odd numbers. Is this a bug? No, it looks like a feature. If you’re delving into the JDK code, you’ll find this comment in ArrayList.ArraListSpliterator:

/*
 * If ArrayLists were immutable, or structurally immutable (no
 * adds, removes, etc), we could implement their spliterators
 * with Arrays.spliterator. Instead we detect as much
 * interference during traversal as practical without
 * sacrificing much performance. We rely primarily on
 * modCounts. These are not guaranteed to detect concurrency
 * violations, and are sometimes overly conservative about
 * within-thread interference, but detect enough problems to
 * be worthwhile in practice. To carry this out, we (1) lazily
 * initialize fence and expectedModCount until the latest
 * point that we need to commit to the state we are checking
 * against; thus improving precision.  (This doesn't apply to
 * SubLists, that create spliterators with current non-lazy
 * values).  (2) We perform only a single
 * ConcurrentModificationException check at the end of forEach
 * (the most performance-sensitive method). When using forEach
 * (as opposed to iterators), we can normally only detect
 * interference after actions, not before. Further
 * CME-triggering checks apply to all other possible
 * violations of assumptions for example null or too-small
 * elementData array given its size(), that could only have
 * occurred due to interference.  This allows the inner loop
 * of forEach to run without any further checks, and
 * simplifies lambda-resolution. While this does entail a
 * number of checks, note that in the common case of
 * list.stream().forEach(a), no checks or other computation
 * occur anywhere other than inside forEach itself.  The other
 * less-often-used methods cannot take advantage of most of
 * these streamlinings.
 */

Now, check out what happens when we tell the stream to produce sorted() results:

list.stream()
    .sorted()
    .peek(list::remove)
    .forEach(System.out::println);

This will now produce the following, “expected” output

0
1
2
3
4
5
6
7
8
9

And the list after stream consumption? It is empty:

[]

So, all elements are consumed, and removed correctly. The sorted() operation is a “stateful intermediate operation”, which means that subsequent operations no longer operate on the backing collection, but on an internal state. It is now “safe” to remove elements from the list!

Well… can we really? Let’s proceed with parallel(), sorted() removal:

list.stream()
    .sorted()
    .parallel()
    .peek(list::remove)
    .forEach(System.out::println);

This now yields:

7
6
2
5
8
4
1
0
9
3

And the list contains

[8]

Eek. We didn’t remove all elements!? Free beers (and jOOQ stickers) go to anyone who solves this streams puzzler!

This all appears quite random and subtle, we can only suggest that you never actually do modify a backing collection while consuming a stream. It just doesn’t work.

9. Forgetting to actually consume the stream

What do you think the following stream does?

IntStream.range(1, 5)
         .peek(System.out::println)
         .peek(i -> { 
              if (i == 5) 
                  throw new RuntimeException("bang");
          });

When you read this, you might think that it will print (1 2 3 4 5) and then throw an exception. But that’s not correct. It won’t do anything. The stream just sits there, never having been consumed.

As with any fluent API or DSL, you might actually forget to call the “terminal” operation. This might be particularly true when you use peek(), as peek() is an aweful lot similar to forEach().

This can happen with jOOQ just the same, when you forget to call execute() or fetch():

DSL.using(configuration)
   .update(TABLE)
   .set(TABLE.COL1, 1)
   .set(TABLE.COL2, "abc")
   .where(TABLE.ID.eq(3));

Oops. No execute()

jOOQ, the best way to write SQL in Java

Yes, the “best” way – with 1-2 caveats ;-)

10. Parallel stream deadlock

This is now a real goodie for the end!

All concurrent systems can run into deadlocks, if you don’t properly synchronise things. While finding a real-world example isn’t obvious, finding a forced example is. The following parallel() stream is guaranteed to run into a deadlock:

Object[] locks = { new Object(), new Object() };

IntStream
    .range(1, 5)
    .parallel()
    .peek(Unchecked.intConsumer(i -> {
        synchronized (locks[i % locks.length]) {
            Thread.sleep(100);

            synchronized (locks[(i + 1) % locks.length]) {
                Thread.sleep(50);
            }
        }
    }))
    .forEach(System.out::println);

Note the use of Unchecked.intConsumer(), which transforms the functional IntConsumer interface into a org.jooq.lambda.fi.util.function.CheckedIntConsumer, which is allowed to throw checked exceptions.

Well. Tough luck for your machine. Those threads will be blocked forever :-)

The good news is, it has never been easier to produce a schoolbook example of a deadlock in Java!

For more details, see also Brian Goetz’s answer to this question on Stack Overflow.

Conclusion

With streams and functional thinking, we’ll run into a massive amount of new, subtle bugs. Few of these bugs can be prevented, except through practice and staying focused. You have to think about how to order your operations. You have to think about whether your streams may be infinite.

Streams (and lambdas) are a very powerful tool. But a tool which we need to get a hang of, first.

Stay tuned for more exciting Java 8 articles on this blog.

Java 8 Friday: Better Exceptions

At Data Geekery, we love Java. And as we’re really into jOOQ’s fluent API and query DSL, we’re absolutely thrilled about what Java 8 will bring to our ecosystem.

Java 8 Friday

Every Friday, we’re showing you a couple of nice new tutorial-style Java 8 features, which take advantage of lambda expressions, extension methods, and other great stuff. You’ll find the source code on GitHub.

Better Exceptions

I had the idea when I stumbled upon JUnit GitHub issue #706, which is about a new method proposal:

ExpectedException#expect(Throwable, Callable)

One suggestion was to create an interceptor for exceptions like this.

assertEquals(Exception.class, 
    thrown(() -> foo()).getClass());
assertEquals("yikes!", 
    thrown(() -> foo()).getMessage());

On the other hand, why not just add something completely new along the lines of this?

// This is needed to allow for throwing Throwables
// from lambda expressions
@FunctionalInterface
interface ThrowableRunnable {
    void run() throws Throwable;
}

// Assert a Throwable type
static void assertThrows(
    Class<? extends Throwable> throwable,
    ThrowableRunnable runnable
) {
    assertThrows(throwable, runnable, t -> {});
}

// Assert a Throwable type and implement more
// assertions in a consumer
static void assertThrows(
    Class<? extends Throwable> throwable,
    ThrowableRunnable runnable,
    Consumer<Throwable> exceptionConsumer
) {
    boolean fail = false;
    try {
        runnable.run();
        fail = true;
    }
    catch (Throwable t) {
        if (!throwable.isInstance(t))
            Assert.fail("Bad exception type");

        exceptionConsumer.accept(t);
    }

    if (fail)
        Assert.fail("No exception was thrown");
}

So the above methods both assert that a given throwable is thrown from a given runnable – ThrowableRunnable to be precise, because most functional interfaces, unfortunately, don’t allow for throwing checked exceptions. See this article for details.

We’re now using the above hypothetical JUnit API as such:

assertThrows(Exception.class, 
    () -> { throw new Exception(); });

assertThrows(Exception.class, 
    () -> { throw new Exception("Message"); },
    e  -> assertEquals("Message", e.getMessage()));

In fact, we could even go further and declare an exception swallowing helper method like this:

// This essentially swallows exceptions
static void withExceptions(
    ThrowableRunnable runnable
) {
    withExceptions(runnable, t -> {});
}

// This delegates exception handling to a consumer
static void withExceptions(
    ThrowableRunnable runnable,
    Consumer<Throwable> exceptionConsumer
) {
    try {
        runnable.run();
    }
    catch (Throwable t) {
        exceptionConsumer.accept(t);
    }
}

This is useful to swallow all sorts of exceptions. The following two idioms are thus equivalent:

try {
    // This will fail
    assertThrows(SQLException.class, () -> {
        throw new Exception();
    });
}
catch (Throwable t) {
    t.printStackTrace();
}

withExceptions(
    // This will fail
    () -> assertThrows(SQLException.class, () -> {
        throw new Exception();
    }),
    t -> t.printStackTrace()
);

Obviuously, these idioms aren’t necessarily more useful than an actual try .. catch .. finally block, specifically also because it does not support proper typing of exceptions (at least not in this example), nor does it support the try-with-resources statement.

Nonetheless, such utility methods will come in handy every now and then.

Next week

Stay tuned for more Java 8 goodness on this blog when we continue our Java 8 Friday series with great new examples.

Java 8 Friday: API Designers, be Careful

At Data Geekery, we love Java. And as we’re really into jOOQ’s fluent API and query DSL, we’re absolutely thrilled about what Java 8 will bring to our ecosystem.

Java 8 Friday

Every Friday, we’re showing you a couple of nice new tutorial-style Java 8 features, which take advantage of lambda expressions, extension methods, and other great stuff. You’ll find the source code on GitHub.

Lean Functional API Design

With Java 8, API design has gotten a whole lot more interesting, but also a bit harder. As a successful API designer, it will no longer suffice to think about all sorts of object-oriented aspects of your API, you will now also need to consider functional aspects of it. In other words, instead of simply providing methods like:

void performAction(Parameter parameter);

// Call the above:
object.performAction(new Parameter(...));

… you should now think about whether your method arguments are better modelled as functions for lazy evaluation:

// Keep the existing method for convenience
// and for backwards compatibility
void performAction(Parameter parameter);

// Overload the existing method with the new
// functional one:
void performAction(Supplier<Parameter> parameter);

// Call the above:
object.performAction(() -> new Parameter(...));

This is great. Your API can be Java-8 ready even before you’re actually targeting Java 8. But if you’re going this way, there are a couple of things to consider.

JDK dependency

The above example makes use of the JDK 8 Supplier type. This type is not available before the JDK 8, so if you’re using it, you’re going to limit your APIs use to the JDK 8. If you want to continue supporting older Java versions, you’ll have to roll your own supplier, or maybe use Callable, which has been available since Java 5:

// Overload the existing method with the new
// functional one:
void performAction(Callable<Parameter> parameter);

// Call the above:
object.performAction(() -> new Parameter(...));

One advantage of using Callable is the fact that your lambda expressions (or “classic” Callable implementations, or nested / inner classes) are allowed to throw checked exceptions. We’ve blogged about another possibility to circumvent this limitation, here.

Overloading

While it is (probably) perfectly fine to overload these two methods

void performAction(Parameter parameter);
void performAction(Supplier<Parameter> parameter);

… you should stay wary when overloading “more similar” methods, like these ones:

void performAction(Supplier<Parameter> parameter);
void performAction(Callable<Parameter> parameter);

If you produce the above API, your API’s client code will not be able to make use of lambda expressions, as there is no way of disambiguating a lambda that is a Supplier from a lambda that is a Callable. We’ve also mentioned this in a previous blog post.

“void-compatible” vs “value-compatible”

I’ve recently (re-)discovered this interesting early JDK 8 compiler bug, where the compiler wasn’t able to disambiguate the following:

void run(Consumer<Integer> consumer);
void run(Function<Integer, Integer> function);

// Remember, the above types are roughly:
interface Consumer<T> {
    void accept(T t);
//  ^^^^ void-compatible
}

interface Function<T, R> {
    R apply(T t);
//  ^ value-compatible
}

The terms “void-compatible” and “value-compatible” are defined in the JLS §15.27.2 for lambda expressions. According to the JLS, the following two calls are not ambiguous:

// Only run(Consumer) is applicable
run(i -> {});

// Only run(Function) is applicable
run(i -> 1);

In other words, it is safe to overload a method to take two “similar” argument types, such as Consumer and Function, as lambda expressions used to express method arguments will not be ambiguous.

This is quite useful, because having an optional return value is very elegant when you’re using lambda expressions. Consider the upcoming jOOQ 3.4 transaction API, which is roughly summarised as such:


// This uses a "void-compatible" lambda
ctx.transaction(c -> {
    DSL.using(c).insertInto(...).execute();
    DSL.using(c).update(...).execute();
});

// This uses a "value-compatible" lambda
Integer result =
ctx.transaction(c -> {
    DSL.using(c).update(...).execute();
    DSL.using(c).delete(...).execute();

    return 42;
});

In the above example, the first call resolves to TransactionalRunnable whereas the second call resolves to TransactionalCallable whose API are like these:

interface TransactionalRunnable {
    void run(Configuration c) throws Exception;
}

interface TransactionalCallable<T> {
    T run(Configuration c) throws Exception;
}

Note, though, that as of JDK 1.8.0_05 and Eclipse Kepler (with the Java 8 support patch), this ambiguity resolution does not yet work because of these bugs:

So, in order to stay on the safe side, maybe you could just simply avoid overloading.

Generic methods are not SAMs

Do note that “SAM” interfaces that contain a single abstract generic method are NOT SAMs in the sense for them to be eligible as lambda expression targets. The following type will never form any lambda expression:

interface NotASAM {
    <T> void run(T t);
}

This is specified in the JLS §15.27.3

A lambda expression is congruent with a function type if all of the following are true:

  • The function type has no type parameters.
  • [ … ]

What do you have to do now?

If you’re an API designer, you should now start writing unit tests / integration tests also in Java 8. Why? For the simple reason that if you don’t you’ll get your API wrong in subtle ways for those users that are actually using it with Java 8. These things are extremely subtle. Getting them right takes a bit of practice and a lot of regression tests. Do you think you’d like to overload a method? Be sure you don’t break client API that is calling the original method with a lambda.

That’s it for today. Stay tuned for more awesome Java 8 content on this blog.

Java 8 Friday: Language Design is Subtle

At Data Geekery, we love Java. And as we’re really into jOOQ’s fluent API and query DSL, we’re absolutely thrilled about what Java 8 will bring to our ecosystem.

Java 8 Friday

Every Friday, we’re showing you a couple of nice new tutorial-style Java 8 features, which take advantage of lambda expressions, extension methods, and other great stuff. You’ll find the source code on GitHub.

Language Design is Subtle

It’s been a busy week for us. We have just migrated the jOOQ integration tests to Java 8 for two reasons:

  • We want to be sure that client code compiles with Java 8
  • We started to get bored of writing the same old loops over and over again

The trigger was a loop where we needed to transform a SQLDialect[] into another SQLDialect[] calling .family() on each array element. Consider:

Java 7

SQLDialect[] families = 
    new SQLDialect[dialects.length];
for (int i = 0; i < families.length; i++)
    families[i] = dialects[i].family();

Java 8

SQLDialect[] families = 
Stream.of(dialects)
      .map(d -> d.family())
      .toArray(SQLDialect[]::new);

OK, it turns out that the two solutions are equally verbose, even if the latter feels a bit more elegant. :-)

And this gets us straight into the next topic:

Backwards-compatibility

For backwards-compatibility reasons, arrays and the pre-existing Collections API have not been retrofitted to accommodate all the useful methods that Streams now have. In other words, an array doesn’t have a map() method, just as much as List doesn’t have such a method. Streams and Collections/arrays are orthogonal worlds. We can transform them into each other, but they don’t have a unified API.

This is fine in everyday work. We’ll get used to the Streams API and we’ll love it, no doubt. But because of Java being extremely serious about backwards compatibility, we will have to think about one or two things more deeply.

Recently, we have published a post about The Dark Side of Java 8. It was a bit of a rant, although a mild one in our opinion (and it was about time to place some criticism, after all the praise we’ve been giving Java 8 in our series, before ;-) ). First off, that post triggered a reaction by Edwin Dalorzo from our friends at Informatech. (Edwin has written this awesome post comparing LINQ and Java 8 Streams, before). The criticism in our article evolved around three main aspects:

  • Overloading getting more complicated (see also this compiler bug)
  • Limited support for method modifiers on default methods
  • Primitive type “API overloads” for streams and functional interfaces

A response by Brian Goetz

I then got a personal mail from no one less than Brian Goetz himself (!), who pointed out a couple of things to me that I had not yet thought about in this way:

I still think you’re focusing on the wrong thing. Its not really the syntax you don’t like; its the model — you don’t want “default methods”, you want traits, and the syntax is merely a reminder that you didn’t get the feature you wanted. (But you’d be even more confused about “why can’t they be final” if we dropped the “default” keyword!) But that’s blaming the messenger (where here, the keyword is the messenger.)

Its fair to say “this isn’t the model I wanted”. There were many possible paths in the forest, and it may well be the road not taken was equally good or better.

This is also what Edwin had concluded. Default methods were a necessary means to tackle all the new API needed to make Java 8 useful. If Iterator, Iterable, List, Collection, and all the other pre-existing interfaces had to be adapted to accommodate lambdas and Streams API interaction, the expert group would have needed to break an incredible amount of API. Conversely, without adding these additional utility methods (see the awesome new Map methods, for instance!), Java 8 would have been only half as good.

And that’s it.

Even if maybe, some more class building tools might have been useful, they were not in the center of focus for the expert group who already had a lot to do to get things right. The center of focus was to provide a means for API evolution. Or in Brian Goetz’s own words:

Reaching out to the community

It’s great that Brian Goetz reaches out to the community to help us get the right picture about Java 8. Instead of explaining rationales about expert group decisions in private messages, he then asked me to publicly re-ask my questions again on Stack Overflow (or lambda-dev), such that he can then publicly answer them. For increased publicity and greater community benefit, I chose Stack Overflow. Here are:

The amount of traction these two questions got in no time shows how important these things are to the community, so don’t miss reading through them!

“Uncool”? Maybe. But very stable!

Java may not have the “cool” aura that node.js has. You may think about JavaScript-the-language whatever you want (as long as it contains swear words), but from a platform marketing perspective, Java is being challenged for the first time in a long time – and being “uncool” and backwards-compatible doesn’t help keeping developers interested.

But let’s think long-term, instead of going with trends. Having such a great professional platform like the Java language, the JVM, the JDK, JEE, and much more, is invaluable. Because at the end of the day, the “uncool” backwards-compatibility can also be awesome. As mentioned initially, we have upgraded our integration tests to Java 8. Not a single compilation error, not a single bug. Using Eclipse’s BETA support for Java 8, I could easily transform anonymous classes into lambdas and write awesome things like these upcoming jOOQ 3.4 nested transactions (API not final yet):

ctx.transaction(c1 -> {
    DSL.using(c1)
       .insertInto(AUTHOR, AUTHOR.ID, AUTHOR.LAST_NAME)
       .values(3, "Doe")
       .execute();

    // Implicit savepoint here
    try {
        DSL.using(c1).transaction(c2 -> {
            DSL.using(c2)
               .update(AUTHOR)
               .set(AUTHOR.FIRST_NAME, "John")
               .where(AUTHOR.ID.eq(3))
               .execute();

            // Rollback to savepoint
            throw new MyRuntimeException("No");
        });
    }

    catch (MyRuntimeException ignore) {}

    return 42;
});

So at the end of the day, Java is great. Java 8 is a tremendous improvement over previous versions, and with great people in the expert groups (and reaching out to the community on social media), I trust that Java 9 will be even better. In particular, I’m looking forward to learning about how these two projects evolve:

Although, again, I am really curious how they will pull these two improvements off from a backwards-compatibility perspective, and what caveats we’ll have to understand, afterwards. ;-)

Anyway, let’s hope the expert groups will continue to provide public feedback on Stack Overflow. Stay tuned for more awesome Java 8 content on this blog.

Java 8 Friday: The Dark Side of Java 8

At Data Geekery, we love Java. And as we’re really into jOOQ’s fluent API and query DSL, we’re absolutely thrilled about what Java 8 will bring to our ecosystem.

Java 8 Friday

Every Friday, we’re showing you a couple of nice new tutorial-style Java 8 features, which take advantage of lambda expressions, extension methods, and other great stuff. You’ll find the source code on GitHub.

The dark side of Java 8

So far, we’ve been showing the thrilling parts of this new major release. But there are also caveats. Lots of them. Things that

  • … are confusing
  • … are wrong
  • … are omitted (for now)
  • … are omitted (for long)

There are always two sides to Java major releases. On the bright side, we get lots of new functionality that most people would say was overdue. Other languages, platforms have had generics long before Java 5. Other languages, platforms have had lambdas long before Java 8. But now, we finally have these features. In the usual quirky Java-way.

Lambda expressions were introduced quite elegantly. The idea of being able to write every anonymous SAM instance as a lambda expression is very compelling from a backwards-compatiblity point of view. So what are the dark sides to Java 8?

Overloading gets even worse

Overloading, generics, and varargs aren’t friends. We’ve explained this in a previous article, and also in this Stack Overflow question. These might not be every day problems in your odd application, but they’re very important problems for API designers and maintainers.

With lambda expressions, things get “worse”. So you think you can provide some convenience API, overloading your existing run() method that accepts a Callable to also accept the new Supplier type:

static <T> T run(Callable<T> c) throws Exception {
    return c.call();
}

static <T> T run(Supplier<T> s) throws Exception {
    return s.get();
}

What looks like perfectly useful Java 7 code is a major pain in Java 8, now. Because you cannot just simply call these methods with a lambda argument:

public static void main(String[] args)
throws Exception {
    run(() -> null);
    //  ^^^^^^^^^^ ambiguous method call
}

Tough luck. You’ll have to resort to either of these “classic” solutions:

    run((Callable<Object>) (() -> null));
    run(new Callable<Object>() {
        @Override
        public Object call() throws Exception {
            return null;
        }
    });

So, while there’s always a workaround, these workarounds always “suck”. That’s quite a bummer, even if things don’t break from a backwards-compatibility perspective.

Not all keywords are supported on default methods

Default methods are a nice addition. Some may claim that Java finally has traits. Others clearly dissociate themselves from the term, e.g. Brian Goetz:

The key goal of adding default methods to Java was “interface evolution”, not “poor man’s traits.”

As found on the lambda-dev mailing list.

Fact is, default methods are quite a bit of an orthogonal and irregular feature to anything else in Java. Here are a couple of critiques:

They cannot be made final

Given that default methods can also be used as convenience methods in API:

public interface NoTrait {

    // Run the Runnable exactly once
    default final void run(Runnable r) {
        //  ^^^^^ modifier final not allowed
        run(r, 1);
    }

    // Run the Runnable "times" times
    default void run(Runnable r, int times) {
        for (int i = 0; i < times; i++)
            r.run();
    }
}

Unfortunately, the above is not possible, and so the first overloaded convenience method could be overridden in subtypes, even if that makes no sense to the API designer.

They cannot be made synchronized

Bummer! Would that have been difficult to implement in the language?

public interface NoTrait {
    default synchronized void noSynchronized() {
        //  ^^^^^^^^^^^^ modifier synchronized
        //  not allowed
        System.out.println("noSynchronized");
    }
}

Yes, synchronized is used rarely, just like final. But when you have that use-case, why not just allow it? What makes interface method bodies so special?

The default keyword

This is maybe the weirdest and most irregular of all features. The default keyword itself. Let’s compare interfaces and abstract classes:


// Interfaces are always abstract
public /* abstract */ interface NoTrait {

    // Abstract methods have no bodies
    // The abstract keyword is optional
    /* abstract */ void run1();

    // Concrete methods have bodies
    // The default keyword is mandatory
    default void run2() {}
}

// Classes can optionally be abstract
public abstract class NoInterface {

    // Abstract methods have no bodies
    // The abstract keyword is mandatory
    abstract void run1();

    // Concrete methods have bodies
    // The default keyword mustn't be used
    void run2() {}
}

If the language were re-designed from scratch, it would probably do without any of abstract or default keywords. Both are unnecessary. The mere fact that there is or is not a body is sufficient information for the compiler to assess whether a method is abstract. I.e, how things should be:

public interface NoTrait {
    void run1();
    void run2() {}
}

public abstract class NoInterface {
    void run1();
    void run2() {}
}

The above would be much leaner and more regular. It’s a pity that the usefulness of default was never really debated by the EG. Well, it was debated but the EG never wanted to accept this as an option. I’ve tried my luck, with this response:

I don’t think #3 is an option because interfaces with method bodies are unnatural to begin with. At least specifying the “default” keyword gives the reader some context why the language allows a method body. Personally, I wish interfaces would remain as pure contracts (without implementation), but I don’t know of a better option to evolve interfaces.

Again, this is a clear commitment by the EG not to commit to the vision of “traits” in Java. Default methods were a pure necessary means to implement 1-2 other features. They weren’t well-designed from the beginning.

Other modifiers

Luckily, the static modifier made it into the specs, late in the project. It is thus possible to specifiy static methods in interfaces now. For some reason, though, these methods do not need (nor allow!) the default keyword, which must’ve been a totally random decision by the EG, just like you apparently cannot define static final methods in interfaces.

While visibility modifiers were discussed on the lambda-dev mailing list, but were out of scope for this release. Maybe, we can get them in a future release.

Few default methods were actually implemented

Some methods would have sensible default implementations on interface – one might guess. Intuitively, the collections interfaces, like List or Set would have them on their equals() and hashCode() methods, because the contract for these methods is well-defined on the interfaces. It is also implemented in AbstractList, using listIterator(), which is a reasonable default implementation for most tailor-made lists.

It would’ve been great if these API were retrofitted to make implementing custom collections easier with Java 8. I could make all my business objects implement List for instance, without wasting the single base-class inheritance on AbstractList.

Probably, though, there has been a compelling reason related to backwards-compatibility that prevented the Java 8 team at Oracle from implementing these default methods. Whoever sends us the reason why this was omitted will get a free jOOQ sticker :-)

The wasn’t invented here – mentality

This, too, was criticised a couple of times on the lambda-dev EG mailing list. And while writing this blog series, I can only confirm that the new functional interfaces are very confusing to remember. They’re confusing for these reasons:

Some primitive types are more equal than others

The int, long, double primitive types are preferred compared to all the others, in that they have a functional interface in the java.util.function package, and in the whole Streams API. boolean is a second-class citizen, as it still made it into the package in the form of a BooleanSupplier or a Predicate, or worse: IntPredicate.

All the other primitive types don’t really exist in this area. I.e. there are no special types for byte, short, float, and char. While the argument of meeting deadlines is certainly a valid one, this quirky status-quo will make the language even harder to learn for newbies.

The types aren’t just called Function

Let’s be frank. All of these types are simply “functions”. No one really cares about the implicit difference between a Consumer, a Predicate, a UnaryOperator, etc.

In fact, when you’re looking for a type with a non-void return value and two arguments, what would you probably be calling it? Function2? Well, you were wrong. It is called a BiFunction.

Here’s a decision tree to know how the type you’re looking for is called:

  • Does your function return void? It’s called a Consumer
  • Does your function return boolean? It’s called a Predicate
  • Does your function return an int, long, double? It’s called XXToIntYY, XXToLongYY, XXToDoubleYY something
  • Does your function take no arguments? It’s called a Supplier
  • Does your function take a single int, long, double argument? It’s called an IntXX, LongXX, DoubleXX something
  • Does your function take two arguments? It’s called BiXX
  • Does your function take two arguments of the same type? It’s called BinaryOperator
  • Does your function return the same type as it takes as a single argument? It’s called UnaryOperator
  • Does your function take two arguments of which the first is a reference type and the second is a primitive type? It’s called ObjXXConsumer (only consumers exist with that configuration)
  • Else: It’s called Function

Good lord! We should certainly go over to Oracle Education to check if the price for Oracle Certified Java Programmer courses have drastically increased, recently… Thankfully, with Lambda expressions, we hardly ever have to remember all these types!

More on Java 8

Java 5 generics have brought a lot of great new features to the Java language. But there were also quite a few caveats related to type erasure. Java 8’s default methods, Streams API and lambda expressions will again bring a lot of great new features to the Java language and platform. But we’re sure that Stack Overflow will soon burst with questions by confused programmers that are getting lost in the Java 8 jungle.

Learning all the new features won’t be easy, but the new features (and caveats) are here to stay. If you’re a Java developer, you better start practicing now, when you get the chance. Because we have a long way to go.

Nonetheless, thigns are exciting, so stay tuned for more exciting Java 8 stuff published in this blog series.

Are you in for another critique about Java 8? Read “New Parallelism APIs in Java 8: Behind the Glitz and Glamour” by the guys over

Java 8 Friday Goodies: Local Transaction Scope

At Data Geekery, we love Java. And as we’re really into jOOQ’s fluent API and query DSL, we’re absolutely thrilled about what Java 8 will bring to our ecosystem. We have blogged a couple of times about some nice Java 8 goodies, and now we feel it’s time to start a new blog series, the…

Java 8 Friday

Every Friday, we’re showing you a couple of nice new tutorial-style Java 8 features, which take advantage of lambda expressions, extension methods, and other great stuff. You’ll find the source code on GitHub. tweet this

Java 8 Goodie: Local Transaction Scope

The JavaScript folks often abuse anonymous functions to create local scope. Like any other language feature, this can be abused, but in some contexts, local scoping is really awesome. Java also allows for local scoping, although until Java 8, this has been equally cumbersome:

JavaScript

(function() {
    var local = function() { 
            scoping(); 
        },
        scoping = function() { 
            alert('If you really must');
        };

    local();
})();

Java

new Object() {
    void local() {
        scoping();
    }
    void scoping() {
        System.out.println(
            "Ouch, my fingers. Too much typing");
    }
}.local();

Both examples look really awkward, although the JavaScript folks call this a design pattern. No one would create such local scope in Java, even if the two pieces of code are roughly equivalent.

tweet thisAwkwardness can be a design pattern in JavaScript.

Local scoping in Java 8

But with Java 8, everything changes, and so does local scoping. Let’s have a look at how we can creat a local semantic scope for transactions. For this, we’ll create two types. The Transactional interface:

@FunctionalInterface
interface Transactional {
    void run(DSLContext ctx);
}

For the example, we’re going to be using jOOQ to avoid checked exceptions and verbose statement creation. You can replace it by your SQL API of choice. So, jOOQ provides us with a locally scoped ctx object, which implicitly contains the transaction state. This transaction state is generated using a TransactionRunner:

class TransactionRunner {
    private final boolean silent;
    private final Connection connection;

    TransactionRunner(Connection connection) {
        this(connection, true);
    }

    TransactionRunner(Connection connection,
                      boolean silent) {
        this.connection = connection;
        this.silent = silent;
    }

    void run(Transactional tx) {
        // Initialise some jOOQ objects
        final DefaultConnectionProvider c =
            new DefaultConnectionProvider(connection);
        final Configuration configuration =
            new DefaultConfiguration()
                .set(c).set(SQLDialect.H2);

        try {
            // Run the transaction and pass a jOOQ
            // DSLContext object to it
            tx.run(DSL.using(configuration));

            // If we get here, then commit the
            // transaction
            c.commit();
        }
        catch (RuntimeException e) {

            // Any exception will cause a rollback
            c.rollback();
            System.err.println(e.getMessage());

            // Eat exceptions in silent mode.
            if (!silent)
                throw e;
        }
    }
}

The above is framework code, which we’ll write only once. From now on, we can use the above API trivially in our Java programs. For this, we’ll set up a TransactionRunner like such:

public static void main(String[] args) 
throws Exception {
    Class.forName("org.h2.Driver");
    try (Connection c = DriverManager.getConnection(
            "jdbc:h2:~/test-scope-goodies", 
            "sa", "")) {
        c.setAutoCommit(false);
        TransactionRunner silent = 
            new TransactionRunner(c);

        // Transactional code here ...
    }
}

And now, behold the wonders of Java 8!

// This is a transaction
silent.run(ctx -> {
    ctx.execute("drop table if exists person");
    ctx.execute("create table person(" + 
                "  id integer," +
                "  first_name varchar(50)," +
                "  last_name varchar(50)," +
                "  primary key(id)"+
                ")");
});

// And this is also one transaction
silent.run(ctx -> {
    ctx.execute("insert into person" +
                "  values(1, 'John', 'Smith');");
    ctx.execute("insert into person" +
                "  values(1, 'Steve', 'Adams');");
    // Ouch, fails -------^
    // Transaction rolls back
});

// And this is also one transaction
silent.run(ctx -> {
    ctx.execute("insert into person" + 
                "  values(2, 'Jane', 'Miller');");
    // Works, yay!
});

// And this is also one transaction
silent.run(ctx -> {
    ctx.execute("insert into person" +
                "  values(2, 'Anne', 'Roberts');");
    // Ouch, fails -------^
    // Transaction rolls back
});

What do we get from the above? Let’s check:

silent.run(ctx -> {
    System.out.println(
        ctx.fetch("select * from person"));
});

The above program will yield this output:

SQL [insert into person values(1, 'Steve', 'Adams');];
Unique index or primary key violation: "PRIMARY KEY ON PUBLIC.PERSON(ID)"; SQL statement:
insert into person values(1, 'Steve', 'Adams'); [23505-174]
SQL [insert into person values(2, 'Anne', 'Roberts');];
Unique index or primary key violation: "PRIMARY KEY ON PUBLIC.PERSON(ID)"; SQL statement:
insert into person values(2, 'Anne', 'Roberts'); [23505-174]
+----+----------+---------+
|  ID|FIRST_NAME|LAST_NAME|
+----+----------+---------+
|   2|Jane      |Miller   |
+----+----------+---------+

So, our commits and rollbacks worked as expected!

Nested transactions

We can also create nested calls to our TransactionRunner, e.g. when we’re inside methods calling other methods. For this, would have to adapt our TransactionRunner to count the nesting level, and remove the “silent” functionality. On the other hand, it would be very easy to implement savepoint functionality this way. Each time we nest another transaction, we’ll create a new savepoint.

Conclusion

As always in this series, we didn’t invent anything new. All of these things could be done with vanilla Java 7. But the client code of this TransactionRunner certainly wouldn’t look as lean as our lambdas.

Next week in this blog series, we’re going to look at how Java 8 will allow you to define local caching scope very easily, so stay tuned!

More on Java 8

In the mean time, have a look at Eugen Paraschiv’s awesome Java 8 resources page

Java 8 Friday Goodies: Lambdas and Sorting

At Data Geekery, we love Java. And as we’re really into jOOQ’s fluent API and query DSL, we’re absolutely thrilled about what Java 8 will bring to our ecosystem. We have blogged a couple of times about some nice Java 8 goodies, and now we feel it’s time to start a new blog series, the…

Java 8 Friday

Every Friday, we’re showing you a couple of nice new tutorial-style Java 8 features, which take advantage of lambda expressions, extension methods, and other great stuff. You’ll find the source code on GitHub. tweet this

Java 8 Goodie: Lambdas and Sorting

Sorting arrays, and collections is an awesome use-case for Java 8’s lambda expression for the simple reason that Comparator has always been a @FunctionalInterface all along since its introduction in JDK 1.2. We can now supply Comparators in the form of a lambda expression to various sort() methods.

For the following examples, we’re going to use this simple Person class:

static class Person {
    final String firstName;
    final String lastName;

    Person(String firstName, String lastName) {
        this.firstName = firstName;
        this.lastName = lastName;
    }

    @Override
    public String toString() {
        return "Person{" +
                "firstName='" + firstName + '\'' +
                ", lastName='" + lastName + '\'' +
                '}';
    }
}

Obviously, we could add natural sorting to Person as well by letting it implement Comparable, but lets focus on external Comparators. Consider the following list of Person, whose names are generated with some online random name generator:

List<Person> people =
Arrays.asList(
    new Person("Jane", "Henderson"),
    new Person("Michael", "White"),
    new Person("Henry", "Brighton"),
    new Person("Hannah", "Plowman"),
    new Person("William", "Henderson")
);

We probably want to sort them by last name and then by first name.

Sorting with Java 7

A “classic” Java 7 example of such a Comparator is this:

people.sort(new Comparator<Person>() {
  @Override
  public int compare(Person o1, Person o2) {
    int result = o1.lastName.compareTo(o2.lastName);

    if (result == 0)
      result = o1.firstName.compareTo(o2.firstName);

    return result;
  }
});
people.forEach(System.out::println);

And the above would yield:

Person{firstName='Henry', lastName='Brighton'}
Person{firstName='Jane', lastName='Henderson'}
Person{firstName='William', lastName='Henderson'}
Person{firstName='Hannah', lastName='Plowman'}
Person{firstName='Michael', lastName='White'}

Sorting with Java 8

Now, let’s translate the above to equivalent Java 8 code:

Comparator<Person> c = (p, o) ->
    p.lastName.compareTo(o.lastName);

c = c.thenComparing((p, o) ->
    p.firstName.compareTo(o.firstName));

people.sort(c);
people.forEach(System.out::println);

The result is obviously the same. How to read the above? First, we assign a lambda expression to a local Person Comparator variable:

Comparator<Person> c = (p, o) ->
    p.lastName.compareTo(o.lastName);

Unlike Scala, C#, or Ceylon which know type inference from an expression towards a local variable declaration through a val keyword (or similar), Java performs type inference from a variable (or parameter, member) declaration towards an expression that is being assigned.

In other, more informal words, type inference is performed from “left to right”, not from “right to left”. This makes chaining Comparators a bit cumbersome, as the Java compiler cannot delay type inference for lambda expressions until you pass the comparator to the sort() method.

Once we have assigned a Comparator to a variable, however, we can fluently chain other comparators through thenComparing():

c = c.thenComparing((p, o) ->
    p.firstName.compareTo(o.firstName));

And finally, we pass it to the List‘s new sort() method, which is a default method implemented directly on the List interface:

default void sort(Comparator<? super E> c) {
    Collections.sort(this, c);
}

Workaround for the above limitation

While Java’s type inference “limitations” can turn out to be a bit frustrating, we can work around type inference by creating a generic IdentityComparator:

class Utils {
    static <E> Comparator<E> compare() {
        return (e1, e2) -> 0;
    }
}

With the above compare() method, we can write the following fluent comparator chain:

people.sort(
    Utils.<Person>compare()
         .thenComparing((p, o) -> 
              p.lastName.compareTo(o.lastName))
         .thenComparing((p, o) -> 
              p.firstName.compareTo(o.firstName))
);

people.forEach(System.out::println);

Extracting keys

This can get even better. Since we’re usually comparing the same POJO / DTO value from both Comparator arguments, we can provide them to the new APIs through a “key extractor” function. This is how it works:

people.sort(Utils.<Person>compare()
      .thenComparing(p -> p.lastName)
      .thenComparing(p -> p.firstName));
people.forEach(System.out::println);

So, given a Person p we provide the API with a function extracting, for instance, p.lastName. And in fact, once we use key extractors, we can omit our own utility method, as the libraries also have a comparing() method to initiate the whole chain:

people.sort(
    Comparator.comparing((Person p) -> p.lastName)
          .thenComparing(p -> p.firstName));
people.forEach(System.out::println);

Again, we need to help the compiler as it cannot infer all types, even if in principle, the sort() method would provide enough information in this case. To learn more about Java 8’s generalized type inference, see our previous blog post.

Conclusion

As with Java 5, the biggest improvements of the upgrade can be seen in the JDK libraries. When Java 5 brought typesafety to Comparators, Java 8 makes them easy to read and write (give or take the odd type inference quirk).

Java 8 is going to revolutionise the way we program, and next week, we will see how Java 8 impacts the way we interact with SQL.

More on Java 8

In the mean time, have a look at Eugen Paraschiv’s awesome Java 8 resources page