This isn’t going to be a rant about functional programming, which is awesome. This is a warning about some practices that you are very likely going to apply to your code, which are
terribly wrong!.
Higher order functions are essential to functional programming, and thus, talking about them will help you be the center of attention at parties.
If you’re writing JavaScript, you’re doing it all the time. For instance:
setTimeout(function() {
alert('10 Seconds passed');
}, 10000);
The above
setTimeout() function is a higher-order function. It is a function that takes an anonymous function as an argument. After 10 seconds, it will call the function passed as an argument.
We can write another easy higher-order function that
provides the above function as a result:
var message = function(text) {
return function() {
alert(text);
}
};
setTimeout(message('10 Seconds passed'), 10000);
If you execute the above,
message() will be executed, returning an anonymous function, which alerts the argument text you have passed to
message()
In functional programming, the above is common practice. A function returned from a higher-order function will capture the outer scope and is able to act on this scope when called.
Why is this practice treacherous in Java?
For the same reasons. A “function” (lambda) returned from a higher-order “function” (method) will capture the outer scope and is able to act on this scope when called.
The most trivial example is given here:
class Test {
public static void main(String[] args) {
Runnable runnable = runnable();
runnable.run(); // Breakpoint here
}
static Runnable runnable() {
return () -> {
System.out.println("Hello");
};
}
}
In the above logic, if you put a breakpoint right where the
runnable.run() call is made, you can see the harmless lambda instance on the stack. A simple generated class, backing the functional interface implementation:
Now let’s translate this example to your average Enterprise™ application (
notice the annotations), which we’ve greatly simplified to fit this blog post:
class Test {
public static void main(String[] args) {
Runnable runnable = new EnterpriseBean()
.runnable();
runnable.run(); // Breakpoint here
}
}
@ImportantDeclaration
@NoMoreXML({
@CoolNewValidationStuff("Annotations"),
@CoolNewValidationStuff("Rock")
})
class EnterpriseBean {
Object[] enterpriseStateObject =
new Object[100_000_000];
Runnable runnable() {
return () -> {
System.out.println("Hello");
};
}
}
The breakpoint is still at the same spot. What do we see on the stack?
Still a harmless little lambda instance:
Fine. Of course. Let’s add some additional logging, just for debugging
class Test {
public static void main(String[] args) {
Runnable runnable = new EnterpriseBean()
.runnable();
runnable.run(); // Breakpoint here
}
}
@ImportantDeclaration
@NoMoreXML({
@CoolNewValidationStuff("Annotations"),
@CoolNewValidationStuff("Rock")
})
class EnterpriseBean {
Object[] enterpriseStateObject =
new Object[100_000_000];
Runnable runnable() {
return () -> {
// Some harmless debugging here
System.out.println("Hello from: " + this);
};
}
}
Suddenly, the “harmless” little
this reference forced the Java compiler to enclose the enclosing instance of the
EnterpriseBean™ in the returned
Runnable class:
And with it that heavy
enterpriseStateObject came along, which can now no longer be garbage collected, until the call site releases the harmless little
Runnable
Sure. Just be careful, know what you’re doing, and don’t reference “this” from a lambda
… you say?
How about a more subtle version?
class EnterpriseBean {
Object[] enterpriseStateObject =
new Object[100_000_000];
Runnable runnable() {
return () -> log(); // implicit this.log()
}
void log() {
// Some harmless debugging here
System.out.println("Hello");
}
}
OK, this is nothing new now, is it?
Indeed, it isn’t. Java 8 doesn’t have first-class functions, and that’s OK. The idea of backing lambda expressions by nominal SAM types is quite cunning, as it allowed to upgrade and lambda-y-fy all existing libraries in the Java ecosystem without changing them.
Also, with an anonymous class, this whole story would not have been surprising. The following coding style has leaked internal state via anonymous classes since good old Swing 1.0 style
ActionListener et al.
class Test {
public static void main(String[] args) {
Runnable runnable = new EnterpriseBean()
.runnable();
runnable.run();
}
}
@ImportantDeclaration
@NoMoreXML({
@CoolNewValidationStuff("Annotations"),
@CoolNewValidationStuff("Rock")
})
class EnterpriseBean {
Object[] enterpriseStateObject =
new Object[100_000_000];
Runnable runnable() {
return new Runnable() {
@Override
public void run() {
System.out.println("Hello from " + EnterpriseBean.this);
}
};
}
}
What’s new? The lambda style will encourage using higher-order functions in Java, all over the place. Which is generally good. But only when the higher-order function is a static method, whose resulting types will not enclose any state.
With the above examples, however, we can see that we’re going to be debugging through a couple of memory leaks and problems in the near future, when we start embracing Java 8’s functional style programming.
So, be careful, and follow this rule:
(“Pure”) Higher order functions MUST be static methods in Java!
Further reading
Enclosing instances have caused issues before. Read about
how the dreaded double curly braces anti pattern has caused pain and suffering among Java developers for the last two decades.
Like this:
Like Loading...
There’s one thing I don’t understand. If I have a static higher order function but one that takes a lambda or anonymous inner class that captures the “this” of the calling class, won’t I still have the same problem? That is, the first class function that I pass to the higher order function has that massive class in its scope.
Yes, that can still happen. However, in most cases, it seems less “risky” to pass a (scope-capturing) function you control to some library method, rather than to return a (scope-capturing) function from a library method to the universe. In the first case, you can assess the risk. In the second case, you cannot as you may not know who will consume your function.
Nice, but it’s a lot of words to say that ‘this’ can be dangerous when poorly understood ;-)
You mean, it could’ve been said in a Tweet? :)
True. But hand-waving is how blogs generate traffic, right?
Cool post, thanks. Once you laid out the problem it’s pretty simple to understand, but I admit I didn’t think of it until I read your explanation.
Instead of using static methods, it might be more appropriate to advocate for pure functions (https://en.wikipedia.org/wiki/Pure_function).
You’re right, of course. But
staticis a good trick to get a bit closer to the goal of writing pure functions.(warning: pedantic/nitpick comment) IMO, with the way functions are handled on the JVM, methods (static or otherwise) != functions. Functions on the JVM are generally addressable and method references are a form of ETA expansion, but methods are generally not addressable. I might rephrase and say that the trick is to make use of method references to static methods.
Technically, you’re right. Conceptually, that *is* pedantic ;)
Yep ~ Splitting Hairs ~ “methods belong to an instance of…, functions do not…” ~ meh…
I’m used to Ruby, where this problem exists in spades… And can’t be gotten rid of with anything like “static”, alas :-/
(Ruby allows you to extract a scope from a lambda, and then pull all the stuff the lambda doesn’t use out from that scope. So it can’t be made safe from this problem, full stop, except by never hanging onto your lambdas for long.)
Interesting, I didn’t know that. I suspect the same is true for JavaScript, though and many other languages that know first-class functions, but not top-level functions…?
I wouldn’t say capturing ‘this’ is something wholly bad, as it follows the same single principle as elsewhere in Java lambda: capture value, not variable. If the captured value is the Object (i.e. its reference) of a local variable, then content of the object is changeable to anyone else that’s still keeping a reference. To make the capturing lambda fully ‘pure’, the captured Object has to be immutable, which is not always the case in Java land. So it’s just the creator of the lambda needs to keep in mind there could be shared states introduced.
It certainly isn’t wholly bad – but even when the shared state is immutable, you’ll be (potentially accidentally) hanging on to it for too long, such that the GC can’t collect it. Given that immutability as a strategy also highly depends on a powerful GC, it’s certainly something worth thinking about.
I don’t think GC is an issue here. The captured Object is there because you are still in need of the lambda, which is in need of the Object.
It’s another question if you just need part of the captured Object, because now either the Object should be spitted into smaller models and thus takes less memory, or you should use extra local variable for the lambda to capture just fields instead of the whole object.
Well, GC isn’t an issue directly, but everything you say has effects on GC behaviour…
I think you don’t understand closures. Closure is a lambda function which captures its outer scope variables as internal state. And it’s a feature not a bug. If you don’t want a closure, don’t use variables from outer scope. I am pleasantly surprised to see that Java has closures.
Please re-read the article thoroughly, and re-assess whether I don’t understand “closures” :)
The problem explained in the article being that a “closure” is likely to be created accidentally by implicitly capturing “this” in a variety of situations, including some situations where this is undesirable.
I think we missed why this problem happens:
Java 8 lambdas create implicit reference only when we are using some method or field from the enclosing class
From https://discuss.kotlinlang.org/t/lambdas-and-implicit-references-to-the-instance-of-the-enclosing-class/2288/
How did “we” miss it? It’s explained in the article…
IMHO, the root issue here is that it creates a reference cycle between the generated function from higher order function and the ‘this’ reference is captured in it. For GC with reference counting strategy, memory leak certainly will happen. But most of JVM GCs are implemented with tracing strategy. There may no memory leak happen when running on those JVM.